Configuration changes — modifications to feature flags and service parameters — are important to operating services, but they can cause severe outages (SEVs) with notable service disruption. While code diff risk prediction is well studied, configuration diff risk has received little attention. The code-oriented Diff Risk Score (DRS) model provides limited discriminatory power for config diffs because config risk stems not from code complexity but from the importance, blast radius, and operational readiness of the affected services. We describe the development of CORA (COnfig R}isk Analyzer), a dedicated risk model for config diffs. A notable insight is that the config-path-to-service mapping bridges from what changed to which services are affected, enabling risk assessment in terms of service criticality rather than code properties.

CORA evolved through three iterations. A logistic regression model with config-specific features achieved an 11% improvement in recall at 5% gating over the DRS model on config diffs. Used for a freeze period, it reduced config gating while outage impact decreased 16.4% and config diff landing volume increased 47.7%. A unified LightGBM model trained on Meta-wide data achieved a 25.95% improvement in recall at 10% gating, with organization-level improvements ranging from 15% to 66%. An enhanced model with Bayesian hyperparameter optimization, SEV-severity-aware training, and enriched service features achieved a 22% improvement in recall at 15% gating with notable additional outage prevention. To our knowledge, CORA is the first system to apply predictive risk modeling specifically to configuration changes.