Graph-based remediation changes the approach to recovering MongoDB infrastructure. Instead of runbook scripts, the system autonomously finds the correct recovery paths and reduces the burden on on-call personnel.
In a distributed database, degradation is rarely linear. In the case of Stripe, the problem manifested at scale: sharded MongoDB clusters regularly entered partially inconsistent states. The old approach relied on hard-coded plugins and runbook sequences. It broke under combinations of failures, depended on specific layouts, and did not account for intermediate states. This led to frequent pager alerts and manual intervention. Over six months, the system triggered 124 times due to incorrect shard configurations and another 32 times during node degradation with additional issues. Even planned operations, such as index builds, were blocked for about an hour.
The solution shifted the focus from scripts to a model. Stripe presented the infrastructure as a graph: nodes are components, edges are their connections, and attributes describe the current state. Instead of fixed steps, the system uses graph search to find valid recovery paths. This allows the same logic to work with different topologies. Initially, BFS was used, but later the Dijkstra algorithm was adopted. The reason was the need to account for the “cost” of operations and avoid unnecessary actions. This choice is a compromise: higher computational complexity, but better control over side effects and more predictable system behavior.
A key architectural shift is the transition from procedural to declarative models. Recovery is described as a set of rules and states (state machine), rather than a rigid workflow. The scheduler dynamically combines operations based on the current state of the graph. This is important for systems with high variability: it is impossible to enumerate all scenarios in advance. The algorithm searches for a path not only to the target state but also to the “least bad” state if full recovery is unattainable. This reduces the blast radius and allows partial stabilization of the system without waiting for an ideal solution. Implementation requires careful modeling of states and transitions: an error in the model will lead to incorrect plans, even if the algorithm is correct.
Results demonstrate the practical value of the approach. The number of database-related pager alerts decreased by approximately 30%, equivalent to about 200 incidents per year. Additionally, around 12 days of unhealthy shard states have been eliminated annually. Importantly, the system adapts to changes in infrastructure without rewriting logic. Metrics for latency or throughput are not disclosed, but an indirect effect is the reduction in operation blocking time and decreased manual intervention. This indicates an improvement in operational resilience, not just optimization of recovery.
From an engineering perspective, this is an example of an evolutionary transition from runbook-driven operations to model-driven management. Runbooks work well for known scenarios but do not scale effectively as complexity increases. State machine + graph search allow the system to “explore” the state space rather than follow a predetermined path. The trade-off is a more complex model and the need for runtime computations. However, at the scale of global infrastructure, this trade-off appears pragmatic.
Interestingly, Stripe plans to extend the approach beyond reactive recovery. On the roadmap are automation of topology changes and blue-green deployments, as well as integration of scheduled maintenance with self-healing. This is a logical continuation: if the system can already find correct transitions between states, it can apply this not only during failures but also during infrastructure evolution.
A similar trend is visible in the industry. Other companies are also investing in self-healing and declarative platforms. The overall direction is to reduce the burden on engineers and move away from manual management of complex systems. In environments where degradation is the norm, the challenge is no longer to avoid failures but to recover quickly and correctly without burning out on-call teams.