Automated RCA in microservices is a practical way to cut incident analysis from guesswork to validation. At Atlassian, the system turns telemetry correlation into ranked root cause hypotheses.
At Atlassian’s scale, hundreds of interconnected microservices across multiple regions make incident response noisy by default. A single user-impacting failure can generate enough telemetry to slow down diagnosis instead of speeding it up. The bottleneck is not visibility. The bottleneck is the cost of finding the causal signal inside metrics, logs, and traces. In practice, that work still depends on human memory, intuition, and manual cross-referencing.
The team’s answer is an automated RCA pipeline built around a simple engineering idea: root cause analysis is a correlation problem across three dimensions. If anomalies can be detected independently, aligned on a shared timeline, and traced through the service dependency graph, the system can produce ranked hypotheses about where the failure started and how it propagated. This is a pragmatic choice, not an attempt to replace incident expertise. It shifts responders from hypothesis generation to hypothesis validation, which is where time matters most during an outage.
The architecture is modular by design. Each anomaly detector is a pluggable component, and the correlation engine consumes normalized anomaly events regardless of their source. That gives the team room to evolve the pipeline without rebuilding it. Statistical detectors can be swapped for ML models later, and new signal types can be added incrementally. The trade-off is clear: modularity adds interface discipline, but it avoids a tightly coupled system that would be hard to extend under incident pressure.
The first step after incident detection is to narrow the blast radius. Instead of scanning the whole platform, the system queries an OpenTelemetry-derived service map to isolate the services in the affected call path. That reduces the problem from hundreds of services to a focused subgraph, typically tens of services. The graph is built from span-level parent-child relationships observed in production traffic, so it reflects how services actually communicate, not how documentation says they should.
Once the relevant services are identified, the system runs specialized detectors per telemetry signal. For metrics, it monitors RED signals, meaning rate, error rate, and duration, and uses statistical methods such as median absolute deviation and percentile bands. For traces, it looks for structural anomalies such as unexpected exceptions, novel propagation patterns, and span-level latency spikes. For logs, it uses embedding-based clustering to surface rare or new error clusters without naively scanning every line. The implementation detail that matters here is normalization. Each detector emits the same event schema, so the correlation engine can reason across signals without caring which detector produced them.
Correlation starts with time. The engine groups anomalies that occur close together using a sliding window, typically plus or minus five minutes. That matters because a fault rarely appears in one signal only. A database error can trigger upstream timeouts, which then surface as frontend 500s. The system assigns each bundle a temporal cohesion score, so tightly clustered events rank above dispersed ones. It also deduplicates repeated failure chains with sequence fingerprinting. Without that step, the same causal pattern can replay dozens of times and drown the real signal in copies.
After temporal grouping, the system uses the dependency graph to infer direction. It starts from the sink node, the service with the highest anomaly severity, and traverses upstream with bounded BFS. If Service A depends on Service B, and B’s anomaly appears earlier in time, B is more likely to be the origin. Each candidate path is scored by anomaly severity, dependency strength, and temporal distance from the sink. The bundle score combines temporal cohesion and path score, and the top-ranked bundles become root cause hypotheses.
The result is not just a list of services. Each hypothesis includes an explanation that responders can read quickly and validate against telemetry. That matters because trust in incident tooling comes from evidence, not from a score alone. The system is also part of a larger incident response platform at Atlassian, where automated user-impact detection, faulty service identification, causal diagnosis, and an AI-powered incident copilot share one incident context. That shared context is the control plane for the whole workflow. It keeps signals, hypotheses, and actions aligned in one place.
The most useful lesson is also the most conservative one. Start with the simplest anomaly detection that works. Statistical methods are easier to debug than complex ML models, especially when they produce false positives. Modularity compounds over time. Deduplication is mandatory at scale. And the dependency graph is the strongest prior when the goal is to distinguish a broken service from a service that is only affected by another failure.
The current system runs once per incident trigger. The next step is more adaptive orchestration, likely with LLM-based agents that can request additional telemetry and refine hypotheses safely. The team also plans to expand the signal set to infrastructure metrics, deployment events, feature flag changes, and synthetic checks. That moves the system closer to the harder question: not only which service failed, but what change caused it to fail.