🤖 AI

Anomaly Detection and Root Cause Analysis for Microservice Systems

This thesis advances automated anomaly detection and root cause analysis for microservice systems by addressing five key limitations through novel end-to-end frameworks (BARO, EventADL, and TORAI) that leverage diverse observability data without requiring service call graphs, alongside the introduction of the RCAEval benchmark and systematic evaluation to standardize future research.

Original authors: Luan Pham

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Luan Pham

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling city where every app you use is a different neighborhood. In the old days, these neighborhoods were all part of one giant, walled city (a "monolith"). If the bakery caught fire, the whole city might go dark, but at least you knew exactly where to look. Today, however, we've replaced that city with thousands of tiny, independent villages called "microservices." They talk to each other constantly to make your online shopping or video streaming work. The problem? If one tiny village has a glitch, it can send a ripple effect that crashes the whole city, causing slow loading times or total blackouts. To keep this digital city running, engineers need to act like super-smart detectives. They have to spot the moment something goes wrong (called "anomaly detection") and then figure out exactly which village started the trouble (called "root cause analysis"). But with millions of data points flying around every second—like traffic counts, power usage, and error messages—it's impossible for humans to find the needle in the haystack manually.

This thesis, written by Qui Luan Pham, tackles the challenge of building an automated detective system for these digital cities. The author argues that most current detective tools are flawed because they treat "spotting the problem" and "finding the culprit" as two separate jobs, often assuming the first job was done perfectly. In reality, automated detectors often get confused by noise or get the timing slightly wrong, which throws the whole investigation off. The paper also points out that many existing tools rely on having a perfect map of how the villages connect, which is often impossible to get in the real world. To solve this, the author developed three new, smarter detective tools and a giant, standardized "training ground" to test them.

The Three New Detective Tools

The thesis introduces three distinct methods, each designed to handle different types of clues and different levels of messiness in the data.

1. BARO: The Metric Detective
Think of "metrics" as the vital signs of the system: heart rate (CPU usage), breathing speed (latency), and body temperature (error rates). The first tool, BARO, is designed to read these vital signs. Previous detectives would look at the heart rate, scream "Heart attack!" and then immediately try to find the cause, assuming the alarm was 100% accurate. BARO is different. It knows that alarms can be shaky. It uses a special statistical technique called "Multivariate Bayesian Online Change Point Detection" to look at how all the vital signs move together. If the heart rate spikes but the breathing stays calm, it might just be a false alarm. But if everything changes together, it knows a real problem is happening. Even if the alarm goes off a few seconds too early or too late, BARO has a "Robust Scorer" that can still figure out which village is the culprit. It's like a detective who doesn't panic if the witness is slightly unsure about the time of the crime; they can still solve the case.

2. EventADL: The Event Detective
While vital signs are great, sometimes the real clues are in the "events"—the specific actions people take, like "User X deleted a file" or "Server Y was restarted." Most previous tools ignored these event logs or treated them like messy, unstructured text. EventADL is the first tool to treat events like a structured story. It learned from analyzing 520 real-world incidents at a major cloud company. It looks for patterns in who did what to which resource. For example, it might notice a pattern where a specific security group is deleted, followed immediately by a server failing to start. It builds a "story map" (an Intervention Graph) to show exactly how one action led to the disaster. This tool is "open-box," meaning it doesn't just say "something is wrong"; it explains the story of why it happened, making it much easier for human engineers to trust and fix.

3. TORAI: The Multimodal Detective
In the real world, sometimes the map is missing pieces. Some villages are "blind spots"—maybe they are old, or they are run by a third party, and we can't see their internal logs or traces. Most tools fail completely if they can't see the whole map. TORAI is the ultimate detective that works even with a broken map. It combines all available clues: vital signs (metrics), action logs (events), and the few traces it does have. It groups services by how "severe" their symptoms are and uses a method called "causal ranking" to figure out the root cause, even if it can't see every single connection. It's like a detective who can solve a mystery even if half the witnesses are missing, by piecing together the few clues that are left.

The Training Ground: RCAEval

To prove these new detectives were actually better than the old ones, the author didn't just run a few tests. They built RCAEval, the first standardized "training ground" for this field. Before this, every researcher used their own different datasets and rules, making it impossible to compare who was actually the best. RCAEval provides a massive library of 735 different failure scenarios across three different digital cities, along with 15 different "baseline" detective methods to compete against. It's like creating a standardized Olympic track so every runner can be judged fairly.

What the Tests Showed

The author put BARO, EventADL, and TORAI through rigorous tests on these benchmark systems.

BARO consistently outperformed existing methods in finding the root cause, even when the initial alarm was slightly inaccurate. It proved that being robust to timing errors is crucial.
EventADL showed that looking at structured events is a game-changer, achieving very high accuracy in both spotting anomalies and explaining them.
TORAI demonstrated that you don't need a perfect map to solve the mystery; it could find the root cause even when parts of the system were invisible.

However, the paper also found that not all existing methods are created equal. The author ran a massive study on 21 different "causal inference" methods (tools that try to figure out cause-and-effect). They found that many of these popular tools struggle when the system gets too big (more than 200 services) or when the data is messy. Some methods were incredibly slow, taking hours to solve a problem that should take seconds, while others failed completely if the data wasn't perfect.

The Bottom Line

This thesis suggests that the future of fixing digital cities lies in combining different types of clues (metrics, events, and traces) and building tools that are tough enough to handle imperfect data. The author didn't just invent new tools; they built the infrastructure to test them fairly. While the paper doesn't claim to have solved every problem in the world, it provides a solid, reproducible foundation that moves the field from "guessing" to "knowing." It shows that with the right statistical tricks and a willingness to look at all types of data, we can build systems that detect and fix themselves much faster than ever before.

Technical Summary: Anomaly Detection and Root Cause Analysis for Microservice Systems

1. Problem Statement

Microservice architectures, while offering modularity and scalability, introduce inherent complexity that makes failures inevitable and difficult to diagnose. When a failure occurs in one service, it often propagates, causing widespread degradation in performance and availability. Manual monitoring is infeasible due to the volume of observability data (metrics, logs, traces, and events). Consequently, automated Anomaly Detection (AD) and Root Cause Analysis (RCA) are critical.

However, existing approaches suffer from five significant limitations identified in this thesis:

Decoupling of AD and RCA: Most methods assume anomaly detection is perfect. In reality, imprecise detection (e.g., delayed or noisy failure timestamps) cascades errors into the RCA phase, degrading localization accuracy.
Underexplored Event Data: While metrics and logs are well-studied, structured event data (e.g., API calls, configuration changes) capturing actor-resource interactions remains largely unexplored for AD and RCA.
Blind Spots in Trace Coverage: Many multimodal RCA methods require a complete service call graph derived from distributed traces. In practice, "blind spots" exist due to third-party services, legacy components, or incomplete instrumentation, rendering these methods ineffective.
Lack of Standard Benchmarks: The field lacks standardized datasets and evaluation frameworks, making fair comparison and assessment of generalizability difficult.
Limited Evaluation of Causal Methods: Causal inference-based RCA methods are dominant but lack systematic empirical evaluation regarding their effectiveness, efficiency, and robustness across diverse scenarios and data lengths.

2. Methodology

The thesis proposes a unified framework addressing these limitations through two groups of contributions: novel methods for detection and analysis, and benchmarking infrastructure.

A. Novel Methods

The proposed methods progress along two axes: the observability modality consumed and the assumptions relaxed.

1. BARO (Metric-based AD & RCA)

Modality: Multivariate time-series metrics.
Core Innovation: An end-to-end framework that jointly performs AD and RCA to mitigate the impact of imprecise detection.
Mechanism:
- Anomaly Detection: Uses Multivariate Bayesian Online Change Point Detection (BOCPD) to model dependencies and correlations among metrics, detecting distribution shifts without user-defined thresholds.
- Root Cause Analysis: Introduces RobustScorer, a non-parametric hypothesis testing technique. Unlike existing methods relying on mean and standard deviation (sensitive to outliers), RobustScorer uses median and Interquartile Range (IQR). This makes it robust to inaccuracies in the detected failure time ( $\hat{t}_A$ ), allowing it to rank root cause services and metrics even when the anomaly detection time is slightly early or delayed.

2. EventADL (Event-based AD & RCA)

Modality: Structured event data (e.g., OCSF schema).
Core Innovation: The first open-box framework for event-based AD and localization, grounded in an analysis of 520 real-world AWS incidents.
Mechanism:
- Pattern Learning: Models normal behavior using Event Semantic Patterns (ESP) (actor-operation-resource relationships) and Event Frequency Patterns (EFP).
- Detection: Detects deviations in semantic or frequency patterns.
- Localization: Constructs an Intervention Graph to trace causal relationships and localizes root causes via time-aware random walks, providing interpretable explanations for why an anomaly was detected.

3. TORAI (Multimodal RCA)

Modality: Metrics, logs, and traces (multimodal).
Core Innovation: A multimodal RCA approach designed to function without a complete service call graph, addressing the "blind spot" problem.
Mechanism:
- Severity Clustering: Groups services by severity to handle incomplete trace coverage.
- CausalRanker: Constructs causal graphs from available data and applies causal ranking to identify fine-grained root causes.
- FineGrainer: Refines the diagnosis to specific indicators.
- Robustness: By not relying on a full trace-derived graph, it effectively diagnoses failures even when parts of the system are uninstrumented.

B. Benchmarking and Evaluation Infrastructure

4. RCAEval

Contribution: The first open-source, comprehensive benchmark for RCA in microservices.
Components: Includes three large-scale datasets (735 failure cases across Online Boutique, Sock Shop, and Train Ticket), fifteen reproducible baselines, and standardized evaluation protocols for both coarse-grained (service-level) and fine-grained (metric/event-level) RCA.

5. Systematic Evaluation of Causal Inference

Contribution: A large-scale empirical study evaluating nine causal discovery methods and twenty-one RCA methods.
Scope: Assesses performance across synthetic and real datasets, varying data lengths, and hyperparameter settings to identify strengths and weaknesses of the causal inference paradigm.

3. Key Results

Experiments were conducted on benchmark systems (Online Boutique, Sock Shop, Train Ticket) and real-world incident data.

BARO Performance:
- Anomaly Detection: Achieved the highest F1-scores (0.82, 0.75, 0.81) across three datasets, outperforming baselines like N-Sigma, SPOT, and BIRCH.
- Root Cause Analysis: Consistently outperformed state-of-the-art methods (e.g., CIRCA, RCD, CausalRCA) in ranking root cause services (Avg@5). Crucially, it maintained high accuracy even when the input anomaly detection time was imprecise, whereas baselines suffered significant performance drops.
EventADL Performance:
- Achieved F1-scores of at least 90% for anomaly detection and 100% top-3 accuracy for root cause localization on real cloud-based systems.
- Demonstrated superior interpretability compared to "closed-box" deep learning approaches.
TORAI Performance:
- Outperformed baselines in scenarios with incomplete trace coverage (blind spots), successfully identifying root causes where trace-dependent methods failed.
Causal Inference Evaluation:
- No Single Best Method: No single causal inference-based method performed best across all settings.
- Scalability Issues: Most causal discovery methods (e.g., PC, FCI, Granger) and RCA methods struggled with large-scale systems (>200 metrics), with running times increasing significantly.
- Data Length: Performance generally improved with longer data lengths for methods like RCD and CausalRCA, but methods like BARO and NSigma performed well even with shorter data lengths if the failure time was accurate.
- Efficiency: BARO and NSigma were the fastest methods, while deep learning-based causal methods (e.g., CausalRCA, RUN) were significantly slower.

4. Significance and Claims

The thesis claims to advance the field of automated anomaly detection and RCA in microservice systems through a cohesive framework that operates across multiple data modalities.

Methodological Advancement: It demonstrates that relaxing the assumption of perfect anomaly detection (via RobustScorer) and moving beyond single-modal analysis (via EventADL and TORAI) significantly improves diagnostic robustness and coverage.
Practical Applicability: The proposed methods are unsupervised (requiring no labeled failure data), computationally efficient for online use, and provide interpretable outputs, addressing the operational constraints of real-world systems.
Standardization: By introducing RCAEval, the thesis addresses the critical lack of standardized benchmarks, enabling fair comparison and accelerating future research.
Empirical Insight: The systematic evaluation of causal inference methods provides novel insights into their practical limitations, particularly regarding scalability and sensitivity to failure timing, guiding future research directions.

The work concludes that while significant progress has been made, the path to truly autonomous, self-healing microservice systems requires continued innovation in explainability, proactive remediation, and scalability to massive, heterogeneous environments.