💻 computer science

KRCA: An Efficient Root Cause Analysis System in Hyper-Scale Microservice Systems via Agentic AI

This paper presents KRCA, an end-to-end root cause analysis system for hyper-scale microservices that combines a multi-stage pipeline with a memory-augmented multi-agent framework to achieve high diagnostic accuracy and efficiency, as evidenced by its successful six-month deployment at Kuaishou which reduced diagnosis time by 77.3%.

Original authors: Jiamin Jiang, Jingfei Feng, Yu Luo, Qingliang Zhang, Yongqian Su, Wenwei Gu, Shenglin Zhang, Tianyu Cui, Yao Wu, Jielong Huang, Nan Qi, Dan Pei

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiamin Jiang, Jingfei Feng, Yu Luo, Qingliang Zhang, Yongqian Su, Wenwei Gu, Shenglin Zhang, Tianyu Cui, Yao Wu, Jielong Huang, Nan Qi, Dan Pei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, futuristic cruise ship with 200,000 tiny, independent engines (microservices) working together to keep the ship running. Every day, new engines are added, old ones are tweaked, and different teams are constantly making changes.

One morning, an alarm blares: "The ship is slowing down!"

In the past, finding out which specific engine caused the slowdown was like looking for a needle in a haystack the size of a city. You'd have to check every single engine, one by one, often getting confused by the noise of thousands of other engines that were just reacting to the problem, not causing it. This took hours, and the ship (or in this case, an app like Kuaishou) would suffer huge losses while you figured it out.

KRCA is a new, super-smart detective system designed to solve this specific problem. Here is how it works, broken down into three simple steps:

1. The "Drill-Down" Flashlight (Narrowing the Search)

Instead of checking all 200,000 engines, KRCA starts at the exact spot where the alarm sounded (a specific API). It uses a special "flashlight" to look only at the engines directly connected to that spot.

The Analogy: Imagine a detective who doesn't search the whole city. Instead, they start at the crime scene and only follow the footprints of people who were there at that exact moment.
How it works: It checks the "failure rate" and "speed" of connected services. If a downstream engine is acting weird, it follows that path. If an engine is acting normal, it ignores it. This instantly shrinks the search from 200,000 suspects down to just a handful of "suspicious" ones.

2. The "Skeleton Map" (Organizing the Clues)

Once the system finds a suspicious engine, that engine might have hundreds of sensors (metrics) attached to it (temperature, pressure, fuel flow, etc.). Trying to figure out which sensor is the real culprit by looking at raw numbers is like trying to solve a puzzle with 400 pieces that are all jumbled up.

The Analogy: Instead of staring at a pile of jumbled puzzle pieces, KRCA first builds a skeleton frame of the puzzle. It knows that "Fuel Flow" usually causes "Engine Speed" issues, and "Temperature" usually causes "Pressure" issues.
How it works: It takes the weird sensor readings and snaps them into a pre-made "skeleton" map based on how the system is supposed to work. This gives the detective a structured starting point, so they don't get lost in the noise.

3. The "Team of Specialists" (The Agentic AI)

Now that the system has a small list of suspects and a structured map, it calls in a team of AI experts.

The Analogy: Imagine a main detective (the Main Agent) who calls in a Traffic Expert, a CPU Expert, and a Database Expert.
- The Main Agent says: "Hey, the CPU is hot. Expert CPU, check if this is a real problem or just a glitch."
- The Expert checks its own memory bank of past cases (like a veteran detective remembering a similar case from last year) and uses tools to gather proof.
- They talk to each other, cross-checking facts, and saying, "No, that's not it," or "Yes, that's definitely the cause."
The Result: They work together to write a final report that says exactly which engine broke and why (e.g., "A memory leak in the storage service caused a chain reaction").

Why is this a big deal?

Speed: Before KRCA, it took engineers about 52 minutes to find the root cause. With KRCA, it takes about 11 minutes. That's a 77% reduction in time.
Accuracy: In tests, KRCA found the right engine 88% of the time and correctly identified the type of failure 79% of the time. This is much better than previous methods, which often got lost in the massive amount of data.
Real-World Success: This isn't just a theory. It has been running on Kuaishou's actual production system for over six months, handling thousands of alerts a day and saving the company significant money by stopping failures faster.

In short: KRCA is like a highly organized, multi-specialist detective team that uses a flashlight to narrow the search, a skeleton map to organize clues, and a memory bank of past cases to solve massive, complex system failures in minutes instead of hours.

Technical Summary: KRCA - An Efficient Root Cause Analysis System in Hyper-Scale Microservice Systems via Agentic AI

1. Problem Statement

Hyper-scale microservice systems, characterized by extreme dynamism (decentralized, continuous updates across hundreds of thousands of services) and massive scale (hundreds of thousands of services monitored by hundreds of metrics each), present unique challenges for Root Cause Analysis (RCA). Existing RCA methods struggle in this environment:

Deep Learning Approaches: Require frequent retraining to adapt to changing system features. In hyper-scale systems, the variation cycle is often faster than the training cycle, rendering these models ineffective for real-time deployment.
LLM-Based Approaches: While capable of in-context learning, they face the "Lost in the Middle" dilemma when processing the prohibitively long contexts of massive service and metric sets. They often hallucinate causal relationships or fail to extract true causality from raw, high-dimensional time-series data.
Traditional Causal Discovery: Algorithms like PC and Granger causality suffer from sharp accuracy degradation and prohibitive computational costs as the number of anomalous metrics increases (e.g., dropping below 20% accuracy with 20 metrics).

The core challenge is to localize the root cause service and classify the failure type within a practical time budget without relying on pre-trained models that cannot keep pace with system evolution.

2. Methodology: KRCA System Design

KRCA is an end-to-end RCA system designed for hyper-scale environments. It employs a progressive multi-stage diagnostic paradigm to systematically narrow the search space before engaging LLM-based reasoning. The system consists of three core modules:

2.1 API-Level Drilldown

Instead of starting at the coarse service level, KRCA initiates diagnosis from the specific API-level alert.

Mechanism: It recursively traverses the service dependency graph starting from the alerting API.
Scoring: A scoring function evaluates downstream APIs based on failure rate (temporal correlation of error rates) and latency (a composite score of anomaly degree, fluctuation contribution, and correlation).
Pruning: Paths with scores below a threshold are pruned. This reduces the search space from thousands of services to a compact set of top- $N$ suspicious services, effectively filtering out irrelevant downstream noise.

2.2 Skeleton-Based Causal Graph Instantiation

For the identified suspicious services, KRCA avoids direct causal discovery from raw time-series data.

Generic Skeleton ( $G_s$ ): A predefined causal skeleton is used, composed of four meta-metric types: External (E) (e.g., traffic), Internal (I) (e.g., CPU/GPU), Dependency (D) (e.g., MySQL, Redis), and KPI (K) (e.g., latency, success rate).
Instantiation: Anomalous metrics detected via a hybrid strategy (Isolation Forest + PatternMatcher) are mapped to these meta-metric types.
Output: This generates an initial, high-recall causal graph ( $G_o$ ) that serves as a structural prior. This step organizes the massive metric set into a sparse, interpretable structure, constraining the subsequent search space for the LLM.

2.3 Multi-Agent Collaboration

To refine the initial graph $G_o$ and generate the final diagnosis, KRCA utilizes a memory-augmented multi-agent framework.

Architecture: A Main Agent orchestrates the process, coordinating specialized Sub Agents (e.g., Traffic Expert, CPU Expert, GPU Expert) aligned with specific failure scenarios.
Reasoning: Sub Agents verify causal links and gather evidence using domain-specific tools (e.g., SQL analyzers) and Retrieval-Augmented Generation (RAG).
Memory System: A three-tier memory system supports reasoning:
- Working Memory: Short-term dialogue history for context.
- Factual Memory: Service-specific historical failures for in-context learning.
- Experiential Memory: Generalized reasoning insights (e.g., "traffic saturation causes MySQL slow queries") transferable across services.
Process: Agents iteratively prune spurious edges, discover implicit intra-meta-metric relations, and synthesize a comprehensive failure report identifying the root cause service and failure type.

3. Key Contributions

First End-to-End System for Hyper-Scale: KRCA is presented as the first RCA system specifically designed for hyper-scale microservice systems, capable of operating within real-world incident response time budgets.
Progressive Multi-Stage Framework: The paper proposes a novel pipeline that combines API-level drilldown to reduce search space, skeleton-based causal constraints to organize anomalous metrics, and multi-agent reasoning to verify causality. This addresses the limitations of both pure statistical and pure LLM-based approaches.
Empirical Validation and Deployment: The system was evaluated on 300 real-world failures and deployed in Kuaishou's production environment for over six months.

4. Experimental Results

The evaluation was conducted on a dataset of 300 real-world failures from Kuaishou, covering nine classical failure types (e.g., CPU saturation, MySQL slow queries).

Accuracy: KRCA achieved an AC@1 score of 0.88 for root cause service localization and 0.79 for failure type classification.
Comparison: These results outperform the strongest baseline (RCA-Agent) by absolute gains of 31% and 32%, respectively. KRCA also demonstrated robustness across different seed LLMs (Claude, GPT, MiniMax, DeepSeek).
Efficiency: In production, KRCA reduced the average diagnosis time for root cause localization by 77.3% (from ~52 minutes to ~11.8 minutes).
Ablation Studies: Removing any core component (API-level drilldown, skeleton graph, multi-agent collaboration, or tiered memory) resulted in significant performance drops, validating the necessity of the full architecture.
Usability: Human evaluation by five senior SREs rated the generated reports highly for readability (4.21/5) and usefulness (4.10/5).

5. Significance and Claims

The paper claims that KRCA successfully bridges the gap between the need for real-time adaptability and the complexity of hyper-scale systems. By shifting the paradigm from "inferring causality from raw data" to "verifying causality within a structured, constrained prior," KRCA mitigates the hallucination risks of LLMs and the scalability limits of statistical methods.

The system's significance is highlighted by its successful production deployment at Kuaishou, where it handles over 2,000 alerts daily across six major business lines. The authors emphasize that the system's ability to reduce Mean Time to Recovery (MTTR) and its high Net Promoter Score (NPS) of 74% among SREs demonstrate its tangible value in mitigating the impact of failures on end users and business operations (e.g., preventing estimated GMV losses of over $130,000 in a single case study).