💻 computer science

TORAI: Unsupervised Fine-grained RCA using Multi-Source Telemetry Data

TORAI is a novel unsupervised root cause analysis approach that effectively identifies fine-grained failures in microservice systems by leveraging multi-source telemetry data to cluster services and perform causal ranking, thereby overcoming the limitations of existing methods that rely on complete service call graphs and struggle with "blind spot" services lacking traces.

Original authors: Luan Pham, Huong Ha, Xiuzhen Zhang, Hongyu Zhang

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luan Pham, Huong Ha, Xiuzhen Zhang, Hongyu Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the chief mechanic for a massive, futuristic city made entirely of thousands of tiny, interconnected robots. Each robot has a specific job: some sell tickets, some deliver packages, some handle money, and some just talk to each other. This is your Microservice System.

When everything works, the city hums along. But sometimes, a robot breaks, causing a traffic jam that stops the whole city. Your job is Root Cause Analysis (RCA): finding out exactly which robot broke and why before the city collapses.

The Problem: The "Blind Spot" Mystery

Traditionally, mechanics tried to solve these mysteries by looking at a Service Call Graph. Think of this graph as a giant, detailed map showing exactly which robot talks to which other robot.

However, there's a big problem:

The Map is Incomplete: In the real world, some robots are "black boxes" (maybe they are old, or made by a third party). You can't see their internal wiring, so they don't appear on the map. We call these Blind Spots.
The Old Methods Fail: Previous detective tools relied entirely on that map. If the broken robot was in a "Blind Spot" (off the map), the old tools would just say, "I don't know," or they'd blame the wrong robot nearby.
Too Much Work: To fix the map, you'd have to go into every single robot and install a camera (instrumentation). That takes forever and is impossible for closed-source robots.
Need for a Training Manual: Some tools needed a huge library of past accidents (labeled data) to learn how to solve new ones. But in a fast-changing city, every accident is unique, so you never have enough training data.

The Solution: TORAI (The Intuitive Detective)

The authors of this paper created TORAI, a new kind of detective that doesn't need a perfect map, doesn't need a training manual, and doesn't need to install cameras everywhere.

Here is how TORAI works, using a simple analogy:

1. The "Pain Meter" (Severity Scorer)

Instead of looking at a map, TORAI first checks the vital signs of every robot.

It looks at Metrics (like heart rate, CPU usage, temperature).
It looks at Logs (like a diary of what the robot said: "Error! Error!").
It looks at Traces (if available, like a call log).

TORAI asks: "How much does this robot hurt?" It gives every robot a "Pain Score." If a robot is screaming in pain, it gets a high score. If it's fine, it gets a low score.

Key Superpower: It doesn't matter if the robot is on the map or not. If it has a heartbeat (metrics) or a diary (logs), TORAI can measure its pain.

2. The "Group Hug" (Symptom Cluster)

Next, TORAI groups the robots together based on how much they hurt.

Imagine a triage nurse in a hospital. They don't treat everyone at once. They group the patients with "severe headaches" together and the patients with "slight stomach aches" together.
TORAI does this with robots. It separates the "sick" group from the "healthy" group. This saves time because the detective doesn't need to waste energy analyzing the healthy robots.

3. The "Who Started It?" (Causal Ranker)

Now, TORAI looks inside the "Sick Group." It asks: "Who is the bully?"

In a group of sick friends, usually, one person caught a cold first and passed it to the others.
TORAI uses Causal Analysis (a fancy way of looking at cause-and-effect) to figure out the order of events. Did Robot A's high temperature cause Robot B to crash? Or did Robot B crash first?
It ranks the robots to find the one that started the chain reaction.

4. The "Fine-Tuning" (Fine Grainer)

Finally, once TORAI identifies the "Bully Robot" (the root cause service), it needs to know exactly what broke.

Did the CPU overheat? Did a specific line of code throw an error?
TORAI uses Hypothesis Testing (a statistical check) to look at the specific logs or metrics of that robot. It asks: "Is this specific error log happening way more often than usual?"
If yes, it points a finger at that specific log line or metric as the Fine-Grained Root Cause.

Why TORAI is a Game Changer

It works in the dark: Even if 50% of the robots are "Blind Spots" (no map, no traces), TORAI can still find the culprit using just their heartbeats (metrics) and diaries (logs).
It's self-taught: It doesn't need a history book of past accidents. It figures it out on the fly using math and logic.
It's fast: In tests, TORAI found the culprit in seconds, even in cities with hundreds of robots, while other methods took minutes or hours (or gave up entirely).

The Real-World Test

The authors tested TORAI on three different "cities" (benchmark systems) and even on 10 real-world accidents from a major internet company.

Result: TORAI was the best at finding the right robot. In the real-world test, it was 100% accurate at putting the correct culprit in its top 3 guesses.
The "Blind Spot" Proof: When they tested a system where no robots had maps (100% blind spots), TORAI still worked perfectly, while the old map-reliant tools failed completely.

Summary

TORAI is like a detective who stops trying to read a broken map and instead listens to the screams of the victims. By grouping the sick ones, figuring out who started the fight, and pinpointing the exact weapon used, it solves the mystery of why the microservice city crashed—even when half the city is invisible.

1. Problem Statement

Microservice systems are complex and prone to failures, making Root Cause Analysis (RCA) critical for minimizing downtime and economic loss. Existing multi-source RCA methods (utilizing metrics, logs, and traces) suffer from three primary limitations that hinder their practical application:

Reliance on Full Trace Coverage: Most methods assume a complete service call graph constructed from distributed traces. In reality, microservice systems often contain "blind spots"—services without traces (e.g., black-box components, compiled software, or uninstrumented legacy services). When blind spots exist, the call graph is incomplete, causing existing methods to miss critical data and fail to diagnose root causes accurately.
Dependence on Labeled Data: Many state-of-the-art approaches are supervised, requiring large volumes of manually labeled training data. This is impractical for dynamic microservice environments where services evolve rapidly and historical fault data is scarce or obsolete.
Intrusive Instrumentation: Some methods require tight integration between data sources (e.g., embedding trace_id in every log line) or heavy source code modification to achieve full trace coverage. This is often infeasible for closed-source components or third-party services.

The Core Challenge: How to perform accurate, fine-grained RCA in microservice systems with incomplete visibility (blind spots), without requiring labeled data or intrusive code changes.

2. Methodology: TORAI

The authors propose TORAI, a novel unsupervised approach that identifies fine-grained root causes using available multi-source telemetry (metrics, logs, and traces) without constructing a service call graph. The methodology follows a five-stage pipeline:

A. Data Transformation

TORAI converts raw multi-source telemetry into time series data:

Metrics: Aggregates the "four golden signals" (Traffic, Saturation, Latency, Errors).
Logs: Parses logs into templates (using the Drain algorithm) and counts occurrences over time, avoiding semantic extraction to reduce overhead.
Traces: Extracts latency and status codes from available traces but does not build a call graph.

B. SeverityScorer (Anomaly Severity Measurement)

Calculates an anomaly severity score for each time series by measuring the deviation of data points during the failure period from the expected behavior (mean and standard deviation) of the normal period.
Generates a severity vector $[\rho_m, \rho_l, \rho_{tc}]$ for each service. If a data source (e.g., traces) is missing for a service, the corresponding value is imputed as 0.
Goal: Quickly identify and separate abnormal services from normal ones to reduce the search space for subsequent steps.

C. SymptomCluster (Clustering)

Uses a Gaussian Mixture Model (GMM) to cluster services based on their severity vectors.
Determines the optimal number of clusters using the Bayesian Information Criterion (BIC).
Goal: Group services with similar anomaly symptoms. This allows the system to focus causal analysis only on relevant groups rather than the entire system, improving efficiency and handling "blind spot" services that may not have traces but show metric/log anomalies.

D. CausalRanker (Coarse-Grained RCA)

Performs causal inference within each severity cluster to identify the root cause service.
Uses a divide-and-conquer strategy: It partitions time series into smaller chunks and applies the $\Psi$ -PC algorithm (a causal discovery method robust to interventions) to construct causal graphs.
Identifies "interventional targets" (nodes with no parents in the causal graph) as potential root causes.
Goal: Rank services within a cluster based on causal influence, rather than just anomaly magnitude.

E. RankAggregation

Combines the cluster-level ranking (from SymptomCluster) and the intra-cluster service ranking (from CausalRanker) to produce a final ranked list of coarse-grained root cause services.
Prioritizes services in high-severity clusters, then orders them by causal influence.

F. FineGrainer (Fine-Grained RCA)

Once a root cause service is identified, this module pinpoints the specific root cause indicator (e.g., a specific metric like CPU usage or a specific log template).
Uses hypothesis testing (comparing distributions before and after the anomaly).
Robustness: Instead of using Mean/Standard Deviation (which are sensitive to outliers), it uses Median and Interquartile Range (IQR) to calculate deviation. This makes the method robust to imprecise anomaly detection timestamps.
Goal: Output a ranked list of specific indicators (e.g., "Service 1 - CPU") to guide engineers to the exact fault.

3. Key Contributions

Identification of Limitations: The paper formally identifies and addresses the "blind spot" problem in existing RCA methods, highlighting the impracticality of assuming full trace coverage in evolving systems.
TORAI Framework: A novel, unsupervised RCA method that:
- Does not require a constructed service call graph.
- Does not require labeled training data.
- Does not require intrusive source code modifications (e.g., adding trace IDs to logs).
Comprehensive Evaluation: Extensive experiments on three benchmark systems (Online Boutique, Sock Shop, Train Ticket) and 10 real-world production failures, comparing against nine state-of-the-art baselines.

4. Experimental Results

The authors evaluated TORAI against 9 baselines (including PDiagnose, CausalRCA, RCD, BARO, etc.) across three datasets with varying fault types (CPU, memory, network, etc.).

Coarse-Grained Accuracy (Service Level):
- TORAI significantly outperformed all baselines. On the Online Boutique dataset, TORAI achieved an Avg@5 score of 0.93, compared to 0.60 for the next best (RCD).
- On the Sock Shop dataset (which has 100% blind spots as it has no traces by default), TORAI achieved an Avg@5 of 0.94, while trace-dependent methods failed completely.
- On the large Train Ticket system (64 services), TORAI achieved 0.89 Avg@5, vastly outperforming CausalRCA (0.43) and CIRCA (0.46).
Fine-Grained Accuracy (Indicator Level):
- TORAI achieved the highest accuracy in identifying specific root cause indicators (e.g., specific metrics or logs), with an average accuracy of 0.68 on Online Boutique, compared to 0.39 for the second-best method (BARO).
Real-World Performance:
- Tested on 10 real-world failures from a major internet provider. TORAI achieved 100% accuracy in AC@3 (ranking the true root cause in the top 3 recommendations) and an Avg@5 of 0.88, outperforming all baselines.
Robustness to Blind Spots:
- Sensitivity analysis showed that TORAI maintains high accuracy even as the percentage of blind spots increases from 0% to 100%. Even with 100% blind spots (no traces), TORAI maintained 80% AC@1 on Online Boutique and 74% AC@1 on Train Ticket, proving its reliance on metrics and logs is sufficient.
Efficiency:
- TORAI is highly efficient, completing RCA in ~12–20 seconds even for large systems (64 services). In contrast, some baselines (e.g., CausalRCA) took hours or exceeded time limits on larger datasets.

5. Significance

Practical Applicability: TORAI solves a critical real-world problem: diagnosing failures in systems where full observability (traces) is impossible or too costly to implement. It allows operators to leverage existing metrics and logs effectively.
Unsupervised Nature: By eliminating the need for labeled data, TORAI is immediately deployable in dynamic production environments where historical fault data is unavailable.
Non-Intrusive: It avoids the heavy engineering burden of instrumenting every service or modifying log formats, making it suitable for systems with third-party or black-box components.
Granularity: It provides not just the faulty service but the specific indicator (e.g., "CPU overload" or "Specific Error Log"), accelerating the mean time to resolution (MTTR).

In conclusion, TORAI represents a paradigm shift in microservice RCA, moving away from the rigid requirement of complete call graphs toward a flexible, severity-driven, and causal inference-based approach that thrives in the presence of blind spots.