Technical Summary: Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis

Problem Definition

Root Cause Analysis (RCA) is critical for maintaining reliability in modern cloud-native, microservice-based systems. However, the distributed and interdependent nature of these systems complicates diagnosis, particularly for multi-hop fault propagation where symptoms appear distant from their true causes. While Large Language Models (LLMs) offer promise for automating RCA through general reasoning and tool-use capabilities, their practical utility remains unclear.

Existing approaches suffer from two primary methodological gaps:

1. Obscured Reasoning: Many frameworks embed LLMs within complex multi-agent pipelines or heavily engineered workflows. This entanglement makes it difficult to isolate whether failures stem from the LLM's reasoning capabilities or from peripheral design choices (e.g., inter-agent dependencies, system-specific heuristics).
2. Lack of Reasoning Evaluation: Prior work typically evaluates only final output accuracy or relies on coarse-grained human judgments. There is no systematic, reproducible, and automated evaluation of the quality of intermediate reasoning traces, which is essential for understanding multi-hop fault propagation.

Consequently, there is a lack of empirical understanding regarding the isolated reasoning capabilities of LLMs in RCA, the factors influencing their performance, and the specific reasoning failures they exhibit.

Methodology

The authors present a controlled empirical evaluation designed to isolate LLM reasoning behavior from confounding factors. The study utilizes a simplified experimental setting that foregrounds the LLM while maintaining the complexity of multi-hop reasoning.

Experimental Framework

• Datasets: Two real-world microservice datasets were used: GAIA (derived from the MicroSS application) and OpenRCA (derived from the Online Boutique application). These provide multi-modal telemetry data (logs, metrics, traces).
• Input Processing:
  • Alert Extraction: Raw telemetry is processed using feature fusion. Logs are parsed (Drain), traces are analyzed for anomalies (Isolation Forest), and metrics are flagged using the 3-sigma rule.
  • Alert Unification: Alerts are unified via time-based or element-based strategies to create a consistent input representation.
  • Knowledge Graph (KG): An explicit, typed KG is constructed from architectural documentation and manifests, encoding entities (services, hosts) and relationships (control flow, data flow, deployment). This serves as the ground truth for system topology.
• Agent Workflows: Six open-source LLMs (including Llama 3.2/3.3, Qwen 3, Command R+, and DeepSeek-R1) were evaluated under three distinct paradigms:
  1. Straight-Shot (Non-agentic): The LLM receives all inputs and returns a diagnosis in a single pass.
  2. ReAct: The LLM interleaves reasoning with tool calls (Thought-Action-Observation loop).
  3. Plan-and-Execute: The LLM generates a high-level plan, executes steps sequentially, and replans based on results.
• Tools: The LLMs were provided with deterministic, semantically minimalist tools for data inspection and graph traversal (e.g., retrieving $r$-hop neighborhoods). Direct access to the raw KG or code execution was restricted to ensure reasoning was the primary variable.

Evaluation Metrics

• Output Quality: Measured via Location Accuracy (LA), Type Accuracy (TA), Hypothesis Accuracy (HA), and Propagation Path Validity (PA) at top-$k$ ($k=1, 3$).
• Reasoning Quality: A novel LLM-as-a-Judge approach was employed to annotate inference traces. A taxonomy of 16 Reasoning Failures (RFs) was developed through open coding and iterative refinement (involving human and LLM annotation). Failures were categorized into:
  • Procedural: Issues in the decision-to-act loop (e.g., failure to update belief, repetition).
  • RCA-specific: Domain errors (e.g., metric interpretation, provenance confusion).
  • General: Logical fallacies, unsupported causal links, and internal contradictions.
• Scale: The study executed 48,000 simulated failure scenarios, totaling 228 days of execution time.

Key Results

1. Effectiveness of LLM Agents (RQ1)

• No One-Size-Fits-All: Model performance is task-specific. Qwen 3 (32B) achieved the highest overall hypothesis accuracy, while Llama 3.3 excelled at propagation path reconstruction. Smaller models (e.g., Llama 3.2, Qwen 3 4B) performed competitively in simple workflows but degraded significantly in complex agentic settings.
• Diminishing Returns of Agentic Workflows: Contrary to the hypothesis that agentic workflows improve performance, Straight-Shot often equaled or outperformed ReAct and Plan-and-Execute in LA, TA, and HA. For smaller models, increased workflow complexity led to performance drops, sometimes falling below random guessing.
• Path Validity: Agentic workflows did improve Propagation Path Validity (PA) for capable models (e.g., Llama 3.3), suggesting better grounding in system context, but this benefit was conditional on the model's ability to reliably plan and execute tool calls.
• Execution Errors: Small models suffered high execution failure rates (e.g., 87.3% for Qwen 3 4B in Plan-and-Execute) due to recursion limits and replanning errors, severely constraining end-to-end utility.

2. Sensitivity to Alert Modalities (RQ2)

• Metrics are Critical for Localization: Withholding metric data caused the largest drop in localization accuracy (LA), indicating LLMs rely heavily on metric anomalies to identify fault locations.
• Logs are Critical for Fault Typing: Excluding logs primarily degraded fault-type attribution (TA), suggesting textual alerts provide essential semantic cues for error classification.
• Traces as Distractions: Surprisingly, excluding traces often improved accuracy. The authors found that trace alerts were often noisy and voluminous, causing LLMs to focus on call relationships while ignoring other critical dependencies (e.g., deployment). Removing them led to more focused analysis grounded in the KG.

3. Reasoning Failures (RQ3)

• Pervasive Failures: Reasoning failures were widespread across all models.
• Procedural Instability: Procedural failures (e.g., failure to update belief, excessive speculation) varied significantly by model and workflow, with smaller models showing higher instability under agentic control.
• RCA-Specific Gaps: Failures related to domain knowledge (e.g., metric interpretation, provenance) were consistent across models, indicating a lack of inherent RCA-specific reasoning capabilities.
• General Weaknesses: General per-hypothesis failures (e.g., fabricated evidence, evidential insufficiency) had the highest average prevalence, suggesting that hypothesis claims are frequently weakly supported.

4. Impact of Failures on Correctness (RQ4)

• Negative Predictors: The presence of specific reasoning failures—particularly anchoring bias (RF-13), repetition/stalled progress (RF-12), arbitrary evidence selection (RF-07), and failure to update belief (RF-09)—was strongly associated with reduced correctness (Risk Difference < -0.15).
• Workflow Interaction: While agentic workflows showed smaller absolute risk differences per failure (suggesting iterative reasoning might dilute the impact of single errors), the co-occurrence of multiple failures and early error compounding in agentic traces contributed to net accuracy loss.
• Model Differences: "Thinking"-oriented models (e.g., Qwen) showed lower prevalence of cross-cutting failures compared to others, suggesting training style influences reasoning coherence more than workflow structure alone.

Significance and Contributions

The paper claims to be the first empirical investigation into the isolated reasoning capabilities of LLM-based RCA agents and their specific failure modes. Its primary contributions are:

1. A Controlled Evaluation Framework: A methodology that isolates LLM reasoning from framework complexity, enabling transparent assessment of reasoning capabilities independent of multi-agent entanglement.
2. A Reasoning Failure Taxonomy: A labeled taxonomy of 16 common RCA reasoning failures, categorized into procedural, RCA-specific, and general types, derived through a rigorous mixed-methods elicitation process.
3. Automated Reasoning Evaluation: An LLM-as-a-Judge system capable of assessing RCA reasoning quality automatically and reproducibly, validated by human reliability checks.
4. Empirical Insights: Extensive results (48,000 samples) quantifying the sensitivity of LLMs to input modalities and identifying specific reasoning failures that predict final correctness.

The authors conclude that while simplified setups do not trivialize the RCA task, current open-source LLMs struggle with multi-hop reasoning, particularly in complex agentic workflows. Progress in automated RCA will likely require improving specific reasoning capabilities (e.g., evidence sufficiency checks, hypothesis diversification) and providing explicit domain guidance, rather than relying solely on agentic structures. The work provides a foundation for developing RCA agents that are not only automated but also interpretable and aligned with the practical demands of fault diagnosis.