Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems

1. Problem Statement

The reliability of cloud-native systems relies heavily on Root Cause Analysis (RCA). While traditional AI for IT Operations (AIOps) models (e.g., VAEs, LSTMs) excel at detecting anomalies via statistical pattern recognition, they lack the semantic understanding to explain failures. The emergence of Large Language Models (LLMs) has shifted the paradigm toward Agentic RCA, where autonomous agents actively interact with systems to verify hypotheses rather than passively classifying data.

However, existing benchmarks fail to support this new paradigm due to three critical gaps:

1. Ecological Validity Gap (Static Artifacts): Current benchmarks (e.g., LogHub, RCAEval) rely on retrospective, static data dumps. They reduce agents to passive readers, failing to evaluate the critical skill of active tool use (e.g., kubectl) and information seeking in a live environment.
2. Reproducibility Gap (Dynamic Environments): Dynamic benchmarks (e.g., AIOpsLab) use live clusters but suffer from non-determinism (e.g., network jitter). This stochasticity makes rigorous A/B testing impossible and introduces experimental noise that obscures model performance.
3. Evaluation Gap (Outcome-Oriented Metrics): Existing metrics focus solely on the final diagnosis (Accuracy@1). This rewards "lucky guesses" or hallucinations where an agent arrives at the correct conclusion via an illogical path, failing to distinguish systematic reasoning from statistical probability.

2. Methodology: Cloud-OpsBench

The authors propose Cloud-OpsBench, a large-scale, reproducible benchmark for Agentic RCA on Kubernetes. Its core innovation is the State Snapshot Paradigm.

A. The State Snapshot Paradigm

Instead of relying on transient live clusters or static files, Cloud-OpsBench creates a deterministic Digital Twin of the cloud system at the precise moment of failure.

• Mechanism: The system freezes the complete operational context (control plane objects, metrics, logs, and configurations) into an immutable persistence layer.
• Interaction: Agents interact with this frozen state via a mocked interface that simulates standard tools (e.g., kubectl, prometheus).
• Benefit: This decouples state storage from tool interaction, guaranteeing 100% reproducibility and zero-latency execution while preserving the complexity of real-world tool usage.

B. Benchmark Construction

• Scale: The benchmark contains 452 distinct fault cases spanning 40 root cause types across the full Kubernetes stack (7 categories: Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, Infrastructure).
• Generation Pipeline:
  1. Knowledge Base Construction: Aggregates data from Kubernetes docs, Stack Overflow, and academic papers. Uses an LLM (Gemini 3 Pro) to extract structured metadata and deployment artifacts.
  2. Automatic Fault Generation: A Multi-Agent System (MAS) orchestrates a closed loop:
     • Generator Agent: Designs the fault injection plan.
     • Executor Agent: Injects the fault into a live cluster.
     • Verifier Agent: Validates the fault symptoms. If the fault is masked by the system (e.g., auto-healing), the loop refines the injection parameters.
  3. Snapshotting: Once verified, the system freezes the state and pre-computes responses for all plausible diagnostic tool invocations (approx. 487 tool calls per case) to create the deterministic environment.

C. Task Formulation & Evaluation

The task is formalized as a trajectory-based decision process. Evaluation moves beyond binary success/failure to Process-Oriented Metrics:

• Outcome Metrics: Top-k Accuracy ($A@k$) and Task Completion Rate (TCR).
• Process Metrics:
  • Trajectory Alignment: Exact Match, Any-Order Match, and In-Order Match against expert reference trajectories.
  • Efficiency: Steps taken, Mean Time to Identify (MTTI).
  • Robustness: Invalid Action Count (IAC), Redundant Action Rate (RAR), and Zero-Tool Diagnosis Rate (ZTDR).

3. Key Contributions

1. A Process-Centric Evaluation Paradigm: The first white-box benchmark that quantifies how an agent verifies a hypothesis, not just what it concludes. It establishes that reasoning alignment is as critical as outcome accuracy.
2. An Enabling Data Engine: Demonstrates that procedural demonstrations (from the benchmark) can bootstrap Supervised Fine-Tuning (SFT) for Small Language Models (SLMs), allowing 14B-parameter models to rival proprietary giants.
3. A Safe RL Sandbox: Provides a high-fidelity, deterministic environment suitable for Reinforcement Learning (RL) and policy optimization, transforming high-risk cloud operations into a safe learning ground.

4. Key Results & Findings

The authors evaluated 7 LLMs (including GPT-5, DeepSeek-V3.2, Qwen-235B, and smaller SLMs) on Cloud-OpsBench.

• Exploration vs. Efficiency: There is a positive correlation between diagnostic accuracy and the depth of exploration. Models that perform redundant verification (e.g., DeepSeek-V3.2) achieve higher accuracy than those that rush to conclusions (e.g., GPT-4o), challenging the "efficiency-first" orthodoxy.
• The "Redundancy Paradox": High-performing agents exhibit higher Redundant Action Rates (RAR). Redundant checks act as cognitive self-correction mechanisms to filter out hallucinations and confirm transient states.
• SLM Bottlenecks: Small Language Models (e.g., Qwen-14B) suffer from Syntactic Fragility. While they can identify the correct tools (high Relevance), they frequently generate invalid API payloads (high IAC), causing reasoning loops.
• Knowledge Efficacy:
  • Procedural > Declarative: In-Context Learning (ICL) using historical diagnostic traces significantly outperforms Retrieval-Augmented Generation (RAG) using static documentation. Agents learn better by mimicking actions than by reading manuals.
  • SLM Resurrection: ICL allows SLMs to bypass syntactic barriers, boosting Qwen-14B's accuracy from 0.34 to 0.71, effectively matching the performance of larger models.
• Failure Modes:
  • Premature Convergence: Some models (e.g., Claude-4-Sonnet) skip investigation and hallucinate answers based on generic alerts (High ZTDR).
  • Schema Hallucination: Mid-tier models often invent non-existent parameters for valid tools.

5. Significance

• Redefining SRE Evaluation: Cloud-OpsBench shifts the industry standard from outcome-based metrics to process-centric evaluation, ensuring that agents are evaluated on their ability to reason logically and gather evidence, not just guess correctly.
• Democratizing Research: By eliminating the need for expensive, unstable live clusters, it lowers the barrier to entry for AIOps research, allowing academic iteration on a standard laptop in seconds.
• Future Directions: The benchmark serves as a foundation for:
  • Multi-Agent Architectures: Specializing agents (e.g., a robust planner + efficient SLM executors).
  • RL for SRE: Training agents via policy optimization to learn optimal exploration strategies.
  • Self-Evolving Systems: Using the benchmark as a data engine to continuously fine-tune and optimize agent instructions.

In summary, Cloud-OpsBench provides the necessary infrastructure to transition Agentic RCA from fragile, stochastic experiments to a rigorous, reproducible scientific discipline.