Technical Summary: SREGym – A Live Benchmark for AI SRE Agents

1. Problem Statement

The paper addresses the critical gap in evaluating AI agents for Site Reliability Engineering (SRE). While AI agents are increasingly deployed to diagnose and mitigate production failures, existing benchmarks are insufficient for three primary reasons:

• Oversimplification: Current benchmarks (e.g., static Q&A datasets, anomaly detection tasks) fail to capture the dynamic, multi-step nature of real-world incident response. They often evaluate agents on static data or isolated symptoms rather than end-to-end resolution.
• Lack of Fidelity: Existing live benchmarks (e.g., AIOpsLab, ITBench) simulate failures but lack the complexity of production environments. They typically focus on application-layer issues, ignore "ambient noise" (unrelated transient events), and fail to model complex failure modes like metastable states or correlated failures.
• Extensibility Issues: Many benchmarks rely on bespoke, hard-coded scripts (e.g., Ansible) that make it difficult to compose new scenarios, coordinate distributed events, or scale the problem space.

Furthermore, the rise of AI-generated code introduces new reliability challenges, with reports indicating AI code introduces significantly more defects and escapes testing more frequently than human-written code. This necessitates robust agentic SRE capabilities that go beyond coding to include real-time mitigation.

2. Methodology: The SREGym Framework

The authors present SREGym, a high-fidelity, live benchmark built atop real-world cloud-native system stacks (Kubernetes). Unlike static datasets, SREGym provides a live environment where agents interact with running systems via a standardized interface.

Core Architecture

• System Environment: SREGym deploys a catalog of cloud-native applications (e.g., DeathStarBench, TrainTicket, Astronomy Shop) and backend systems (MongoDB, TiDB, Kafka) managed by Kubernetes operators.
• Agent Interface: Agents interact via the Model Context Protocol (MCP), accessing tools for metrics (Prometheus), logs (Loki), traces (Jaeger), and cluster control (kubectl).
• Fault and Noise Injection: The framework uses a modular set of injectors to simulate:
  • Faults: Root causes across the stack, including hardware faults (via eBPF), OS kernel faults, misconfigurations, code bugs, and operator errors.
  • Noises: Transient, self-recovering disturbances (e.g., pod crashes, dropped packets) that are unrelated to the target failure, designed to test an agent's ability to distinguish signal from noise.
• Problem Definition: A problem is defined as a tuple $P = (E, I, F, O)$, comprising the environment, agent interface, faults/noises, and oracles.
• Evaluation Oracles:
  • Diagnosis Oracle: Uses an LLM-as-a-Judge approach with a structured checklist (9 Yes/No questions across three dimensions: Fault Localization, Fault Characterization, and Failure Scope) to evaluate natural language diagnoses. This avoids brittle string-matching and allows for nuanced evaluation.
  • Mitigation Oracle: Programmatically verifies if the target fault is resolved and the system has returned to a healthy state, using both client-side (request success rates) and system-side observability.

Key Design Principles

1. Simulating Faults, Not Symptoms: The system injects root causes (e.g., a missing environment variable) rather than just symptoms (e.g., a crashed pod), requiring agents to reason about the underlying defect.
2. Composability: A Python-based runtime orchestrates fault and noise injectors, enabling complex scenarios like metastable failures (self-sustaining degradation) and concurrent/correlated failures.
3. Extensibility: The framework is designed to be "push-button" for problem creation, allowing researchers to mutate existing scenarios or create new ones via APIs.

3. Key Contributions

• SREGym Benchmark: A live, open-source benchmark containing 90 realistic, challenging SRE problems that span hardware, OS, and application layers.
• High-Fidelity Failure Modeling: Introduction of complex failure modes previously unaddressed in benchmarks, including:
  • Metastable Failures: Systems that degrade in response to transient events and fail to recover without specific intervention.
  • Concurrent Failures: Multiple independent faults occurring simultaneously.
  • Correlated Failures: Multiple components failing due to a shared root cause.
• Ambient Noise Simulation: The ability to inject transient, low-impact disturbances that challenge agents to filter out irrelevant data.
• Robust Evaluation Protocol: A checklist-based LLM evaluation method that achieves high agreement with human experts ($\kappa = 0.90$) while scaling to large datasets.

4. Experimental Results

The authors evaluated three agents: Stratus (a specialized SRE agent), Claude Code, and OpenAI Codex, powered by frontier models (Sonnet-4.6, GPT-5.4, Kimi K2.5).

Overall Performance

• Success Rates: Diagnosis success rates ranged from 38.9% to 72.6%, and mitigation success rates from 57.3% to 78.5%.
• End-to-End (E2E) Performance: The E2E success rate (correct diagnosis + correct mitigation) varied significantly, with differences of up to 40% between agent-model pairs.
• Impact of Noise: The presence of noise consistently reduced diagnosis success rates across all agents. Mitigation was more robust but still impacted.

Key Findings

1. Layer-Specific Weaknesses: Agents performed well on application-layer issues but struggled significantly with failures rooted in lower layers (OS, hardware) or complex patterns. For example, in hardware fault scenarios (disk sector errors), agents failed to identify the hardware root cause, often misattributing it to application logic.
2. Metastable Failure Challenges: Agents could identify application-level triggers but frequently failed to connect them to infrastructure constraints (e.g., resource quotas) required to resolve the metastable state.
3. Greedy Diagnosis: A recurring failure mode was the "greedy approach," where agents latched onto the first plausible anomaly (often a noise or unrelated issue) and failed to investigate further, leading to incorrect root cause attribution.
4. Token Efficiency: Specialized SRE agents (Stratus) were significantly more token-efficient (using ~3x fewer tokens) than coding agents (Claude Code, Codex) by preprocessing observability data, without sacrificing E2E success rates.
5. Diagnosis-Mitigation Correlation: Correct diagnosis strongly correlated with successful mitigation ($P(M|D) \approx 0.69\text{--}0.88$). However, agents could sometimes mitigate failures even with incorrect diagnoses by iteratively observing and tuning the system, though this was less efficient.

5. Significance and Claims

The paper positions SREGym as a foundational step toward production-ready agentic SRE.

• Realism: By simulating the "noisy" and "eventful" nature of production environments, SREGym provides a more accurate measure of an agent's readiness for real-world deployment than static benchmarks.
• Community Resource: As an open-source, extensible framework, it aims to accelerate research by providing a common ground for evaluating diverse agent architectures and models.
• Limitations: The authors modestly acknowledge limitations, including the smaller scale of deployed applications compared to hyperscale production systems, simplified noise models, and the current evaluation of only a few agent-model pairs. They emphasize that high scores on SREGym are a necessary but not sufficient condition for production safety.

In conclusion, SREGym demonstrates that while frontier AI agents show promise in SRE tasks, significant gaps remain in handling high-fidelity, multi-layered, and noisy failure scenarios, highlighting the need for continued development in agentic reliability engineering.