💬 NLP

ORCA-bench: How Ready Are Language Model Agents for Oncall?

The paper introduces ORCA-bench, a production-fidelity benchmark for evaluating language model agents on oncall root cause analysis tasks, revealing that even the most advanced frontier models currently achieve low accuracy (25.3% on medium difficulty) and struggle with hallucinations, indicating a significant gap before they can be safely entrusted with real-world production reliability.

Original authors: Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, Raaz Dwivedi

Published 2026-07-31

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, Raaz Dwivedi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, high-tech spaceship cruising through the galaxy. Suddenly, the ship's computer starts acting weird: the lights flicker, the gravity is off, and the coffee machine is spitting out sparks. You don't know exactly what's wrong, but you know the ship is in trouble. In the real world, these spaceships are the websites and apps we use every day, and the "captains" are special engineers called Site Reliability Engineers (SREs). Their job is to figure out why things broke, often starting with a vague complaint like "the checkout button isn't working" or "the site is slow." They have to dig through mountains of digital clues—like traffic logs, error messages, and code—to find the single broken part before the whole ship crashes.

For a long time, people have been trying to teach computers to do this detective work using Artificial Intelligence (AI), specifically Large Language Models (LLMs). These are the same smart computers that can write stories, answer questions, and even help write code. The big question everyone is asking is: "Are these AI detectives ready to take the wheel and fix our real-world systems when things go wrong?" It's a high-stakes game because if an AI guesses wrong, it could make a small glitch turn into a massive disaster.

This paper, titled ORCA-BENCH, sets up a giant, realistic training ground to test exactly that. The researchers built a fake but incredibly detailed online store (called the "Astronomy Shop") that runs for six days, generating a massive amount of digital noise, just like a real busy website. They then created 1,079 different "mystery incidents" where they secretly broke things in the system and asked five of the smartest AI agents available to solve the case. They gave the AI the same tools a human engineer would use: access to the live data streams, the source code, and a vague user complaint.

The results? The AI detectives are still very much in training. Even the best AI models could only solve about 25% of the "medium difficulty" cases correctly, and they managed just 10% on the hardest cases. To put that in perspective, if a human engineer got a 10% score on a test, they would be fired; for these AIs, it's the current state of the art. The paper found that when the AI couldn't see the source code, their performance got even worse. Perhaps most worryingly, the AI often "hallucinated," meaning they confidently invented fake reasons for the problems that didn't exist at all. In one case, an AI blamed a browser crash in a test tool for a user's shopping cart failing, completely missing the real culprit.

The authors conclude that while these AI agents are impressive at writing code, they are not yet ready to be trusted with the safety of real, live systems. The gap between what the AI can do in a controlled test and what is needed to keep the internet running safely is still huge. The paper suggests that before we let AI take over the "on-call" shift, we need to do a lot more engineering work to make them reliable, because right now, they are more likely to guess wrong than to solve the mystery.

Technical Summary: ORCA-BENCH: How Ready Are Language Model Agents for Oncall?

Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in writing, patching, and searching code within static repositories, they have not yet proven capable of managing the reliability of live production systems. Root Cause Analysis (RCA) in a Site Reliability Engineering (SRE) context differs fundamentally from traditional software engineering benchmarks. In production, an oncall engineer receives an ambiguous, natural-language report (e.g., "users are having issues") hours after an incident begins. They must reason over a noisy, dynamic stack of metrics, logs, and traces from a live microservice mesh to identify the specific feature, deployment, or component failure. Existing benchmarks fail to capture this complexity because they often rely on synthetic faults, lack access to raw telemetry interfaces or source code, or do not systematically vary the specificity of user reports and the time-to-detection (TTD).

Methodology

The authors introduce ORCA-bench, a benchmark designed to evaluate general-purpose coding agents in a production-fidelity oncall setting. The methodology consists of four core components:

1. The Environment (The Full Evidence Stack)

ORCA-bench utilizes a live, OpenTelemetry-instrumented microservice system (the "Astronomy Shop") running six days of continuous simulated user load. Unlike prior works that provide raw CSVs or custom tools, ORCA-bench exposes the full evidence stack through realistic interfaces:

Metrics: Accessed via Prometheus.
Logs: Accessed via OpenSearch.
Traces: Accessed via Jaeger.
Interface: All telemetry is queried through a Grafana API.
Source Code: Full access to the system's source code is provided via a terminal.
The dataset comprises 50 GB of data across six days, covering 19 microservices written in 13 different languages.

2. Task Generation and Schedule

The benchmark generates 1,079 RCA tasks by toggling 11 real feature flags rather than injecting synthetic faults. The tasks are structured around a six-day schedule with 13 scenarios, including isolated, independent, conflicting, cascading, and sequential fault types.

Variable Specificity: Tasks vary in "issue specificity" (Easy, Medium, Hard), ranging from precise error messages to vague reports like "users are reporting site issues."
Time-to-Detection (TTD): The time between incident onset and investigation start varies from 15 minutes to 24 hours.
Report Time: The time the user reports the issue is parameterized (exact, exact range, broad range) to simulate real-world ambiguity.
Control Tasks: 195 tasks are "control" tasks where no incident is active, testing the agent's ability to correctly identify the absence of a problem.

3. Ground Truth Curation

Ground truth is curated and signed off by expert SREs.

Symptoms: Frontend symptoms are verified by replaying feature flags in a browser. Telemetry symptoms (metrics, logs, traces) are collected via a semi-automated workflow validated by two expert SREs.
Root Causes: For each task, the ground truth is defined as the set of plausible root causes (feature flags active during the window) that could explain the user-facing issue.
Verification: A subset of 40 tasks ("ORCA-bench Verified") has every ground-truth label and per-model score manually confirmed by humans.

4. Evaluation Framework

The evaluation uses an LLM-as-a-judge (GPT-5.4) to score agent reports, with human re-scoring on the Verified subset to ensure alignment (Cohen's $\kappa_w = 0.90$ ).

Scoring Rubric: Reports are graded on a 0–3 scale per plausible root cause, assessing whether the agent confirmed symptoms, identified specific telemetry signals, and correctly named the mechanism.
Metrics:
- RCA Accuracy: The fraction of tasks where the agent named every listed plausible root cause.
- RCA Depth: The mean score (0–3) across all rubrics, indicating how far the agent progressed toward the root cause.
- Hallucination Rate: The fraction of non-empty reports where the agent named a root cause matching none of the plausible causes.

Key Results

The paper evaluates five frontier agents (Opus 4.7, Sonnet 4.6, GPT-5.5, GLM-5, DeepSeek-V4-Pro) on the benchmark.

Low Overall Performance: Even the best-performing models struggle. On Medium-difficulty tasks (the realistic-input setting), the best RCA Accuracy is 25.3%. On Hard tasks, accuracy drops to 10.0%.
High Hallucination Rates: Agents frequently hallucinate implausible root causes. The hallucination rate ranges from 7% (DeepSeek-V4-Pro) to 40% (GLM-5).
Impact of Context Removal: Removing source-code access degrades performance across all metrics. RCA Accuracy drops by 9–16 percentage points, and hallucination rates spike significantly.
Telemetry Retrieval Issues: Agents struggle to effectively query telemetry. Up to 40% of telemetry calls either error out or return empty results. GPT-5.5 is the most efficient, while others often fail to retrieve necessary data.
Failure Modes: Agents tend to get distracted by background noise (loud but irrelevant symptoms) and struggle to exhaustively find all root causes when multiple issues occur concurrently. They also fail to consistently quantify severity increases or attribute them to distinct events.
Claude Fable 5: On a subset of tasks, Claude Fable 5 achieved higher scores (58.2% depth, 40.6% accuracy on Verified tasks) but still fell short of perfect reliability, with gains primarily seen in Medium tasks rather than Easy ones.

Significance and Claims

The authors position ORCA-bench as the first SRE benchmark to expose language model agents to a live, OpenTelemetry-instrumented system end-to-end, combining real telemetry interfaces, raw signals, and source code under production-style ambiguity.

The paper's primary claim is that frontier coding agents are not yet ready for oncall. The reported performance gaps (e.g., 25.3% accuracy on Medium tasks) are presented as a lower bound on the engineering investment required before agents can be safely entrusted with production reliability. The authors argue that real production systems are orders of magnitude larger, more dynamic, and more idiosyncratic than the 50 GB testbed used here. Furthermore, the simplifications in the benchmark (fixed codebase, isolated tasks, no action loop) mean the actual gap in real-world scenarios is likely even wider.

The work highlights that current agents lack the ability to reason over noisy, multi-modal telemetry data in conjunction with source code to handle ambiguous, time-delayed incident reports, a critical prerequisite for autonomous system reliability.