🤖 machine learning

On Randomness in Agentic Evals

This paper demonstrates that single-run evaluations of agentic systems suffer from substantial variance due to early trajectory divergence, arguing that reliable assessment of genuine progress requires multiple runs, statistical power analysis, and broader performance metrics to distinguish true algorithmic improvements from evaluation noise.

Original authors: Bjarni Haukur Bjarnason, André Silva, Martin Monperrus

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bjarni Haukur Bjarnason, André Silva, Martin Monperrus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. The goal is to see which chef can make the perfect soufflé.

In the world of AI, these "chefs" are Agentic Systems (smart computer programs that can use tools, like a web browser or a code editor, to solve problems). The "soufflé" is a task, like fixing a bug in a software program.

For a long time, the judges (researchers) have been doing this: They let each chef try the recipe exactly once. If the soufflé rises, they give a "Pass." If it falls, they give a "Fail." They then calculate the score based on that single attempt.

This paper says: "Stop doing that! It's a terrible way to judge."

Here is the breakdown of why, using simple analogies:

1. The "One-Try" Lottery

The authors ran a massive experiment. They didn't just let the chefs try once; they let them try 10 times for every single task. They used three different "chefs" (AI models) and two different "kitchens" (software environments).

The Result? The scores were all over the place.

Sometimes a chef would get a 30% success rate.
The very next time, with the exact same instructions and the exact same chef, they might get a 36% success rate.

The Analogy: Imagine a basketball player shooting free throws. If you only watch them shoot one ball, and it goes in, you might think they are a 100% shooter. But if you watch them shoot 10 times, you might see they actually make 6 out of 10.
In AI research, if you only watch one shot, you might think a new AI is "better" just because it got lucky on that one specific day. The paper found that a reported "improvement" of 2 or 3 percentage points is often just noise (luck), not a real upgrade in skill.

2. The "Butterfly Effect" in the Kitchen

Why does the same chef get different results? The authors looked at the "thought process" of the AI, token by token (word by word).

The Discovery: The AI's path diverges (splits) extremely early.

Imagine two identical twins starting a journey. They walk the same path for the first 10 steps.
At step 11, Twin A thinks, "I'll turn left," and Twin B thinks, "I'll turn right."
Because they are walking in a maze, that tiny decision at step 11 leads them to completely different destinations.

The Paper's Finding: Even when the AI is set to be "deterministic" (meaning it should act exactly the same every time, like a robot), tiny, invisible differences in the computer's hardware or math cause the AI to make a tiny decision change within the first 1% of its thinking. This small change cascades, leading to a completely different solution strategy. One path leads to success; the other leads to failure.

3. The "Optimist vs. Pessimist" View

The paper suggests we stop looking at just one number (Pass@1) and start looking at a range.

The Optimist (Pass@k): "If we let the AI try 5 times, what are the odds it solves the problem at least once?" This shows the AI's potential.
The Pessimist (Pass^k): "If we let the AI try 5 times, what are the odds it solves the problem every single time?" This shows the AI's reliability.

The Analogy:

Optimist: "This car can drive 100mph if the road is perfect and the driver is lucky!"
Pessimist: "This car can only drive 20mph without stalling, no matter how many times you try."

The paper found that for some AI models, the gap between "Best Case" and "Worst Case" was huge (up to 25 percentage points). This means the AI isn't consistently smart; it's just sometimes lucky.

4. What Should We Do? (The New Rules)

The authors propose three simple rules to fix this mess:

Don't trust a single try: If you want to know if a new AI is better, don't just run it once. Run it many times (like 10 or more) and take the average.
Do the math first: Before you run your experiment, calculate how many times you need to run it to be sure your results aren't just luck. (The paper provides a table for this).
Report the whole story: Don't just say "We got 30%." Say "We got 30% on average, but it ranged from 25% to 35%." This tells the truth about how reliable the AI really is.

The Bottom Line

Right now, the AI field is full of "fake news" about progress. A new model might claim to be "3% better" than the old one, but that might just be because the researchers got lucky with their single test run.

The paper is a wake-up call: To truly know if AI is getting smarter, we need to stop gambling on single tries and start measuring the consistency and variance of these systems. Otherwise, we are just building a leaderboard based on luck.

1. Problem Statement

Current evaluation practices for agentic AI systems rely heavily on single-run pass@1 scores (the probability of solving a task in one attempt). Researchers and leaderboards typically report the success rate of a single execution per task, assuming this provides a reliable estimate of model performance.

The paper argues this methodology is methodologically unsound because:

High Variance: A single binary outcome per task yields a high-variance estimate of the true success probability.
Stochasticity: Even with temperature $>0$ , sampling introduces randomness. Crucially, even at temperature 0 (theoretically deterministic), non-determinism persists due to inference engine optimizations, floating-point precision, and hardware effects.
Misleading Progress: Small reported improvements (e.g., 2–3 percentage points) may simply reflect evaluation noise rather than genuine algorithmic advancement, leading to flawed deployment and research decisions.

2. Methodology

The authors conducted a large-scale empirical study to quantify and analyze this variance.

Dataset & Scale:
- Benchmark: SWE-Bench-Verified (500 software engineering tasks).
- Volume: 60,000 total agent trajectories generated across 120 experimental runs.
- Resources: 25.58 billion tokens processed and 1.88 million tool calls executed.
Experimental Setup:
- Models: Three distinct models were tested:
  1. Qwen/Qwen3-32B (General purpose).
  2. agentica-org/DeepSWE-preview (Fine-tuned for agentic coding).
  3. mistralai/Devstral-2-123B-Instruct (Large open-weights model).
- Scaffolds: Two different agent frameworks were used to rule out implementation artifacts:
  1. nano-agent (Minimal, custom scaffold).
  2. R2E-Gym (Feature-rich, used in DeepSWE training).
- Conditions: Each configuration was run 10 times independently at various temperatures (0.0, 0.2, 0.6, 1.0).
Metrics:
- Single-run resolution rate ( $r_j$ ): Success rate of a specific run.
- pass@k: Probability that at least one of $k$ attempts succeeds (Optimistic bound).
- passˆk: Probability that all $k$ attempts succeed (Pessimistic bound/Consistency).
- First Token Divergence ( $\tau_{div}$ ): The position where two trajectories first differ, analyzed at both absolute and relative (percentage of total trajectory) levels.

3. Key Results

A. Substantial Variance in Single-Run Scores

Range of Variability: Single-run pass@1 estimates varied by 2.2 to 6.0 percentage points depending on which specific run was selected.
Standard Deviation: Even at temperature 0, standard deviations exceeded 1.5 percentage points.
- Example: DeepSWE-preview on nano-agent at temp 0.0 had a mean of 20.4% but ranged from 18.2% to 21.4%.
Implication: A reported improvement of 3% could easily be a statistical fluctuation rather than a real gain.

B. Performance Bounds (pass@k vs. passˆk)

The study revealed a massive gap between the "best-case" and "worst-case" scenarios for the same model:

Optimistic vs. Pessimistic: The gap between pass@5 (best of 5) and passˆ5 (all 5 succeed) reached up to 24.9 percentage points.
Interpretation: This indicates that success is heavily dependent on stochastic exploration. An agent might solve a task 52.9% of the time with retries (pass@5) but only solve it consistently 15.5% of the time (passˆ5).

C. Mechanism of Divergence

Early Divergence: Trajectories diverge extremely early, often within the first 1% of tokens (sometimes as early as the 5th token).
Butterfly Effect: A minor difference in the initial reasoning tokens (e.g., "Let me search" vs. "Let me check") alters the context for subsequent autoregressive steps. This cascades into different tool calls, observations, and ultimately opposite outcomes (success vs. failure).
Temperature Impact: While temperature 0 delays divergence slightly, it does not eliminate it. Non-determinism in inference engines ensures that even "deterministic" settings produce divergent trajectories.

D. Statistical Power Analysis

The authors calculated the number of runs required to detect specific improvements with 80% statistical power:

Detecting a 1% improvement requires 36 runs (at median variance).
Detecting a 2% improvement requires 9 runs.
Current single-run practices are statistically insufficient for detecting anything less than massive performance jumps.

4. Key Contributions

Empirical Quantification: Provided the first large-scale evidence (60k trajectories) that agentic evals suffer from high variance, even under "deterministic" settings.
Divergence Analysis: Identified that trajectory divergence happens in the first few percent of tokens, explaining how small stochastic perturbations lead to fundamentally different solution strategies.
Performance Envelope Characterization: Demonstrated the utility of pass@k and passˆk to bound model capabilities, revealing that single-run pass@1 captures neither the full potential nor the reliability of an agent.
Actionable Recommendations: Proposed a new standard for evaluation to distinguish signal from noise.

5. Significance and Recommendations

The paper argues that the current reliance on single-run scores is creating a "false sense of progress" in the field. To ensure scientific rigor and reliable engineering decisions, the authors recommend:

Multiple Runs: Estimate pass@1 from multiple independent runs (e.g., $n \ge 10$ for small improvements) rather than a single execution.
Statistical Power Analysis: Researchers should calculate the required number of runs ( $n$ ) based on the expected effect size and desired significance level (e.g., $p < 0.05$ ).
Report Performance Bounds: Move beyond single metrics. Report:
- pass@1: Expected performance.
- pass@k ( $k>1$ ): Optimistic bound (potential with retries).
- passˆk ( $k>1$ ): Pessimistic bound (consistency/robustness).

Conclusion: Randomness is not a minor perturbation but a fundamental component of agentic performance. Ignoring it leads to unreliable leaderboards and wasted resources. The field must adopt multi-run, statistically rigorous evaluation protocols to distinguish genuine algorithmic progress from evaluation noise.