Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy

This paper demonstrates that while higher behavioral consistency in LLM-based agents generally correlates with improved accuracy on complex tasks like SWE-bench, consistency ultimately amplifies existing interpretations—whether correct or incorrect—meaning that interpretation accuracy is a more critical factor for production reliability than execution consistency alone.

The Gist

Imagine you hire three different mechanics to fix a very specific, tricky problem in your car's engine. You give them the exact same broken car and the same instructions. You ask them to try fixing it five times each.

This paper is like a report card on how those mechanics behaved. The researchers wanted to know: If you ask an AI (a "smart computer program") to do a task five times, will it do it the same way every time? And more importantly, does doing it the same way mean it's doing a good job?

Here is the breakdown of what they found, using simple analogies:

1. The Three Mechanics (The Models)

The researchers tested three different AI "mechanics" on fixing software bugs (like fixing a broken app):

• Claude (The Meticulous Master): This AI is slow, careful, and reads the manual thoroughly.
• GPT-5 (The Speed Demon): This AI is incredibly fast, tries things quickly, and moves on.
• Llama (The Wild Card): This AI is a bit chaotic, sometimes guessing wildly, and often gets lost.

2. The Big Discovery: "Consistency is a Double-Edged Sword"

The most important lesson from the paper is this: Being consistent doesn't mean you are right. It just means you are reliably doing whatever you decided to do.

Think of it like a GPS navigation app:

• Scenario A (The Good Consistency): You tell the GPS to go to the beach. It calculates the route, and every time you ask, it gives you the exact same perfect route. Result: You get to the beach every time.
• Scenario B (The Bad Consistency): You tell the GPS to go to the beach, but it thinks the beach is actually a swamp. Because it is so confident and consistent, every single time you ask, it drives you to the swamp. It never changes its mind.

The paper found that the "Meticulous Master" (Claude) was the most consistent. However, 71% of the times it failed, it failed in the exact same way every single time. It got stuck on a wrong idea and refused to let go. It was "consistently wrong."

3. The Trade-Off: Speed vs. Reliability

The researchers noticed a funny triangle of trade-offs:

• The Speed Demon (GPT-5): It was 4.7 times faster than the Master. It finished the job in a flash. But because it rushed, it was less accurate and its behavior was all over the place (sometimes it fixed it, sometimes it broke it, sometimes it took a weird path).
• The Meticulous Master (Claude): It took a long time (lots of steps), but it was very steady. When it understood the problem, it solved it perfectly every time. When it misunderstood, it failed perfectly every time.
• The Wild Card (Llama): It was the least consistent and the least accurate. It was like a mechanic who tries a new tool every time you ask, often making things worse.

4. The "First Step" Trap

You might think, "If they all start by looking at the same file, they should be consistent."
The paper found that starting the same way doesn't guarantee staying the same way.

• The Speed Demon and the Master both started their work at almost the exact same moment (Step 3).
• But after that, the Master stayed on a straight, narrow path. The Speed Demon started wandering off in different directions every time.
• Lesson: It's not about when they start disagreeing; it's about how well they stick to a plan after they start.

5. Why This Matters for the Real World

If you are building a robot or an AI assistant to work in a hospital or a bank, you need to know this:

• Don't just trust consistency. If an AI is very consistent, it might just be confidently wrong.
• The real problem is "Understanding." The biggest reason these AIs failed wasn't that they made a typo or forgot a step. It was that they misunderstood the job description in the first place.
• The Fix: We need to teach AIs to double-check their understanding of the problem, not just to be faster or more consistent in their typing.

Summary in One Sentence

Being a reliable robot is great, but if that robot is confidently wrong, it's actually more dangerous than a robot that is a little chaotic but occasionally gets the right answer.

The paper tells us that for AI to be truly useful, we need to focus on making sure they understand the task correctly first, because once they understand it, their consistency will naturally make them excellent workers.

Technical Summary

1. Problem Statement

As Large Language Model (LLM) agents are deployed in production systems (e.g., code assistants, autonomous research tools), behavioral consistency—the ability of an agent to produce similar action sequences when given identical tasks—has become a critical reliability metric. While prior research has established that LLMs exhibit variance in simple reasoning tasks, the relationship between consistency, task complexity, and accuracy in complex, multi-step agent trajectories remains underexplored.

The core questions addressed are:

• Do consistency challenges scale with the size of the action space in complex tasks?
• Is behavioral consistency a prerequisite for accuracy, or does it merely amplify whatever strategy the agent chooses (correct or incorrect)?
• How do different models trade off between speed, accuracy, and consistency?

2. Methodology

The authors conducted a rigorous empirical study using the SWE-bench Verified benchmark, which requires agents to resolve real GitHub issues through multi-step code modifications.

• Dataset: 10 diverse tasks from the astropy repository, varying in bug type, fix complexity (median 4.5 lines), and file count.
• Models Evaluated: Three models representing different capability tiers:
  1. Claude 4.5 Sonnet: Frontier model with strong coding capabilities.
  2. GPT-5: Frontier model with strong reasoning capabilities.
  3. Llama-3.1-70B-Instruct: Open-weights model widely used in production.
• Experimental Protocol:
  • Runs: 5 independent trials per model per task (50 runs per model, 150 total trajectories).
  • Parameters: Temperature set to 0.5 (moderate stochasticity), max steps 250, isolated Docker containers for each run.
  • Framework: mini-SWE-agent, a minimal bash-based scaffold ensuring identical tool access and system prompts.
• Metrics:
  • Consistency: Measured via the Coefficient of Variation (CV) of step counts across runs. Lower CV indicates higher consistency.
  • Accuracy: Binary "resolved" status based on the official SWE-bench evaluation harness (all previously failing tests must pass).
  • Phase Decomposition: Actions categorized into Explore, Understand, Edit, Verify, and Other to trace variance origins.
  • Divergence Analysis: Identifying the specific step where independent runs of the same task first execute different action categories.

3. Key Contributions

1. Quantitative Hierarchy: Established a clear correlation where higher capability models exhibit both higher accuracy and higher consistency (Claude > GPT-5 > Llama).
2. The Amplification Insight: Demonstrated that consistency amplifies outcomes rather than guaranteeing correctness. A consistent agent will reliably execute a correct strategy, but also reliably repeat a wrong one.
3. Speed-Accuracy-Consistency Tradeoff: Identified a fundamental tradeoff where GPT-5 is significantly faster but suffers in both accuracy and consistency compared to the more thorough Claude.
4. Fixation Failure Mode: Identified a specific failure mode where thorough agents (like Claude) become "fixated" on an incorrect initial interpretation, repeating the same error across all runs.
5. Divergence Timing vs. Variance: Showed that early strategic agreement (divergence timing) does not solely determine consistency; post-divergence behavior is equally critical.

4. Key Results

A. Consistency and Accuracy Correlation

• Claude 4.5: Lowest variance (CV: 15.2%) and highest accuracy (58%).
• GPT-5: Intermediate variance (CV: 32.2%) and accuracy (32%).
• Llama-3.1: Highest variance (CV: 47.0%) and lowest accuracy (4%).
• Crucial Nuance: While models correlate across tiers, within a single model, consistency does not predict accuracy at the task level. A model can be consistently wrong or inconsistently right.

B. The "Consistent Wrong" Phenomenon

• 71% of Claude's failures were classified as "consistent wrong interpretation." In these cases, the agent made the same incorrect assumption in all 5 runs, leading to identical failures.
• When Claude correctly interpreted a task, it achieved 100% accuracy (5/5 runs). When it misinterpreted, it achieved 0% accuracy (0/5 runs).
• This suggests that for capable agents, the bottleneck is interpretation quality, not execution consistency.

C. The Speed-Accuracy-Consistency Tradeoff

• GPT-5 was 4.7× faster than Claude (9.9 steps vs. 46.1 steps).
• However, this speed came at a cost: 1.8× lower accuracy and 2.1× worse consistency.
• GPT-5 spent significantly more time on Verification (32.3% of steps) and used specific commands like nl heavily, whereas Claude invested heavily in Understanding (41.2% of steps).

D. Divergence Analysis

• Surprising Finding: Claude and GPT-5 diverge at nearly identical steps (Mean: 3.2 vs. 3.4), yet Claude is 2.1× more consistent.
• Implication: Early strategic agreement is necessary but not sufficient. Consistency is determined by how the agent behaves after the initial divergence. Llama, by contrast, diverged immediately (60% of tasks at Step 1).

E. Case Studies

• Case 1 (Thoroughness Backfires): On task astropy-13236, Claude and GPT-5 consistently misinterpreted the bug (adding a deprecation warning instead of removing code) and failed 0/5 times. Llama, despite high variance, stumbled upon the correct interpretation in 1/5 runs.
• Case 2 (Efficiency Paradox): On task astropy-14309 (simple import error), GPT-5 matched Claude's 5/5 accuracy but in only 7 steps (vs. 53 for Claude), proving that thoroughness is not always required for simple tasks.

5. Significance and Implications

• Re-evaluating Reliability: The paper challenges the assumption that "consistent = reliable." Consistency is a double-edged sword; it ensures reliability only if the underlying interpretation is correct.
• Deployment Strategy:
  • Production Systems: Should prioritize models with high interpretation accuracy (like Claude) even if they are slower, as "consistent wrong" behaviors are harder to debug than inconsistent ones.
  • Rapid Prototyping: May favor faster models (like GPT-5) where speed is prioritized over perfect consistency.
• Benchmarking Standards: Single-run evaluations are misleading. The authors advocate for multi-run evaluations that report both accuracy and consistency (CV) to distinguish between "60% of tasks solved consistently" and "100% of tasks solved 60% of the time."
• Agent Design: Future agent architectures should focus on adaptive strategies that balance thoroughness with efficiency based on estimated task complexity, and prioritize mechanisms to detect and correct initial interpretation errors before execution begins.

Conclusion

The study concludes that interpretation accuracy is the primary bottleneck for reliable agent deployment, not execution consistency. Consistency acts as an amplifier: it magnifies the success of correct interpretations and the failure of incorrect ones. Therefore, improving agent reliability requires focusing on the quality of the initial task understanding rather than solely optimizing for execution consistency or speed.