💻 computer science

Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure

This large-scale empirical study of 9,374 coding agent trajectories reveals that while patch complexity fails to explain task difficulty, success is primarily driven by the underlying LLM's capabilities and specific behavioral strategies like pre-editing context gathering and validation, rather than framework design or simple trajectory length.

Original authors: Tural Mehtiyev, Wesley Assunção

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tural Mehtiyev, Wesley Assunção

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've hired a team of robot interns to fix bugs in a massive, complex library of software code. These aren't just simple scripts; they are "Coding Agents" powered by advanced AI brains (Large Language Models) that can read files, run tests, and edit code just like a human developer.

The big question the researchers asked was: "Why do some of these robot interns succeed while others fail, even when they are working on the same problems?"

They didn't just look at the final score (Did they fix it? Yes/No). Instead, they watched the entire movie of how the robots worked, analyzing nearly 10,000 attempts. Here is what they discovered, explained through simple analogies.

1. The "Simple Fix" Trap (Why Easy Tasks Are Hard)

You might think a task is easy if the fix only requires changing one line of code. It's like a mechanic saying, "Just tighten this one bolt."

The Finding: The researchers found 12 tasks that were labeled "easy" by humans because they only needed tiny code changes. Yet, every single robot failed at these tasks.
The Analogy: Imagine a car that won't start because the battery is dead. A "simple fix" is just swapping the battery. But if the robot mechanic thinks the problem is the engine and tries to rebuild the engine, they will fail, even though the fix was simple.
The Real Reason: The robots failed because they lacked architectural reasoning. They couldn't understand where the problem actually lived in the big picture. They fixed the symptom (the engine) instead of the root cause (the battery). They knew how to type the code, but not why the code was broken in the first place.

2. The "Longer is Worse" Myth (The Length Trap)

There was a popular belief that if a robot takes a long time (many steps) to solve a problem, it's struggling and will likely fail. It's like thinking, "If a student takes 3 hours to do a 10-minute math test, they probably got it wrong."

The Finding: This is actually a trick!
- Scenario A (Looking at one robot): If you watch one robot, yes, its failed attempts are usually longer than its successful ones.
- Scenario B (Looking at the same problem): But if you look at the same difficult problem and compare different robots, the successful robots actually took more steps than the failed ones.
The Analogy: Think of a maze.
- If you are a bad navigator, you might wander around for a long time and get lost (long path = failure).
- But if the maze is extremely hard, even the smartest navigator has to walk a long, careful path to find the exit. The "short" path was taken by a robot that gave up too early or guessed wrong immediately.
The Lesson: Don't judge a robot by how long it takes. Judge it by how it spends its time.

3. The Secret Sauce: How They Spend Their Time

The researchers found that the structure of the robot's work matters more than the length. They identified three "habits" that separate the winners from the losers:

Read Before You Write: The winners spent their first few steps reading the code and understanding the context. The losers jumped straight into editing the code immediately.
- Analogy: A good chef tastes the soup before adding salt. A bad chef just dumps in salt immediately.
Don't Spam Edits: Losers tended to make the same mistake over and over, editing the code, getting an error, and editing again without thinking.
- Analogy: It's like trying to open a stuck jar by hitting it with a hammer 28 times. The winner would stop, look for a tool, or ask for help.
Test Thoroughly: The winners spent a lot of time running tests to make sure their fix actually worked. The losers often skipped this or did it too late.

Crucially: These habits were fixed. A robot didn't say, "This is a hard problem, so I'll read more." It just had a personality. Some robots were "Readers," and some were "Hammers." The "Readers" won more often.

4. The Brain vs. The Body (LLM vs. Framework)

Finally, they asked: "What matters more? The Brain (the AI model) or the Body (the software framework that holds the tools)?"

The Finding: The Brain is everything.
The Analogy: Imagine a Formula 1 car (the Framework) driven by a novice driver vs. a Formula 1 car driven by a world champion.
- If you put a world champion in a beat-up sedan, they will still drive better than a novice in a Ferrari.
- The researchers found that if you swap the AI "Brain" for a smarter one, the robot gets much better, regardless of the software "Body" it's in.
- As the AI brains get smarter, the differences between the different software bodies disappear. A smart brain can make a simple body perform like a champion.

Summary: What Should We Do?

This paper tells us that to build better coding robots, we shouldn't just focus on making the software tools fancier or giving them longer instructions.

Upgrade the Brain: We need smarter AI models that can understand the "big picture" of how software works, not just how to type code.
Teach Good Habits: We need to train robots to read first, edit second, and test often.
Stop Judging by Time: A long process isn't necessarily bad; a rushed process is often fatal.

In short: It's not about how fast the robot runs; it's about whether it knows where it's going.

1. Problem Statement

Coding agents, which combine Large Language Models (LLMs) with tool-augmented interaction loops, have shown promise in automated software engineering. However, state-of-the-art agents still fail on over 20% of benchmarked tasks (e.g., SWE-bench Verified). Existing research primarily focuses on outcome-level metrics (resolution rates) or proxy metrics for difficulty (e.g., patch size, human-estimated time).

The paper identifies three critical gaps in current understanding:

Lack of Causal Explanation: We do not systematically understand why agents fail (e.g., is it task complexity, LLM reasoning, or framework constraints?).
Behavioral Ambiguity: It is unclear how failure unfolds behaviorally. Prior studies suggest failed trajectories are longer, but this correlation may be confounded by task difficulty.
Attribution Gap: There is no clear consensus on whether agent success is driven more by the underlying LLM capability or the framework design (orchestration, prompts, tool management).

2. Methodology

The authors conducted a large-scale empirical study analyzing 9,374 trajectories from 19 distinct agents (comprising 8 frameworks and 14 different LLMs) evaluated on the same 500 SWE-bench Verified tasks.

Key Design Features:

Complete Matrix Design: Every agent attempted every task. This enables within-task comparisons (controlling for task difficulty) and within-agent comparisons (controlling for agent identity), addressing confounds present in prior studies.
Enriched Trajectory Encoding: Instead of just counting steps, the authors encoded trajectories into 13 sub-phase symbols (e.g., Lb for browsing, Ps for patch with syntax error, Vp for validation pass). This captures execution quality and error types.
Feature Extraction: 90 task-level features were extracted, categorized into Patch Complexity, Test Demand, Issue/Prompt quality, and Metadata.
Analytical Approaches:
- RQ1 (Task Difficulty): Quantitative comparison of "never-solved" simple tasks vs. "always-solved" simple tasks, followed by qualitative trajectory analysis.
- RQ2 (Behavioral Patterns): Paired comparisons of resolved vs. failed trajectories using two approaches: (A) Fix agent, vary tasks; (B) Fix task, vary agents.
- RQ3 (LLM vs. Framework): Natural experiment design holding one variable constant (LLM or Framework) while varying the other to isolate effects.

3. Key Contributions & Results

RQ1: Why do agents fail on certain tasks?

Finding: Patch complexity (lines changed) is an insufficient predictor of difficulty.
Evidence: The study identified 12 "never-solved" tasks that required only simple, single-file patches (≤10 changes) and were labeled "easy" by human annotators. Yet, 0/19 agents solved them.
Root Cause: The failure is not due to code volume but an Architectural Reasoning Gap.
- Agents correctly locate the bug file but intervene at the wrong architectural layer (e.g., fixing the symptom in a display backend instead of the root cause in the serialization logic).
- They lack the domain knowledge or reasoning to understand component boundaries and causal dependencies.

RQ2: How do behavioral patterns differentiate success from failure?

Finding 1: Trajectory Length is a Confound.
- Approach A (Fix Agent): Failed trajectories appear longer (consistent with prior work).
- Approach B (Fix Task): When controlling for task difficulty, successful trajectories are actually longer (10% longer on average) than failed ones.
- Conclusion: Length correlates with task difficulty, not failure. Harder tasks take more steps regardless of outcome.
Finding 2: Trajectory Structure is the True Discriminator.
- Context Gathering: Agents that delay their first edit (gathering context first) succeed significantly more ( $\rho = +0.68$ ).
- Patch Intensity: Agents that "front-load" patching (editing immediately) fail more often ( $\rho = -0.78$ ).
- Validation: Agents that invest more steps in validation (running tests) succeed more ( $\rho = +0.50$ ).
Finding 3: Strategy is Agent-Determined, Not Task-Adaptive.
- Agents apply fixed strategies regardless of task complexity. Strong agents consistently read before editing; weak agents patch immediately. They do not dynamically adjust their workflow based on task difficulty.

RQ3: Does LLM capability or framework design drive success?

Finding: LLM capability is the primary driver of both outcome and behavior.
Evidence:
- Outcome Agreement: Agents sharing the same LLM (but different frameworks) agree on outcomes for 85–93% of tasks. Agents sharing the same framework (but different LLMs) agree on only 47–88% of tasks.
- Convergence: As LLMs improve (e.g., from Claude 3.5 to Claude 4), the performance gap between different frameworks shrinks significantly (from ~19 pp to <1 pp).
- Prompt Influence: While framework prompts influence which tools are used, they do not change the amount of work or the core strategy. This influence diminishes as LLMs become more capable; strong LLMs develop their own strategies and are less swayed by verbose system prompts.

4. Significance and Implications

Redefining Difficulty: Benchmark designers must move beyond patch size. Metrics should account for architectural complexity and domain knowledge requirements, as simple patches can still defeat agents due to reasoning gaps.
Monitoring & Debugging:
- Do not use trajectory length as a failure signal; it is ambiguous.
- Use structural signals: Early detection of "premature patching" (editing before reading) can flag likely failures before compute is wasted.
Agent Design:
- Invest in LLMs: For practitioners, upgrading the underlying LLM yields higher returns than redesigning the orchestration framework.
- Adaptive Strategies: Current agents use rigid, one-size-fits-all workflows. Future systems need mechanisms to assess task complexity and adapt their strategy (e.g., deciding when to read vs. when to edit).
Prompt Engineering: For strong LLMs, verbose system prompts (e.g., 5,000+ characters) add overhead without improving outcomes. Lean prompts may be preferable as the model's internal reasoning compensates for lack of explicit guidance.

5. Conclusion

This study shifts the focus from whether agents fail to how and why they fail. It demonstrates that the bottleneck for coding agents is no longer just code generation but architectural reasoning and strategic adaptability. The results suggest that while framework engineering matters, the trajectory of agent improvement is now dominated by the evolution of LLM reasoning capabilities.