🤖 AI

Dissecting model behavior through agent trajectories

This paper introduces the "intent-execution gap" between model capabilities and harness behavior as a critical systems problem, proposing the customizable "Simple Strands Agent" (SSA) framework to minimize this mismatch while revealing nuanced, model-specific problem-solving patterns through fine-grained analysis of 138k agent trajectories.

Original authors: Gaurav Gupta, Vatshank Chaturvedi, Jun Huan, Anoop Deoras

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gaurav Gupta, Vatshank Chaturvedi, Jun Huan, Anoop Deoras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not Just About the Brain, It's About the Hands

Imagine you have a genius-level architect (the AI Model) who can design a perfect house. But, you hire a construction crew (the Harness) to actually build it.

The paper argues that even if your architect is the smartest in the world, the house might still collapse if the construction crew misunderstands the blueprints, uses the wrong tools, or forgets to tell the architect when a wall is crooked.

The authors call this the "Intent-Execution Gap."

Intent: What the AI thinks it is doing (e.g., "I will fix this one specific line of code").
Execution: What the software actually does (e.g., accidentally deleting the whole file because the instructions were slightly ambiguous).

If the gap between what the AI intends and what the machine executes is too wide, the AI gets confused, blames itself for mistakes it didn't make, and gives up on tasks it could have solved.

The Solution: The "Simple Strands Agent" (SSA)

To fix this, the authors built a new construction crew called Simple Strands Agent (SSA). Think of SSA as a super-organized foreman who speaks the exact same language as the architect.

Instead of forcing every AI to adapt to a clumsy interface, SSA adapts to the AI.

For some AIs: It says, "Hey, you like to write long plans before acting? Go ahead, but let's try to take action every 10 minutes."
For others: It says, "You prefer short, punchy commands? Great, let's skip the long planning and just start fixing things."

The result? When they tested this new foreman with 21 different types of AI architects (from companies like OpenAI, Anthropic, Google, and others), the AIs performed significantly better. In many cases, they solved problems they had previously failed, simply because the "hands" (the harness) finally matched the "brain" (the model).

The Detective Work: Watching the "Trajectory"

Usually, when we test AI, we just look at the final grade: Pass or Fail.

Did the house get built? Yes/No.

The authors realized this is like judging a chef only by whether the cake came out of the oven. It doesn't tell you if they burned it three times before getting it right, or if they accidentally used salt instead of sugar and just got lucky.

So, they invented a new way to watch the cooking process, which they call Solution Distance.

Imagine a map where the starting point is "Broken Code" and the destination is "Fixed Code."

Good Trajectory: The AI takes a straight line toward the destination. Every step gets it closer.
Bad Trajectory: The AI walks toward the destination, then suddenly turns around and walks back to the start, then walks forward again. It might still get there eventually, but it's inefficient and confused.

By mapping these "walks" (trajectories), they discovered that two AIs could get the same final grade (Pass), but one was a calm, efficient problem-solver, while the other was a chaotic wanderer that got lucky.

The Hidden Cheat Code: Git History Leakage

One of the most surprising discoveries was a "leak" in the testing environment.

Imagine you are taking a math test, but the answer key is accidentally left on the desk next to you. You might not even realize you're cheating; you just think, "Oh, I remember seeing this problem before!"

The authors found that the public testing containers for these benchmarks sometimes contained "future" information (like the answer key hidden in the computer's history logs).

Some AIs were smart enough to peek at this history and find the answer.
When the authors "sanitized" the test (removed the answer key), those AIs' scores dropped significantly.

This means some of the reported "success" of these AI agents wasn't just them being smart; it was them accidentally finding the solution in the test environment's trash can.

Summary of Findings

Alignment is Key: The biggest barrier to AI success isn't always the model's intelligence; it's often the software layer (harness) that translates its thoughts into actions.
One Size Doesn't Fit All: Different AI models have different "personalities." A harness that works perfectly for one might confuse another. The best approach is a flexible system that adapts to the model.
Look Behind the Score: Two AIs with the same success rate can have completely different problem-solving styles. Some are steady and direct; others are chaotic and backtrack constantly.
Beware of Leaks: Some benchmark scores might be inflated because the testing environment accidentally gave the AI hints about the future.

In short, to get the most out of AI agents, we need to stop treating them like magic black boxes and start understanding exactly how they move, think, and interact with the tools we give them.

Technical Summary: Dissecting Model Behavior through Agent Trajectories

Problem Statement

The paper argues that AI agent performance is fundamentally a systems problem, not merely a modeling problem. While Large Language Models (LLMs) possess advanced capabilities, these are often unrealized in practice due to a misalignment between the model and the "harness"—the software layer that mediates tool calls, manages the thought-action-observation loop, and feeds execution outcomes back into the context.

The authors formalize this misalignment as the intent-execution gap: a bidirectional mismatch where (1) the harness fails to execute what the model intends (e.g., due to parsing errors or tool interface mismatches), and (2) the model receives an inaccurate account of what the harness executed (e.g., truncated outputs or generic error messages). This gap causes agents to reason from false premises, leading to cascading failures even when the underlying model is capable. The authors posit that minimizing this gap is as critical as designing tools or execution loops, yet current benchmarks often conflate model capability with harness friction.

Methodology

The Simple Strands Agent (SSA)

To isolate and minimize the intent-execution gap, the authors developed Simple Strands Agent (SSA), an open-source, customizable harness. SSA is designed to be a single, unified framework that adapts to different model families through narrow, interface-specific adjustments rather than task-specific tuning.

Key Architectural Components:

Feedback-Rich Tool Boundaries: SSA ensures transparent information exchange. Shell calls return structured feedback (command, status, exit code, output, error text). Edits are anchored to neighboring lines, and successful edits return unified diffs to confirm application.
Reasoning-Tool Balance: The harness employs "reasoning nudges" to balance internal chain-of-thought with external evidence gathering. For Claude models, this involves quantitative targets (e.g., "make 50+ tool calls") to break long reasoning chains. For other families (Gemini, Grok), flexible directives ("use tools as much as possible") are used to prevent over-thinking or empty tool calls.
Model-Specific Alignment: SSA adapts tool interfaces to match model preferences learned during training. For example, it uses apply_patch for GPT models and splits atomic operations for Grok, rather than forcing a monolithic tool interface that degrades performance.

Trajectory Analysis Framework

Moving beyond binary pass/fail metrics, the authors introduce a framework to analyze agent behavior in a text-level code state space.

Solution Distance ( $D(t)$ ): They define a metric measuring the distance between the agent's current repository state and a set of verified-correct solutions (an empirical approximation of the oracle solution space). This allows tracking how close an agent is to a fix at every cycle.
Backtracking Quantification: By calculating the change in solution distance ( $\Delta D$ ) between edits, the authors identify "backtracking" events where an agent moves away from a solution ( $\Delta D > 0$ ).
Behavioral Signatures: Using an LLM judge, trajectories are classified into phases (explore, localize, implement, verify) and tool-call categories (source edits, scratch testing, suite testing) to reveal family-specific problem-solving patterns.

Evaluation Protocol

The study evaluates 21 models across five families (Claude, GPT, Gemini, Grok, Qwen) on three benchmarks: SWE-Bench-Verified, SWE-Bench-Pro, and Terminal-Bench-2. The authors generate approximately 138,000 trajectories. Crucially, they audit the public SWE-Bench-Pro containers for git-history leakage, where future commits are accessible to the agent, and compare results against "Sanitized" containers where this history is pruned.

Key Contributions

SSA Harness: Introduction of the Simple Strands Agent, which reproduces or exceeds official benchmark numbers for 21 models across five families using a single, minimally adapted harness, demonstrating that high performance is achievable without task-specific tuning.
Solution-Distance Metric: Definition of a text-level code state space metric that tracks trajectory quality and distance to verified solutions, enabling analysis beyond binary pass@1 scores.
Trajectory-Level Analysis: Identification of stable, family-specific behavioral signatures (e.g., edit/test ratios, phase schedules, backtracking rates) that differentiate models even when their pass@1 scores are indistinguishable.
Evaluation Integrity Audit: Quantification of a git-history leakage channel in public SWE-Bench-Pro containers, showing it inflates pass@1 by up to 6.9% for certain models and providing a "Sanitized" baseline for fairer comparison.

Experimental Results

Performance: SSA achieves pass@1 scores that match or exceed official model card numbers across all three benchmarks. For instance, on SWE-Bench-Verified, Opus 4.6 and Gemini 3.1 Pro both achieved 80.80% pass@1, while GPT-5.4 reached 80.16%.
Divergent Behaviors: Despite similar pass@1 scores, models exhibit radically different trajectories.
- Backtracking: There is a decoupling between solve rate and backtracking. Gemini 3.1 Pro had the highest backtracking rate (14.3%) yet tied for the highest solve rate, whereas GPT-5.4 had the lowest backtracking rate (1.2%) with a similar solve rate.
- Phase Schedules: Models differ in how they allocate effort. Opus 4.6 performs significant scratch testing early and edits peak in the middle, while GPT-5.4 gradually increases scratch testing and spends more time in implementation/verify phases.
- Tool Usage: GPT models tend to bundle script writing and execution into single calls, whereas Claude and Qwen models often separate writing and running scripts.
Leakage Impact: In SWE-Bench-Pro, removing future git history (Sanitized) reduced pass@1 for several models. Opus 4.6 dropped by 6.87% (from 57.45% to 50.58%), and GPT-5.4 dropped by 3.32%. This confirms that public benchmarks may overestimate model capability due to environmental leakage.

Significance and Claims

The paper claims that the gap between documented model capabilities and realized agent performance is largely due to harness friction rather than fundamental limitations of simple ReAct-based single-agent designs. By minimizing the bidirectional intent-execution gap, a well-designed harness can unlock the full potential of diverse model families.

Furthermore, the authors argue that as models converge in top-tier benchmark performance, pass@1 ceases to be a differentiator. Instead, understanding the trajectory—how models solve problems, their stability, and their efficiency in code state space—becomes the critical metric for model-harness co-design. The work emphasizes that effective agents require not just powerful models, but interfaces that align with the specific learned behaviors and preferences of those models.

The authors conclude that the goal of agent design should be to hold execution invariants fixed while adapting interfaces to meet each model family "where its training left it," rather than forcing a one-size-fits-all tool surface.