Technical Summary: Handoff Debt: The Rediscovery Cost When Coding Agents Take Over

Problem Statement

Current coding-agent benchmarks, such as SWE-bench, evaluate whether a single, uninterrupted agent can resolve a repository issue. While effective for measuring raw model capability on isolated tasks, this abstraction ignores a pervasive real-world scenario: takeover. In practice, software work is often interrupted, reassigned, or resumed from partial states left by another agent or engineer.

The core problem is that a predecessor's partial progress is only valuable if a successor can understand it. When a predecessor leaves opaque or incomplete state (e.g., unexplained edits, scratch files, hidden assumptions, or missing validation evidence), the successor incurs a rediscovery cost. The paper defines this cost as handoff debt: the additional effort a successor agent must expend to reconstruct the predecessor's intent, evidence, and intermediate artifacts before continuing the task. A metric based solely on final resolution fails to distinguish between efficient continuation and costly rediscovery.

Methodology

The authors introduce a takeover protocol to measure handoff debt by converting standard SWE-bench Verified tasks into handoff-point tasks.

Experimental Setup

• Source Tasks: 75 tasks from SWE-bench Verified (15-minute to 4-hour difficulty tiers).
• Predecessor: A single agent (Qwen) initiates each task.
• Interruption Points: The predecessor is interrupted at deterministic, observable handoff points:
  1. After the first source code edit.
  2. After the first validation result (build, lint, or test).
  3. After the first post-failure edit.
• Successors: Three models (Qwen, Gemma, Devstral) resume the task from the checkpointed repository.
• Handoff Views: The protocol evaluates four distinct context formats provided to the successor:
  1. Repository Only: The successor receives only the checkpointed filesystem state and the original task prompt.
  2. Raw Trace: The successor receives the full, unstructured event log of the predecessor (commands, edits, observations, failures).
  3. Summary Notes: The successor receives free-form natural language notes generated from the predecessor's event log.
  4. Structured Notes: The successor receives a concise, fixed-field record (a "continuation contract") containing changed files, validation evidence, uncertainty, and rollback risks. Some fields are filled deterministically from logs; others are filled by the predecessor model.

Metrics

• Primary Outcome: Official SWE-bench resolution (does the patched repository pass validation?).
• Cost Metrics:
  • Agent Events: The number of tool interactions and actions required.
  • Cumulative Prompt Tokens: The total token consumption, reflecting the volume of information processed.
• Handoff States: Tasks are labeled by the state of the repository at takeover (e.g., "Needs completion," "Already solved; preserve," "Existing behavior broken").

Key Contributions

1. Formulation of Handoff Debt: Defining handoff debt as a measurable property of coding-agent takeovers that captures the cost of inheriting partial work, separating resumability from raw model capability.
2. Takeover Protocol: A method to convert SWE-bench tasks into handoff-point tasks, generating 181 handoff points and 2,172 takeover runs across three successor models.
3. Isolation of Handoff Views: A controlled comparison holding the predecessor, handoff point, and repository state fixed while varying only the handoff view to isolate the effect of context format.
4. Structured Handoff Schema: The proposal of a bounded continuation contract with fixed fields for changed files, validation evidence, uncertainty, and verification, designed to make handoffs auditable and reusable.
5. Empirical Evidence on Rediscovery: Demonstration that predecessor context primarily reduces rediscovery effort (agent events and tokens) even when gains in final solved rates are modest.

Results

The study found that context-bearing handoffs consistently reduce rediscovery costs across all three successor models, though the impact on final solved rates varies.

• Efficiency Gains: Compared to the "Repository only" baseline, context-bearing handoffs significantly reduced median agent events and cumulative prompt tokens:
  • Raw Trace: Reduced median agent events by 57–59% and prompt tokens by 42–63%.
  • Summary Notes: Reduced median agent events by 20–46% and prompt tokens by 42–63%.
  • Structured Notes: Reduced median agent events by 20–46% and prompt tokens by 42–63%.
• Solved-Rate Effects: Improvements in solved rates were positive but less consistent and model-dependent. Raw traces showed the strongest gains (e.g., +14.9 percentage points for Qwen-to-Devstral), while note-based gains were smaller and sometimes statistically insignificant depending on the model.
• Handoff Point Sensitivity: The "After first post-failure edit" point incurred the highest rediscovery debt for repository-only successors (median 122–191 agent events). Context-bearing views provided the largest solved-rate improvements at this stage, as they conveyed critical failure evidence that the repository state alone could not.
• State Sensitivity: In "Needs completion" scenarios (where the task was unfinished), repository-only successors resolved only 13.6–23.6% of tasks, whereas context-bearing views raised this to 23.6–28.2%.
• Trade-offs: While Raw Traces offered the highest information density, they resulted in massive initial prompts (median 87k characters vs. ~10k for notes). However, this large initial context often reduced the total token count by preventing redundant exploration later.

Significance and Claims

The paper claims that coding-agent evaluation must evolve beyond binary success/failure metrics to include the cost of resumption.

• Evaluation Shift: Benchmarks should report not only whether a task is solved but also how costly that work is for another agent to resume. A task solved by an agent that leaves an opaque state creates "debt" for the next agent, which is a critical failure mode in collaborative software engineering.
• Practical Implications:
  1. Reporting: Benchmarks should report handoff debt alongside final resolution to assess understandability, verifiability, and resumability.
  2. Workflow Design: Agent systems should treat handoff output as a deliberate part of the workflow (e.g., generating structured notes or bounded contracts) rather than merely logging a run when it stops.
• Modesty: The authors note that their conclusions are specific to the OpenHands-style runtime and the selected task pool. They do not claim to rank models universally but rather to isolate the effects of handoff formats. They acknowledge that while structured notes are promising, they require verification, as successors may over-trust predecessor interpretations.

The findings suggest that the "messiness" of real-world software work—interruptions and handoffs—is a fundamental dimension of coding-agent performance that current benchmarks overlook.