Technical Summary: SWE-Together

Problem Statement

Current benchmarks for coding agents, such as the SWE-Bench family and Terminal-Bench, predominantly rely on static, single-turn protocols. In these setups, an agent receives a complete task description upfront and is evaluated solely on the correctness of its final code submission. This approach fails to reflect the reality of software engineering assistance, which is inherently interactive and multi-turn. In real-world workflows, users incrementally reveal intent, clarify ambiguous requests, refine requirements, and provide corrective feedback over multiple turns.

Existing benchmarks suffer from two primary mismatches with practice:

1. Task Design: They treat tasks as fixed instructions, ignoring that real task-relevant information is often distributed across multiple turns and initial requests are frequently incomplete.
2. Evaluation Metrics: They focus exclusively on final-task success, failing to capture the interaction cost. Two agents may achieve the same final score, but one might succeed from a coarse request while the other requires extensive detailed specifications and repeated corrective feedback.

Furthermore, constructing a benchmark that preserves realistic user interaction while ensuring verifiability is challenging. Raw conversation logs often lack reproducible repository states, identifiable goals, or observable outcomes. Additionally, simply replaying original user turns to a new agent is flawed because later user turns must be conditioned on the new agent's trajectory to prevent interaction drift, yet must remain anchored to the original user's intent.

Methodology

To address these challenges, the authors introduce SWE-Together, a benchmark comprising 109 repository-level tasks reconstructed from real user–agent coding sessions. The methodology consists of three core components:

1. Session-to-Task Construction Pipeline

The authors curate 109 tasks from 11,260 recorded sessions sourced from four upstream datasets: DataClaw, Pi-staging, Hyperswitch, and SWE-chat. The construction pipeline involves three stages:

• Deterministic Eligibility Filtering: Raw sessions are filtered to retain those with genuine multi-turn interactions, concrete code edits, and recoverable repository contexts. Sessions where the human user authored the final change are excluded to ensure the agent performed substantive work.
• Viability Screening: An LLM judge screens candidates to ensure the coding work can be reconstructed as a self-contained, locally executable task. Sessions relying on external state (e.g., live service deployments, private credentials) are rejected.
• Task Construction: A sandbox orchestrator converts viable sessions into executable benchmark packages. This includes restoring the repository at a pinned commit, defining deterministic verifier artifacts, and generating task-specific user-simulation prompts.

2. Anchored, State-Conditional User Simulator

To replay interactions with new agents, the authors develop a reactive LLM-based user simulator. Unlike static replays, this simulator:

• Conditions on Trajectory: It observes the evaluated agent's latest turn, repository changes, and elapsed time.
• Anchors to Original Intent: It uses a pre-computed analysis of the original session (objectives, constraints, and intervention conditions) to ensure the simulation remains faithful to the original user's goals.
• State-Conditional Decision Policy: The simulator decides whether to intervene (send a message) or remain silent (no-op) based on whether the agent's current trajectory warrants feedback according to the original session's logic. This prevents premature or irrelevant interventions.

3. Joint Evaluation Protocol

Evaluation measures two distinct dimensions:

• Task Correctness: The final repository state is scored against a frozen, implementation-agnostic rubric. This rubric is derived in Phase 1 (offline) from the original session and reference patch, then applied in Phase 2 to candidate repositories. Scoring combines deterministic executable checks with an agentic rubric judge to assess behavioral completeness rather than code similarity.
• Interaction Diagnostics:
  • User Correction: Quantifies the corrective steering elicited by the agent. It counts explicit corrections and weighted "nudges" (softer feedback) provided by the simulator.
  • Intent Coverage: A diagnostic metric measuring how faithfully the simulator preserves the original user's intents and scope across different agent runs.

Key Contributions

1. SWE-Together Benchmark: A 109-task benchmark reconstructed from real multi-turn user–agent sessions, featuring a pipeline that converts recorded sessions into verifiable, sandboxed tasks.
2. State-Conditional User Simulator: An anchored LLM simulator that preserves original user intents and intervention orders while adapting feedback timing to the specific trajectory of each evaluated agent.
3. Joint Evaluation Protocol: A framework that scores final repository correctness against fixed rubrics while characterizing interaction trajectories via User Correction and Intent Coverage, moving beyond simple pass/fail metrics.

Results

The authors evaluated seven frontier coding models on the 109-task benchmark using the opencode harness with two replicates per task.

• Performance Ranking: Claude Opus 4.8 achieved the highest performance across all metrics (pass@1: 63%, Stable Solve Rate: 59%, Mean Judge Score: 0.801). GPT-5.5 ranked second (Mean Judge: 0.763), followed by Claude Opus 4.6. The weakest models were DeepSeek-V4-Pro and MiniMax-2.7.
• Capability vs. Intervention: A strong negative correlation was observed between model capability and User Correction (Pearson correlation of -0.92 with pass@1). Stronger models required significantly fewer corrective interventions to reach comparable performance levels (e.g., Claude Opus 4.8 required 1.38 corrections on average, while MiniMax-2.7 required 2.17).
• Efficiency: GPT-5.5 was the most efficient, requiring the fewest tokens (29.9k) and wall-clock time (10.7 min) per task while maintaining high capability.
• Simulator Fidelity: Intent Coverage remained stable across all model cohorts (scores between 0.70–0.72), indicating the simulator consistently preserved original intents regardless of the agent's behavior.
• Human Indistinguishability: In a Turing-style test, human annotators could not statistically distinguish between simulated and real user trajectories (pass rate of 46%, CI including 50%).

Significance and Claims

The paper claims that SWE-Together reframes coding-agent evaluation from one-shot tasks to replayable, multi-turn sessions, offering a more realistic assessment of the user experience.

• Beyond Final Correctness: The results suggest that evaluating agents solely on final code is insufficient; the amount of corrective steering required is a critical indicator of capability and user experience.
• Realistic Interaction: By deriving tasks from real sessions and using a state-conditional simulator, the benchmark captures the dynamic nature of software engineering where requirements evolve.
• Reliable Comparison: The stability of Intent Coverage and the indistinguishability of simulated trajectories from real ones validate the benchmark's ability to provide fair, comparable evaluations across different model cohorts.

The authors conclude that SWE-Together demonstrates the importance of evaluating coding agents not just by whether they complete a task, but by how much human guidance they require to do so, thereby aligning benchmark metrics more closely with actual user workflows.