💻 computer science

ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance

This paper introduces ChainSWE, the first benchmark designed to evaluate coding agents on sequential, dependent bug fixes within a shared codebase, revealing that agent performance drops significantly as the chain length increases compared to traditional isolated bug-fixing evaluations.

Original authors: Qirui Jin, Lingching Tung, Kenan Li, Qiyang Shi, Yushi She, Huanzhong Jia, Harrison Zhao, Kejing Xia, Zhenbang Du, Yikai Zhang, Jiaxin Pei, Zhenyu Zhang, Zhen Qi, Yuyan Duan, Wenke Lee, Zijian Jin

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qirui Jin, Lingching Tung, Kenan Li, Qiyang Shi, Yushi She, Huanzhong Jia, Harrison Zhao, Kejing Xia, Zhenbang Du, Yikai Zhang, Jiaxin Pei, Zhenyu Zhang, Zhen Qi, Yuyan Duan, Wenke Lee, Zijian Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: From "One-and-Done" to "The Long Haul"

Imagine you are hiring a team of super-smart robot mechanics to fix a fleet of cars.

The Old Way (Current Benchmarks):
Every time you give a mechanic a problem, you hand them a brand-new, pristine car. They fix the flat tire, you check their work, and then you send them home. The next day, you give them a different car with a different problem.

The Problem: This doesn't test if they are good at maintaining a car over time. In the real world, mechanics don't get a new car every day. They work on the same car, fixing a flat tire, then a squeaky brake, then a weird engine noise, all on the same vehicle.

The New Way (CHAINSWE):
The researchers built a new test called CHAINSWE. Instead of giving the mechanics new cars, they give them one car and a list of 304 problems that happened over several years.

The mechanic fixes the first problem.
Then, without resetting the car, they have to fix the second problem based on the state of the car after the first fix.
Then the third, and so on.

The Two Main Ways Robots Fail

The paper found that when robots try to fix a long list of problems on the same code (the "car"), they make two specific types of mistakes that they don't make when fixing single problems:

1. The "Over-Decorator" Mistake (Overshoot)

The Scenario: The robot is asked to fix a leaky faucet. It fixes the faucet perfectly. But, in its excitement, it also repaints the kitchen cabinets and changes the floor tiles, even though nobody asked for that.
The Consequence: Later, a human comes in to fix a broken light switch. Because the robot changed the floor tiles and cabinets earlier, the instructions for the light switch no longer make sense. The robot's "extra" work broke the next task.
In the paper: The robot changes files it wasn't supposed to touch, which breaks the tests for future bugs.

2. The "Half-Finished" Mistake (Undershoot)

The Scenario: The robot is asked to fix a leaky faucet. It realizes the faucet needs a new pipe and a new valve to work. It only replaces the faucet handle (because that's what the note said) and leaves the broken pipe and valve alone.
The Consequence: The faucet looks fixed, but it still leaks. Later, a human tries to fix the water pressure. Because the robot didn't fix the pipe earlier, the water pressure fix fails completely.
In the paper: The robot fixes the specific file mentioned in the bug report but forgets to update the supporting files that the bug report didn't explicitly mention, leaving the code in a broken state for the next bug.

What Happened When They Tested the Robots?

The researchers tested 7 different "AI mechanics" (Language Models) using this new "Long Haul" test.

The Results: When the robots worked on single bugs (the old way), they were pretty good (about 60% success). But when they had to work on a chain of bugs (the new way), their performance dropped by up to 70%.
The "Chain Reaction": The deeper they got into the list of bugs, the worse they did. By the time they reached the 3rd or 4th bug in a row, they were failing almost constantly.
Why? The robots were getting confused by their own previous work. They couldn't remember which files they had changed, or they forgot that their earlier "quick fixes" had broken the foundation for the next task.

Did "Memory" Help?

The researchers tried to help the robots by giving them different ways to remember what they did:

Full Memory: Reading the entire history of everything they ever said.
Summarized Memory: Asking the robot to write a short summary of what it did earlier.
Helper Robots: Using a small robot to do the file editing while the main robot just gave instructions.

The Surprise: None of these tricks really helped much. In fact, asking the robot to summarize its work or use a helper often made things worse. The robots just couldn't handle the "mess" they created in the code, no matter how much they tried to remember it.

The Bottom Line

The paper concludes that we are currently testing AI coders like they are one-hit wonders (fixing one thing and leaving). But in the real world, software maintenance is a marathon, not a sprint.

To build AI that can actually maintain software, we need to stop testing them on isolated tasks and start testing them on chains of tasks where they have to deal with the messy, imperfect code they created themselves. Right now, even the smartest AI models struggle to keep a codebase clean when they have to fix bug after bug in a row.

Technical Summary: CHAINSWE – Benchmarking Coding Agents on Multi-Bug Software Maintenance

Problem Statement

Current Large Language Model (LM) agents are increasingly deployed for software engineering (SWE) tasks, including code repair and repository maintenance. However, existing benchmarks (e.g., SWE-bench, SWE-bench Live, SWE-rebench) evaluate these agents on isolated, single-issue tasks. In these protocols, the repository is reset to a clean base commit for every evaluation, and the agent solves one self-contained bug in a fresh container.

This evaluation schema fails to capture the reality of continuous software maintenance, where engineers fix streams of related defects on a codebase they have just modified. The paper identifies two critical failure modes that emerge only when bugs are evaluated sequentially and repository state carries across sessions:

Overshoot: An agent fixing an earlier bug modifies files beyond the intended scope. When a subsequent bug targets those same files, the agent's edits are applied to a "polluted" baseline, causing correct changes to fail.
Undershoot: An agent fails to apply necessary multi-file refactors required by a prior bug (shipping only the subset mentioned in the issue title). Downstream tests fail not because the current bug is mishandled, but because the preconditions established by the missing refactor are absent.

The core problem is that current benchmarks ignore cumulative dependencies and state drift, leading to an overestimation of agent capabilities in real-world, long-horizon maintenance scenarios.

Methodology

To address this gap, the authors introduce CHAINSWE, the first benchmark designed to evaluate agents on sequential, dependent bug fixes within a shared codebase.

Data Construction Pipeline

The dataset is constructed by mining and validating chains from six existing SWE-bench-family datasets (SWE-bench, SWE-bench Live, SWE-rebench, SWE-rebench-v2, SWE-Gym, and SWE-bench Pro).

Mining: Instances are sorted chronologically by commit date. A sliding window groups consecutive instances into chains if:
- The latter's base commit is the immediate successor of the former's, OR
- The latter's gold/test patches apply cleanly on top of the former's resolved state.
Filtering:
- Code-overlap heuristic: Discards windows where bugs share no modified files, functions, or classes.
- Test re-execution: A containerized filter applies gold patches in order to verify that all FAIL_TO_PASS tests pass and no PASS_TO_PASS tests regress.
- Difficulty selection: Chains where a single bug is intrinsically too hard (preventing >50% of bugs from being solved even in "Oracle" mode) are discarded to ensure the benchmark tests state management rather than raw bug-solving ability.
Final Dataset: The resulting dataset contains 100 bug chains spanning 304 instances across 54 Python repositories, with an average chain length of 3.04.

Evaluation Protocol

The authors evaluate seven state-of-the-art language models (including GPT-5.5, Claude-Opus-4.7, and DeepSeek-V4-Pro) using a fixed agent scaffold (SWE-EDIT) under three context-management configurations:

BASELINE: Retains full interaction history.
SUMMARIZE: Replaces older history with a structured summary once the context exceeds 50k tokens.
SUB-AGENT: Offloads file viewing/editing to specialized sub-agents to keep the main context window clean.

Evaluation occurs in three modes to isolate specific failure dimensions:

ORACLE: The container is reset to the base commit, and all gold patches for prior bugs are applied before the agent starts. This measures per-bug ability without state drift.
SEQ (Sequential): The agent works through bugs 1 to $n$ in a single container without resetting. The repository carries the agent's own generated patches. The agent harness restarts, but the codebase state persists.
SEQ+MEM: Identical to SEQ, but the agent's conversation history is also preserved across bugs.

Key Results

The evaluation reveals a consistent and severe performance degradation when moving from isolated to sequential evaluation:

Performance Drop: Across all models and configurations, per-bug accuracy drops by approximately 38% when moving from ORACLE to SEQ (from ~59% to ~36.5%). In the deepest positions of a chain, performance drops by up to 70%.
Chain Errors: Of the failures in sequential mode (positions 2 and 3), 48% are "chain errors"—failures induced by the agent's own accumulated state rather than the intrinsic difficulty of the current bug.
- Overshoot (breaking working code) accounts for ~40% of these errors.
- Undershoot (leaving bugs unfixed or preconditions unmet) accounts for ~25%.
Context Management Strategies:
- Summarization and Sub-agents degrade performance: Surprisingly, strategies designed to manage long contexts (SUMMARIZE and SUB-AGENT) perform worse than the raw history baseline (BASELINE). The relative drop from ORACLE to SEQ increases from 33% (BASELINE) to 40-41% (SUMMARIZE/SUB-AGENT), suggesting that information loss or inconsistency in rewriting contexts is more harmful than retaining raw history.
- Memory benefits are limited: Preserving conversation history (SEQ+MEM) offers negligible gains for most models. Only GPT-5.5 shows a modest improvement (+7 to +8 points) in specific configurations, likely because it can effectively reason over the longer history. For other models, the added tokens do not translate to better state tracking.
Positional Analysis: In ORACLE mode, later bugs are not inherently harder; success rates often rise with position. In SEQ mode, success rates plummet as the chain deepens, confirming that the difficulty is driven by accumulated state drift, not task complexity.

Significance and Contributions

The paper makes three primary contributions:

CHAINSWE Benchmark: It provides the first open-source benchmark explicitly evaluating LMs on sequential, multi-bug rollouts over real-world repositories, exposing failure modes invisible to single-issue benchmarks.
Scalable Pipeline: It introduces a pipeline for constructing and validating chained SWE tasks, enabling continuous evaluation over arbitrarily long bug-fix sequences.
Empirical Insights: It demonstrates that current SWE agents lack robust dependency tracking and long-horizon reasoning. The results suggest that future progress requires advances in managing repository state and understanding cross-task dependencies, rather than solely improving isolated issue resolution.

The authors conclude that the current "reset-and-solve" paradigm masks critical failure modes. Real-world software maintenance requires agents that can leave the codebase in a state that is not only correct for the current issue but also usable as a starting point for future work. The significant performance drop observed in CHAINSWE indicates that current agents are not yet ready for continuous, autonomous software maintenance.