Technical Summary: SWE Atlas – Benchmarking Coding Agents Beyond Issue Resolution

Problem Statement

The current landscape of Large Language Model (LLM) coding benchmarks has largely converged on "functional resolution"—tasks focused on fixing bugs or implementing specific features (e.g., SWE-Bench, TerminalBench). The authors argue that this focus creates a critical blind spot in evaluating professional software engineering. Real-world engineering involves maintaining code health, preventing regressions, understanding complex architectures, and adhering to best practices—tasks that are distinct from simple patching. Existing benchmarks often fail to evaluate these complementary capabilities, risking the creation of agents that are excellent "patchers" but poor "engineers" incapable of maintaining long-term repository health. Furthermore, many of these workflows (like codebase Q&A or refactoring) are difficult to evaluate using purely programmatic checks like unit tests, which often miss qualitative aspects of code quality such as maintainability, anti-patterns, and architectural hygiene.

Methodology

To address these gaps, the authors introduce SWE Atlas, a benchmark suite comprising 284 expert-authored tasks drawn from 18 actively maintained open-source repositories. The suite targets three underrepresented but practically important professional workflows:

1. Codebase Q&A (124 tasks): Tasks requiring deep code comprehension, architecture analysis, and runtime anomaly investigation. Unlike static repository Q&A, these tasks require agents to set up, run, and trace data flow through live executions to provide evidence-backed answers.
2. Test Writing (90 tasks): Tasks requiring the authoring of production-grade tests (unit, integration, acceptance) for specified behaviors. These tasks demand an adversarial mindset to anticipate edge cases rather than a compliant mindset to pass checks.
3. Refactoring (70 tasks): Tasks requiring the restructuring of code without altering observable behavior. These involve extracting modules, evolving interfaces, and cleaning up artifacts while adhering to strict interface specifications.

Evaluation Framework

SWE Atlas employs a hybrid evaluation protocol combining programmatic checks with rubric-based assessment to capture engineering rigor:

• Programmatic Checks: Includes mutation testing for test suites (verifying tests fail when code is broken) and regression testing for refactoring (verifying existing tests still pass).
• Rubric-Based Assessment: Utilizes an "LLM-as-a-Judge" approach with expert-written structured rubrics. Rubrics act as checklists for binary (YES/NO) decisions on criteria such as code placement, maintainability, documentation updates, artifact cleanup, and the absence of anti-patterns.
• Quality Control: The benchmark underwent a multi-step quality control process involving expert engineers, independent reviews, and memorization screens to ensure solution leakage was not a factor.

Key Contributions

1. New Benchmark Suite: The release of 284 tasks spanning Codebase Q&A, Test Writing, and Refactoring, designed with under-specified, agentic task formulations that better reflect real-world usage than fully specified problem statements.
2. Multifaceted Evaluation Framework: A novel combination of programmatic verifiers and rubric-based LLM grading. This framework evaluates not just functional correctness but also software engineering quality, including test completeness, refactor maintainability, and codebase hygiene.
3. Public Release: The full task suite, evaluation harness, and judge prompts are released publicly. The authors provide evidence that despite the open-source provenance of the repositories, there is no clear evidence of solution memorization by current frontier models.

Experimental Results

The authors evaluated a range of frontier models (GPT-5.4, Opus 4.7, etc.) and open-weight models on SWE Atlas using the Harbor Framework and provider-specific scaffolds (Codex CLI, Claude Code, etc.).

• Performance Leaders: GPT-5.4 (Codex) and Opus 4.7 (Claude Code) achieved the strongest overall performance. However, even the best models scored modestly, with Pass@1 rates generally below 50% across workflows.
• Consistency Issues: A significant drop was observed between Pass@1 (mean per-trial pass rate) and Pass3 (fraction of tasks where all three trials pass), indicating that models struggle to consistently produce correct answers.
• Open-Weight Models: Even the best open-weight models (e.g., GLM 5) scored significantly lower than frontier closed models.
• Functional vs. Engineering Quality: A critical finding is the gap between functional correctness and engineering quality. For example, in Refactoring, models often passed regression tests (60–80% pass rate) but failed rubric checks for maintainability and artifact cleanup (scores dropped significantly). Similarly, in Test Writing, models passed mutation checks but often failed rubric checks for test comprehensiveness and edge-case coverage.
• Failure Modes:
  • Q&A: Top models often fail to provide complete runtime evidence or miss subtle edge cases in their analysis.
  • Test Writing: Agents tend to write tests that cover "happy paths" but fail to test negative spaces or invariant conditions, leading to tests that pass even on broken code.
  • Refactoring: Models struggle with multi-file refactors and coordinated cross-file edits, often leaving behind dead code or failing to update all call sites.
• Agentic Behavior: Top-performing models rely heavily on extensive codebase exploration and runtime-driven reasoning. The use of sub-agents (delegation) was found to be a significant differentiator for Codebase Q&A tasks.

Significance and Claims

The paper claims that SWE Atlas provides a necessary complementary evaluation suite for coding agents, moving beyond simple issue resolution to measure the "engineering quality" of AI-generated code. The authors assert that while frontier models are saturating simple functional benchmarks, there is substantial room for improvement in professional software engineering capabilities.

The significance of SWE Atlas lies in its ability to:

1. Expose Engineering Gaps: Highlight that current models are often ineffective at maintaining long-term repository health, adhering to best practices, and writing maintainable code.
2. Introduce New Evaluation Axes: Demonstrate that rubric-based evaluation reveals model differentiation that programmatic verification alone misses.
3. Guide Future Development: Suggest that top models rely on deep, agentic exploration and runtime analysis, and that future progress requires better handling of subtle edge cases, complex runtime analysis, and strict adherence to software engineering best practices.

The authors conclude that SWE Atlas serves as a held-out signal of progress rather than a target for optimization, aiming to improve the transparent evaluation of coding agents before their deployment in real-world, safety-critical workflows.