BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

1. Problem Statement

Current benchmarks for Large Language Model (LLM) code agents, such as SWE-bench and its variants, primarily evaluate performance on narrow, repository-specific bug fixes. These benchmarks typically assume that all necessary context resides within a single codebase and do not require external knowledge.

However, real-world software engineering involves complex challenges that extend far beyond single-repository contexts, including:

• Cross-repository reasoning: Leveraging solutions from external repositories.
• Domain specialization: Solving problems requiring expertise in fields like bioinformatics or quantum physics.
• Dependency-driven migration: Refactoring entire codebases to adapt to breaking changes in upstream libraries.
• Full-system generation: Constructing functional repositories from natural language specifications.

The central question posed by the authors is: Can current code agents survive beyond single-repo bug fixing? The paper argues that existing evaluations fail to capture these critical dimensions, leading to an overestimation of agent capabilities in practical scenarios.

2. Methodology

A. BeyondSWE Benchmark

The authors introduce BeyondSWE, a comprehensive benchmark designed to evaluate code agents along two new axes: Resolution Scope (granularity of the fix) and Knowledge Scope (source of information). It comprises 500 real-world instances across four distinct settings:

1. Cross-Repository Issue Resolution (CrossRepo): Agents must fix issues in a target repository by explicitly leveraging code, solutions, or documentation from external repositories.
   • Construction: Scanned GitHub for PRs with external links; filtered for stability.
2. Domain-Specific Issue Resolution (DomainFix): Agents must solve problems requiring specialized scientific knowledge (e.g., quantum physics, materials science) beyond general programming.
   • Construction: Collaborated with 11 domain experts to curate 21 repositories and verify 72 high-complexity issues.
3. Dependency-Driven Migration (DepMigrate): Agents must migrate entire codebases to accommodate breaking changes in upstream dependencies (e.g., Pydantic v1 to v2, NumPy 1.x to 2.x).
   • Construction: Identified 23 major package upgrades; collected 178 migration instances across 120 repositories.
4. Document-to-Repository Generation (Doc2Repo): Agents must construct a fully functional repository from scratch based solely on a natural language specification document, without access to the original source code.
   • Construction: Generated specification documents from 50 recent repositories, masking implementation details and directory structures.

Infrastructure & Quality Control:

• Environment Construction: Used an LLM agent (Gemini 3 Pro) to automatically configure Docker environments, resolving dependency decay and ensuring reproducibility.
• Evaluation Protocol: Strict isolation where patches are applied to a fresh container. "Pass-to-Pass" (P2P) and "Fail-to-Pass" (F2P) tests are used to verify correctness.
• Anti-Cheating: Removed post-target commits and metadata to prevent agents from accessing solution hints in git history.

B. SearchSWE Framework

To investigate the role of external knowledge, the authors developed SearchSWE, an agentic framework that integrates deep research capabilities with coding.

• Architecture: Combines a local context (Docker container for code execution) with a global context (Search Tool and Browser Tool).
• Mechanism: Agents can iteratively search the web, read documentation, and synthesize information to solve tasks.
• Safety: Implements a "blocklist" mechanism to prevent agents from accessing the target repository's solution directly via search, forcing them to rely on indirect resources.

3. Key Contributions

1. BeyondSWE Benchmark: The first benchmark to systematically evaluate code agents across cross-repo reasoning, domain expertise, large-scale migration, and full-system generation.
2. SearchSWE Framework: A unified baseline to study the integration of search and coding, revealing the disconnect between these two capabilities in current LLMs.
3. Public Release: The authors released the full evaluation suite and SearchSWE implementation to accelerate research.

4. Experimental Results

The authors evaluated multiple frontier models (e.g., Gemini 3 Pro, GPT-5.2, DeepSeek-V3.2, Qwen3) using both the standard OpenHands framework and the SearchSWE framework.

Performance on BeyondSWE

• Significant Capability Gap: Even the best-performing frontier models plateaued below 45% success rate on average. This is a stark contrast to the 80%+ success rates often seen on SWE-bench Verified.
• No Dominant Model: No single model excelled across all task types. For example, Gemini 3 Pro led in migration tasks, while Seed-Coder performed better on cross-repo tasks.
• Task-Specific Weaknesses:
  • DomainFix: Consistently the hardest task (avg. <36% success), highlighting a bottleneck in domain-specific reasoning.
  • Doc2Repo: While pass rates were moderate (45–55%), the number of fully correct repositories was extremely low (max 2 instances), indicating agents struggle to architect coherent systems from scratch.

Impact of SearchSWE (Search Augmentation)

• Inconsistent Gains: Integrating search did not universally improve performance.
  • Positive: Search helped significantly in DomainFix (e.g., +7.5% for Gemini 3 Pro) where external documentation provided necessary context.
  • Negative/Degradation: In some cases, search degraded performance (e.g., Seed-Coder dropped 5.8% on CrossRepo).
• The Disconnect: The study reveals a critical misalignment between search and coding capabilities.
  • Information Landscape Gap: Search engines prioritize high-level documentation, often missing the precise, low-level logic (source code) required for implementation.
  • Temporal Misalignment: Agents often retrieve information for the latest library versions, conflicting with the legacy versions pinned in the local Docker environment.
  • Semantic Drift: Search results for niche libraries often contain noise from unrelated domains (e.g., "Family" in BIM vs. software), confusing the agent.
• Efficiency: Models that searched more frequently did not necessarily perform better. Effective agents (like Gemini 3 Pro) used search sparingly but strategically, whereas others searched indiscriminately, introducing noise.

5. Significance and Conclusion

BeyondSWE fundamentally shifts the evaluation paradigm for code agents from isolated bug fixing to holistic software engineering capabilities. The results demonstrate that:

1. Current Limitations: State-of-the-art LLMs are not yet ready for complex, real-world software engineering tasks that require cross-repo reasoning, domain expertise, or large-scale refactoring.
2. Search Integration Challenge: Simply adding search tools to code agents is insufficient. The "search-code" workflow requires a sophisticated ability to filter noise, reconcile version conflicts, and synthesize information from fragmented sources—skills that current models lack.
3. Future Direction: The paper calls for explicit attention during model development to unify search and reasoning capabilities, moving beyond isolated training tracks for "coding" and "research."

This work provides a rigorous, realistic benchmark and a flexible framework to guide the development of the next generation of autonomous software engineers.