From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents

1. Problem Statement

The paper addresses the critical scalability bottleneck in training autonomous Software Engineering (SWE) agents. Current state-of-the-art approaches rely heavily on execution-based training, where agents must interact with isolated, containerized environments (e.g., Docker) to verify code fixes via unit tests. This paradigm faces three major limitations:

• Data Scalability: Many real-world repositories cannot be reliably containerized due to complex or legacy dependencies, forcing researchers to discard vast amounts of potential training data.
• Training Scalability: Orchestrating thousands of task-specific Docker images creates massive infrastructure overhead, hindering large-scale optimization and Reinforcement Learning (RL).
• Inference Scalability: The cost of spinning up environments and running tests limits the ability of agents to explore multiple solution branches during inference (Test-Time Scaling).

The authors argue that an absolute dependency on physical execution prevents open-source models from reaching the performance levels of frontier proprietary models.

2. Methodology: The SWE-ZERO to SWE-HERO Pipeline

The authors propose a two-stage Supervised Fine-Tuning (SFT) recipe that decouples semantic reasoning from physical verification. This approach transitions agents from "intuitive" understanding to "rigorous" engineering workflows.

Stage 1: SWE-ZERO (Foundational Reasoning)

• Concept: An execution-free approach that leverages the internal "world models" of frontier Large Language Models (LLMs) to resolve issues without runtime feedback.
• Data Generation:
  • Utilizes a teacher model (Qwen3-Coder-480B) to generate trajectories on 150k real-world GitHub Pull Requests (PRs).
  • The agent operates in a restricted sandbox (no Docker, no Python execution, no pytest). It is explicitly forbidden from running code or tests.
  • The agent must rely on static analysis, codebase exploration, and semantic reasoning to identify bugs and implement patches.
• Filtering: A multi-stage pipeline filters out trajectories where the agent attempts prohibited commands (e.g., python, pytest) or fails to produce logical code changes.
• Outcome: A dataset of 300k execution-free trajectories. These are significantly more concise (40% fewer tokens) than execution-based trajectories because they bypass iterative test-validation loops.

Stage 2: SWE-HERO (Grounded Refinement)

• Concept: A targeted, execution-backed refinement stage to transition the agent's semantic intuitions into verified engineering workflows.
• Data Generation:
  • Uses the same teacher model to generate trajectories on a subset of 13k task instances that have verified, containerized Docker environments.
  • Agents operate in full environments, allowing for code execution, test running, and iterative debugging.
• Training Process:
  • The model is first fine-tuned on the SWE-ZERO dataset to establish broad repository-level reasoning.
  • This base model is then fine-tuned on the SWE-HERO dataset to learn the nuances of verification and robust patching.
• Outcome: A refined dataset of 13k execution-based trajectories.

3. Key Contributions

1. Novel Training Paradigm: The introduction of SWE-ZERO to SWE-HERO, a curriculum that prioritizes large-scale, low-cost semantic reasoning before introducing high-cost, high-fidelity execution.
2. Open-Source Datasets: Release of 300k SWE-ZERO and 13k SWE-HERO trajectories distilled from Qwen3-Coder-480B, enabling the community to train agents without needing massive Docker infrastructure.
3. Agent Suite: Release of fine-tuned agents based on the Qwen2.5-Coder series (7B, 14B, and 32B parameters) that outperform existing open-source agents.
4. Cross-Language Generalization: Demonstration that training primarily on Python execution-free data enables robust zero-shot transfer to multilingual SWE tasks.

4. Experimental Results

The models were evaluated on SWE-bench Verified (500 high-quality Python issues) and SWE-bench Multilingual (300 cross-language issues).

• State-of-the-Art Performance:

  • SWE-HERO-32B achieves a 62.2% resolution rate on SWE-bench Verified, setting a new benchmark for open-source models of comparable size.
  • It outperforms competitors like SERA-32B (54.2%), daVinci-Dev-32B (56.1%), and OpenSWE-32B (62.4%, though OpenSWE uses a much more resource-intensive training setup).
  • Smaller models show significant gains: SWE-HERO-7B reaches 52.7% (vs. 46.8% for SWE-ZERO-7B), outperforming same-scale competitors by a wide margin.

• Ablation Studies:

  • Necessity of SWE-ZERO: Training directly on SWE-HERO data (skipping the execution-free stage) results in a drop to 55.7% on Verified and 30.8% on Multilingual. This proves SWE-ZERO provides a critical inductive bias for semantic reasoning.
  • Data Scaling: Increasing the SWE-ZERO training set from 4k to 150k samples improved resolution rates from 42.1% to 49.1%, confirming that data volume in the execution-free phase is a primary driver of performance.

• Multilingual Transfer:

  • Despite being trained exclusively on Python, SWE-HERO-32B achieves 44.1% on SWE-bench Multilingual (with execution), demonstrating that the foundational reasoning patterns learned in SWE-ZERO generalize across languages.

• Efficiency:

  • SWE-ZERO models resolve tasks with 40% fewer tokens and interaction turns compared to execution-based models, as they skip the verbose diagnostic and testing loops.
  • SWE-HERO models trade this efficiency for higher accuracy on complex tasks by utilizing execution feedback for "double-checking."

5. Significance and Impact

• Democratization of SWE Agents: By removing the dependency on Docker-based environments for the majority of training data, this framework lowers the barrier to entry for training high-performance SWE agents. Researchers can now leverage the "long tail" of GitHub data that was previously inaccessible due to containerization failures.
• Cost-Effectiveness: The two-stage approach drastically reduces the computational cost of training. The bulk of the learning (300k samples) happens without expensive environment orchestration, reserving high-cost execution only for the final refinement stage.
• Paradigm Shift: The paper challenges the notion that physical execution is strictly necessary for learning code semantics. It suggests that frontier LLMs possess sufficient internal "world models" to reason about code architecture, which can be effectively distilled and then grounded with limited execution data.
• Future Directions: The authors highlight that while current open-source verifiers limit Test-Time Scaling (TTS) gains, the SWE-ZERO to SWE-HERO pipeline provides a scalable foundation for future work in autonomous software engineering, particularly for reasoning-based LLM architectures.

In summary, SWE-ZERO to SWE-HERO represents a scalable, efficient, and highly effective recipe for training software engineering agents, achieving state-of-the-art results in open-source models by strategically combining massive-scale semantic reasoning with targeted, grounded refinement.