Hybrid-Gym: Training Coding Agents to Generalize Across Tasks

1. Problem Statement

Current benchmarks for evaluating coding agents, such as SWE-Bench, predominantly focus on single-issue resolution (fixing bugs in GitHub repositories). However, real-world software engineering involves a diverse range of complex tasks, including test generation, library implementation, and codebase exploration.

• Limitation of Current Methods: Existing training approaches often focus on a single task (e.g., issue-solving), leading to overfitting. Agents trained on issue resolution fail to generalize to other tasks like test generation or dependency management.
• Scalability Bottleneck: Constructing high-quality training data for coding agents typically requires setting up executable repositories (installing all dependencies, generating tests, ensuring the environment runs). This process is time-consuming for humans and challenging for agents, limiting the scale and diversity of training datasets.
• Core Question: How can we design training tasks that teach transferable skills (reasoning, exploration, implementation) without the heavy overhead of executable repository setup, thereby enabling agents to generalize across diverse real-world coding tasks?

2. Methodology

The authors propose HYBRID-GYM, a scalable training framework based on a systematic analysis of coding agent trajectories.

A. Analysis of Task Transferability

The authors decomposed real-world coding trajectories (from SWE-Bench, SWT-Bench, and Commit-0) into five intermediate components:

1. Reasoning: Planning and decision-making.
2. Repo-Exploration: Navigating the codebase (using grep, find, ls, etc.).
3. Execution: Running existing code.
4. Implementation: Writing/Editing code.
5. Verification: Testing the solution.

Key Findings:

• Dominant Components: Reasoning, Repo-Exploration, and Implementation account for approximately 68% of agent actions in successful trajectories.
• Tool Consistency: The specific bash commands (e.g., grep, find) and tools used for exploration are highly similar across different downstream tasks.
• Setup Independence: Crucially, these dominant components do not require an executable repository. They can be performed on static codebases.

B. Design Principles for Training Tasks

Based on the analysis, the authors established four principles for designing synthetic training tasks:

1. Output Format Matching: Tasks must require generating a code patch (file editing), not just a text plan, to match downstream task formats.
2. Meaningful Repo-Exploration: Tasks must involve navigating a repository structure, not just script-level generation.
3. Non-Trivial Reasoning: Tasks must require substantive decisions (strategy selection, argument generation) rather than simple string replacement.
4. Scalable Setup: Tasks must avoid complex environment setup (e.g., no full repository installation or executable test generation).

C. The HYBRID-GYM Tasks

The authors constructed four scalable synthetic tasks that satisfy the above principles:

1. Function Localization: Given a description, locate the specific function in the codebase and write a docstring with a fix plan.
2. Issue Localization: Given a GitHub issue, locate the relevant code file and leave a comment with a fix plan.
3. Dependency Search: Given a function, identify and comment on all modules/classes it directly calls (requires static analysis reasoning).
4. Function Generation: Given a function signature and description, re-implement the function body. Note: Evaluation uses a lightweight script extraction method (RepoST) rather than full repo execution.

3. Key Contributions

• HYBRID-GYM Dataset: A large-scale dataset containing 4,470 trajectories across 762 repositories. It is built with only 2 Docker images (vs. 128+ for other datasets) and costs $0.07 per example (16x cheaper than SWE-Smith).
• Task Design Principles: A formalized set of guidelines proving that executable repository setup is not a prerequisite for teaching core agent skills.
• Comprehensive Analysis: Controlled experiments revealing that trajectory complexity (length), task complexity, and output format are critical for transfer, while repository diversity in training data improves generalization.

4. Experimental Results

The authors evaluated the approach using Qwen2.5Coder-32B (and 7B) on three diverse downstream benchmarks:

• SWE-Bench Verified: Issue resolution.
• SWT-Bench Verified: Test generation.
• Commit-0 Lite: Library generation.

Key Performance Metrics:

• Zero-Shot Transfer: Without training on any downstream task data, HYBRID-GYM improved the base model by:
  • +25.4% on SWE-Bench Verified.
  • +7.9% on SWT-Bench Verified.
  • +5.1% on Commit-0 Lite.
• Comparison: These results match or exceed models trained on in-domain datasets (e.g., SWE-Play).
• Complementary Effect: Combining HYBRID-GYM with in-domain data (e.g., SWE-Play) yielded further gains (e.g., +4.9% on SWT-Bench), suggesting HYBRID-GYM teaches general agent skills while in-domain data teaches task-specific nuances.
• Scaling Law: Performance continued to improve as the training data size increased from 250 to 4,400 trajectories.

Ablation Studies:

• Output Format: Removing file-editing actions (str_replace) from training caused a collapse in performance, proving patch generation is a learnable skill, not just formatting.
• Script vs. Repo: Script-level tasks (e.g., LiveCodeBench) failed to transfer to repo-level tasks, confirming the need for repository exploration.
• Teacher Model: The choice of teacher model matters; trajectories where reasoning and actions were separated (e.g., o3-mini) degraded student performance unless stitched together.

5. Significance

• Paradigm Shift: The paper challenges the assumption that "executable environments" are necessary for training coding agents. It demonstrates that static code analysis and reasoning are the primary drivers of generalization.
• Cost Efficiency: By reducing the cost of data construction by orders of magnitude, HYBRID-GYM makes it feasible to generate massive, diverse training datasets for coding agents.
• Generalization: It provides a blueprint for training agents that are robust across the entire spectrum of software engineering tasks, moving beyond the narrow scope of bug-fixing.
• Future Direction: The work highlights the importance of trajectory quality (how reasoning and actions are interleaved) and data diversity (exposure to many repositories) over simply memorizing specific codebases.

In summary, HYBRID-GYM proves that by focusing on the fundamental, transferable components of coding (reasoning, exploration, and implementation) through scalable synthetic tasks, we can build coding agents that generalize effectively to complex, real-world software engineering challenges without the prohibitive cost of executable environment setup.