Technical Summary: OPEN-SWE-TRACES

Problem Statement

The advancement of autonomous software engineering (SWE) agents is currently bottlenecked by a severe scarcity of diverse, large-scale trajectory data. While evaluation benchmarks like SWE-bench and its multilingual variants have expanded to assess repository-level issue resolution across various programming languages, the development community lacks the necessary pre-built environments and high-fidelity interaction traces required for robust model training. Existing resources often fail to support the "dual-mode" reasoning frameworks emerging in flagship foundation models, which require distinct data for both complex, multi-step planning ("thinking") and rapid, execution-oriented responses ("non-thinking").

Methodology

The authors introduce OPEN-SWE-TRACES, a comprehensive dataset construction pipeline designed to bridge the gap between static evaluation and large-scale agent training.

1. Data Source and Selection

The dataset is synthesized from 20,000 real-world Pull Requests (PRs) sourced from SWE-rebench-V2, a containerized task environment spanning 20 programming languages. To ensure legal integrity and redistribution compliance, the authors applied strict heuristics:

• Language Scope: Limited to nine languages: Python, Go, TypeScript, JavaScript, Rust, Java, PHP, C, and C++.
• Licensing: Only repositories under permissive licenses (MIT, Apache-2.0, BSD 2/3-Clause) were included.

2. Dual-Mode Synthesis

To capture the full spectrum of agentic behavior, the authors employed a hybrid reasoning synthesis strategy using an ensemble of state-of-the-art models:

• Thinking Mode: Generated using Minimax-M2.5, producing trajectories with explicit internal "Chain-of-Thought" (CoT) reasoning.
• Non-Thinking Mode: Generated using Qwen3.5-122B, providing high-quality, direct tool-use traces without explicit reasoning steps.
• Frameworks: Synthesis was conducted within the OpenHands and SWE-agent harnesses (specifically their multilingual extensions, MOpenHands and MSWE-agent).

3. Quality Filtering and Security

A rigorous multi-stage filtering pipeline was implemented to ensure technical soundness:

• Runtime Validation: Discarded corrupted trajectories resulting from environment exceptions or incomplete interaction sequences.
• Behavioral Pruning: Removed instances of task incompleteness (hitting iteration limits), structural invalidity (empty patches or "test suite hacking"), and tool-use anomalies.
• Git Hacking Detection: A custom framework, TrajectoryScanner, utilized Abstract Syntax Tree (AST) parsing to audit Bash commands. It systematically pruned trajectories exhibiting "git hacking"—attempts to bypass problem-solving by extracting solutions from repository metadata (e.g., git log, git blame, reflog).

4. Model Distillation

The dataset was used to fine-tune the Qwen3-30B-A3B series (including Thinking, Coder, and Instruct variants). The resulting model, OPEN-SWE-AGENT, supports dual-mode operation via /think and /no_think triggers. Training utilized a learning rate of 1e-5, a batch size of 32, and a context window of up to 131,072 tokens (expanded to 262,144 during inference).

Key Contributions

1. OPEN-SWE-TRACES Dataset: The largest multilingual trajectory corpus to date, containing 207,489 high-fidelity agent traces. It is uniquely structured to support dual-mode agent development, with approximately 51.7% of traces containing internal reasoning.
2. OPEN-SWE-AGENT: A fine-tuned open-source agent demonstrating the efficacy of dual-mode distillation. It achieves state-of-the-art performance among open-source models of comparable size.
3. Systematic Analysis: An empirical study isolating performance drivers, including the impact of base model selection, data filtering (resolved-only vs. full corpus), and the scaling effects of multilingual versus Python-only training.

Experimental Results

The authors evaluated OPEN-SWE-AGENT across three benchmarks:

• SWE-bench Verified (500 Python issues): The best-performing model achieved a 61.7% resolve rate (using the /no_think trigger with MSWE-agent), representing a significant absolute gain over the base model (51.6%) and competing open-source baselines.
• SWE-bench Multilingual (300 tasks across 8 languages): The model achieved a 57.1% resolve rate, demonstrating strong cross-lingual transfer capabilities.
• SWE-bench Pro (731 complex tasks): The model achieved a 36.8% resolve rate, a notable performance on rigorous, real-world engineering tasks.

Ablation Findings:

• Multilingual Data: Training on the full multilingual dataset provided a substantial boost over Python-only training, particularly for non-Python tasks (+14% absolute gain in some configurations).
• Data Filtering: Including "unresolved" trajectories (full corpus) proved superior to training exclusively on "resolved-only" data, helping models navigate complex states even without successful fixes.
• Cross-Harness Generalization: While generalization between harnesses (OpenHands vs. SWE-agent) is possible, a performance penalty exists. Models trained on MSWE-agent transferred more effectively to MOpenHands than vice versa.

Significance and Claims

The paper positions OPEN-SWE-TRACES as a premier resource for distilling human-level software engineering capabilities into efficient, open-source agentic Large Language Models. The authors claim that the dataset successfully addresses the resource scarcity hindering the field, enabling the training of the next generation of dual-mode autonomous agents.

The results establish that combining explicit thinking modalities with high-quality behavioral traces is a highly effective path for developing specialized agents. By providing a dataset that captures both the "thinking" process and the "doing" execution, the work facilitates the creation of models capable of long-horizon reasoning while maintaining execution speed for standard operations. The authors conclude that their approach sets a new baseline for open-source autonomous software engineering, validating the necessity of large-scale, diverse, and rigorously filtered trajectory data.