Technical Summary: daVinci-Dev: Agent-native Mid-training for Software Engineering

1. Problem Statement

The frontier of Large Language Model (LLM) capabilities in software engineering has shifted from single-turn code generation to agentic software engineering, where models must autonomously navigate, edit, and test complex repositories. While current state-of-the-art approaches rely on post-training (Supervised Fine-Tuning followed by Reinforcement Learning), this paradigm faces critical limitations:

• Data Scarcity and Mismatch: Post-training is constrained by the base model's intrinsic capacities and the limited quantity of high-quality, executable trajectories available. Most training data consists of static, completed artifacts (e.g., merged commits) that lack the sequential action-observation loops inherent in real development.
• Distribution Mismatch: Traditional training data shows what was produced but obscures how a developer or agent discovered files, examined context, and iteratively refined solutions based on test feedback. This creates a gap between static training distributions and the dynamic, feedback-rich environment of real deployment.
• Underexplored Mid-Training: While mid-training (MT) has proven transformative for domains like mathematics and code, existing efforts for software engineering often adopt a factorized approach, synthesizing isolated samples for atomic tasks (e.g., localization or editing) without preserving the procedural context and interaction dynamics an agent experiences.

2. Methodology: Agent-Native Mid-Training

The paper proposes daVinci-Dev, a systematic approach to agentic mid-training that instills foundational agentic behaviors before post-training. The core innovation is the construction of Agent-Native Data, defined as supervision that preserves the complete information flow and environmental dynamics of real software engineering.

The methodology relies on two complementary trajectory types:

A. Contextually-Native Trajectories ($D_{ctx}$)

• Goal: To provide broad coverage and diversity by preserving the full development context.
• Source: Constructed from GitHub Pull Requests (PRs).
• Synthesis: Instead of factorizing PRs into isolated tasks, the authors reconstruct complete workflows. For each PR, they bundle:
  • Issue Description: The problem specification.
  • Relevant Files: Complete file contents at the base commit (simulating the "read" phase).
  • Temporal Edits: A sequence of commits with refined commit messages and code diffs (simulating the "edit" phase).
  • Contextual Reasoning: LLM-generated summaries bridging the issue and edits.
• Scale: 68.6B tokens derived from 10M+ PRs across diverse languages (General subset) and Python-focused repositories (Python subset).

B. Environmentally-Native Trajectories ($D_{env}$)

• Goal: To provide interaction authenticity and capture the iterative feedback loop (Edit $\to$ Test $\to$ Revise).
• Source: Real executable environments derived from GitHub PRs.
• Synthesis: Using the SWE-AGENT framework and GLM-4.6, the authors generate agent rollouts in Docker environments containing actual unit tests.
  • The agent interacts with the environment, receiving real observations (tool outputs, test failures, runtime errors).
  • Trajectories are not filtered for success; both passing and failing trajectories are included to teach debugging and error handling.
• Scale: 3.1B tokens (approx. 74k trajectories), split into passing ($D_{env}^{pass}$) and failing ($D_{env}^{fail}$) sets.

C. Training Pipeline

The models (based on Qwen2.5-Base) undergo a two-stage process:

1. Mid-Training (MT): Training on the large-scale, unlabeled agent-native corpora ($D_{ctx}$ and/or $D_{env}$). This shifts the model's capability distribution to be more "agent-ready."
2. Post-Training: Supervised Fine-Tuning (SFT) on high-quality agentic trajectories (e.g., $D_{env}^{pass}$ or public datasets like SWE-smith) to align the model with specific scaffolds.

3. Key Contributions

1. Formulation of Agent-Native Mid-Training: The paper defines and implements a mid-training paradigm that treats agentic behavior as a first-class objective, moving beyond factorized subtask training to holistic, context-preserving supervision.
2. Large-Scale Data Construction: The authors release a practical recipe for constructing two complementary datasets:
   • A 68.6B-token contextually-native corpus ($D_{ctx}$) preserving full PR workflows.
   • A 3.1B-token environmentally-native corpus ($D_{env}$) capturing authentic execution feedback loops.
3. State-of-the-Art Performance: The resulting models, daVinci-Dev-32B and daVinci-Dev-72B, achieve new state-of-the-art results on SWE-Bench Verified among open training recipes using agentic scaffolds, despite starting from non-coder base models.
4. Resource Efficiency: The approach achieves superior performance using less than half the mid-training tokens (73.1B vs. ~150B) compared to the previous open recipe (KIMI-DEV), demonstrating higher information density and token efficiency.

4. Experimental Results

Evaluated on SWE-Bench Verified using the SWE-AGENT scaffold:

• Performance Gains:
  • daVinci-Dev-72B: Achieves a 58.5% resolution rate, surpassing the previous open SOTA (KIMI-Dev at 48.6% with SFT+RL) and outperforming baselines using strong SFT alone.
  • daVinci-Dev-32B: Achieves a 56.1% resolution rate, setting a new SOTA for 32B models using agentic scaffolds, despite starting from a non-coder base model (Qwen2.5-32B-Base).
• Robustness: The mid-training recipe yields consistent gains across different post-training regimes (Weak SFT vs. Strong SFT) and model scales (32B vs. 72B).
• Generalization: The models show significant improvements on general code generation benchmarks (HumanEval, EvalPlus) and scientific reasoning tasks (GPQA, SciBench), suggesting that agentic mid-training fosters transferable reasoning skills.
• Scaling Laws: The paper observes a strong log-linear relationship between training steps and performance ($R^2 \approx 0.90$), indicating that the paradigm has not yet saturated and benefits from further scaling.

5. Significance and Claims

The paper argues that agent-native data is the critical missing link for scaling software engineering agents. By bridging the distribution mismatch between static training data and dynamic deployment environments, the authors demonstrate that:

• Foundational Behaviors: Agentic capabilities (localization, iterative editing, test-driven refinement) can be instilled during mid-training, creating a stronger foundation for subsequent post-training.
• Synergy of Data Types: The combination of contextually-native data (for breadth and code diversity) and environmentally-native data (for depth and feedback authenticity) is essential. Contextually-native data alone improves performance, but adding environmentally-native trajectories allows the model to internalize execution dynamics.
• Scalability: The approach offers a scalable path for future agent development, leveraging the vast availability of public repositories and automated environment verification to expand beyond current language and task limitations.

The authors conclude that their work provides a "substantial computational investment" (datasets, recipes, checkpoints) to the community to facilitate further research in this underexplored paradigm.