Technical Summary: SERA: Soft-Verified Efficient Repository Agents

Problem Statement

Open-weight coding agents possess a theoretical advantage over closed-source systems: the ability to specialize to private codebases by encoding repository-specific patterns and domain knowledge directly into model weights. However, realizing this advantage has been hindered by the high cost and complexity of training. Traditional approaches rely on either Reinforcement Learning (RL), which demands complex sandboxed execution environments and distributed infrastructure, or synthetic data generation pipelines (e.g., SWE-smith) that require rigorous unit test infrastructure to verify bug fixes. These barriers have concentrated coding agent development within well-resourced industry labs, leaving smaller teams and academic researchers unable to practically specialize models to their own private repositories.

Methodology: Soft-Verified Generation (SVG)

The authors introduce Soft-Verified Generation (SVG), a method designed to generate high-quality training trajectories without unit tests or complex bug-injection pipelines. SVG operates on two core observations derived from simplifying previous methods:

1. Soft Verification over Hard Verification: Instead of executing unit tests to verify synthetic data, SVG compares the partial line-by-line overlap (recall) between two patches generated by the same teacher model.

   • Rollout 1: A teacher model ($M$) is prompted with a vague instruction to make a change to a randomly selected function in a codebase ($C$), producing a trajectory ($T_1$) and a patch ($P_1$).
   • Synthetic PR Generation: $T_1$ is converted into a synthetic Pull Request (PR) description.
   • Rollout 2: The teacher model attempts to reproduce the original patch ($P_1$) given only the synthetic PR description, producing a second trajectory ($T_2$) and patch ($P_2$).
   • Verification: The patches are compared using line-level recall ($r = |P_2 \cap P_1| / |P_1|$). Data is selected based on this recall score. The authors find that even unverified data ($r=0$) or soft-verified data ($0 < r < 1$) performs comparably to hard-verified data ($r=1$), removing the need for test infrastructure entirely.

2. Vague Instructions for Diversity: Rather than focusing solely on bug fixes, the method uses vague instructions (e.g., "fix issues related to state management") to generate diverse changes, including refactoring and documentation improvements. This diversifies the training data and improves performance on benchmarks like SWE-bench.

Training Pipeline:

• Teacher Model: GLM-4.5-Air (or GLM-4.6) is used as the teacher due to its open-weight nature, reasoning traces, and cost efficiency.
• Base Model: Qwen 3-32B is used for supervised fine-tuning (SFT).
• Data Scale: The method generated over 200,000 synthetic trajectories from 121 codebases.
• Truncation Strategy: To handle long contexts, the authors found that preserving the "truncation ratio" (the proportion of a trajectory that fits within the context window) is critical. Selecting trajectories with high truncation ratios (e.g., 0.95) yields better performance than random truncation.

Key Contributions

1. SERA Models: The release of SERA-32B, a 32B parameter coding agent trained via SFT on SVG data. It achieves state-of-the-art results among fully open-source models (open data, method, code) on SWE-bench Verified, matching or exceeding the performance of open-weight models like Devstral-Small-2.
2. Cost Efficiency: Creating SERA models is 26x cheaper than Reinforcement Learning approaches (e.g., SkyRL) and 57x cheaper than previous synthetic data methods (e.g., SWE-smith) to reach equivalent performance. The total cost for data generation and training was approximately $2,000 (40 GPU days).
3. Repository Specialization: The authors demonstrate that open-weight models can be specialized to private codebases (e.g., Django, Sympy, Sphinx) using only ~8,000 synthetic trajectories per repository. These specialized models match or exceed the performance of the teacher model on specific repository tasks, a feat impossible for generalist models that rely solely on context window access.
4. Soft Verification Validation: The paper provides empirical evidence that unit test verification is not strictly necessary for generating effective coding training data, significantly lowering the barrier to data generation.
5. Scaling Laws and Robustness: The study establishes scaling laws for coding agents, showing that performance follows a predictable power law with data size. It also highlights the critical importance of using multiple random seeds (at least 3) to distinguish signal from noise, noting that many reported gains in the field fall within the margin of run-to-run variance.

Results

• SWE-bench Verified Performance:
  • SERA-32B (GLM-4.6 Teacher): Achieved 49.5% at 32K context and 54.2% at 64K context.
  • This outperforms other fully open-source models (e.g., SWE-smith at 25.6–32.6%, SkyRL at 39.4%) and matches strong open-weight models like Devstral-Small-2 (50.0% at 32K).
• Specialization Efficiency:
  • Specializing a model to the Django repository with 8,000 samples achieved 52.23% accuracy, surpassing the teacher model (GLM-4.5-Air) at 51.2% and the open-weight Devstral-Small-2 at 51.3%.
  • Scaling laws indicate that full specialization ($\alpha=1.0$) reaches teacher performance at 8k samples, whereas general data alone ($\alpha=0.0$) fails to match this even with double the samples.
• Verification Thresholds: Experiments showed no statistically significant difference in performance between models trained on hard-verified ($r=1.0$), soft-verified ($r=0.5$), or completely unverified ($r=0.0$) trajectories, suggesting that the skill of translating intent to code is learned even from imperfect trajectories.

Significance and Claims

The paper claims that SERA makes the specialization of coding agents to private codebases practical and affordable. By stripping away the need for test infrastructure and complex RL setups, the authors enable small teams and academic researchers to build specialized agents that outperform frontier systems on their own codebases.

Key implications highlighted by the authors include:

• Democratization: Reducing the barrier to entry for coding agent research, moving it away from large industry labs with massive GPU clusters.
• Privacy and IP: Enabling organizations to train models on proprietary code without exposing data to third-party API providers, addressing critical privacy and intellectual property concerns.
• Open Science: The release of 200,000 synthetic trajectories, training code, and model weights provides a foundation for reproducible research, allowing the community to iterate without rebuilding expensive infrastructure.

The authors remain modest regarding their claims, acknowledging limitations such as the lack of evaluation on truly private codebases (due to the absence of ground truth for evaluation) and the potential for model-specific optimization (results are based on Qwen and GLM families). They emphasize that while their method accelerates progress, the field requires rigorous statistical practices (multiple seeds, signal-to-noise analysis) to avoid mistaking noise for genuine improvement.