💬 NLP

Anchored Self-Play for Code Repair

This paper proposes Anchored Self-Play (ASP), a reinforcement learning method that stabilizes the automatic curriculum of generator-fixer self-play by anchoring it with a reference set, thereby preventing drift toward unrealistic bugs and significantly improving code repair performance across diverse bug sources compared to standard self-play.

Original authors: Caroline Choi, Zeyneb Kaya, Shirley Wu, Tengyu Ma, Tatsunori Hashimoto, Ludwig Schmidt

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Caroline Choi, Zeyneb Kaya, Shirley Wu, Tengyu Ma, Tatsunori Hashimoto, Ludwig Schmidt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fix broken code. The robot needs to learn two things: how to break code on purpose (to understand what goes wrong) and how to fix it back (to learn the solution).

This paper describes a new training method called Anchored Self-Play (ASP) that helps robots learn this skill much better than before. Here is how it works, using simple analogies.

The Problem: The "Fake Bug" Trap

The researchers tried a method called Self-Play. Imagine a single robot playing two roles:

The Saboteur: It takes a perfect program and tries to break it.
The Mechanic: It tries to fix the broken program.

They used a simple rule to judge them: If the Saboteur makes the code fail a test, it gets a point. If the Mechanic makes the code pass the test, it gets a point.

The Issue: The robot Saboteur got too clever. It realized it could break the code in weird, nonsensical ways that the tests would catch, but which no real human programmer would ever make.

Analogy: Imagine a student taking a math test. To get the teacher to mark an answer wrong, the student writes the answer in invisible ink or uses a language the teacher doesn't speak. The teacher marks it wrong (the test fails), but the student hasn't actually learned how to solve the math problem. They just learned how to "game the test."

As the robot kept training, it got better at breaking code in these weird, "fake" ways. But when the researchers tested it on code broken by real humans, the robot got worse at fixing it. It had over-specialized on its own weird game.

The Solution: The "Anchor"

To fix this, the researchers introduced Anchored Self-Play (ASP). They added a small "reference library" of real-world bugs (bugs written by humans or other AI models) to the training process.

They did this in two ways:

1. The "Similarity Buzzer" (For the Saboteur)
When the Saboteur breaks the code, the system checks: "Does this break look like a real human mistake?"

They use a special tool (an embedding model) to measure how similar the "fake bug" looks to the "real bugs" in the reference library.
If the Saboteur creates a weird, invisible-ink bug, it gets a low score. If it creates a bug that looks like a real human error (like forgetting a comma or using the wrong number), it gets a bonus score.
Analogy: It's like a coach telling the Saboteur, "Don't just break the car by removing the wheels; break it by forgetting to put oil in the engine, because that's what real drivers do."

2. The "Real-World Mix" (For the Mechanic)
When the Mechanic is practicing, the system occasionally swaps out the "fake" bugs the Saboteur made and gives the Mechanic a "real" bug from the reference library to fix.

This ensures the Mechanic never forgets what real-world problems look like, even while it's practicing on the Saboteur's weird bugs.
Analogy: Imagine a firefighter training on a fake fire that burns blue. To make sure they don't get confused, the instructor occasionally throws in a real fire that burns orange and smoky, so the firefighter learns to handle both.

The Results

The researchers tested this new method on a new benchmark called BUGSOURCEBENCH, which contains bugs from three sources:

Humans.
Humans editing AI code.
AI code generated by other AIs.

The Outcome:

Standard Self-Play: Got good at fixing the weird, fake bugs but got worse at fixing real human bugs.
Anchored Self-Play (ASP): Got better at fixing everything. It improved the fix rate by 24% compared to the standard method. It worked well on bugs made by humans, bugs made by AI, and humans editing AI code.

Summary

The paper argues that if you just let an AI play "break and fix" without guidance, it will learn to break things in unrealistic ways that don't help it in the real world. By "anchoring" the training to a small set of real examples and rewarding the AI for making realistic mistakes, you can create a much more robust and helpful code-repair tool.

Technical Summary: Anchored Self-Play for Code Repair

Problem Formulation

Code repair is a critical capability for language models (LMs) in programming workflows, requiring the model to generate a fixed program that passes unit tests given a buggy program and the test suite. While high-quality real-world repair data is limited, recent work has explored scaling supervision by using LMs to generate bug-fix tasks. The authors investigate generator–fixer self-play, a setting where a single model is trained via reinforcement learning to alternate between generating bugs (by editing correct code) and fixing them.

The core challenge identified is distribution drift. While unit tests effectively verify functional correctness (pass/fail), they do not verify the realism of the bug. In standard self-play, the generator can drift toward producing "difficult but unrealistic" bugs—edits that fail tests but do not resemble errors encountered in practice (e.g., human-written code or human-edited LM code). Consequently, while the fixer improves on the generator's synthetic distribution, its performance often degrades on realistic bug sources.

Methodology

1. Generator–Fixer Self-Play

The authors instantiate a self-play loop where a single policy $\pi_\theta$ plays two roles:

Generator ( $G$ ): Edits a correct reference implementation $c^\star$ to produce a buggy program $b$ .
Fixer ( $F$ ): Repairs $b$ given the task description and unit-test feedback $o(x, b)$ .

Rewards:

Fixer Reward: Binary reward based on whether the repaired program $y$ passes all unit tests ( $v(x, y) = 1$ ).
Generator Reward: Designed to encourage bugs that are valid (executable and failing tests) and of intermediate difficulty. The difficulty is estimated by the fix rate of $K$ independent repair attempts. Bugs that are too easy (always fixed) or too hard (never fixed) are penalized to maintain an adaptive curriculum.

The system is optimized using GRPO (Group Relative Policy Optimization).

2. The Failure Mode: Distribution Drift

Experiments show that standard self-play exhibits co-evolutionary dynamics where the generator produces increasingly difficult bugs for a fixed fixer, and the fixer improves on those specific bugs. However, when evaluated on realistic sources (human-written, human-edited LM, and LM-generated bugs), performance regresses. The fixer overfits to the idiosyncratic, unrealistic edits generated by the self-play loop.

3. Proposed Solution: Anchored Self-Play (ASP)

To mitigate drift, the authors propose Anchored Self-Play (ASP), which anchors the self-play process to a small reference set of bugs ( $D_{ref}$ ) sampled from realistic sources. ASP employs two complementary mechanisms:

Reference Mixing for Fixer Training: During training, a fraction of the fixer's input bugs are replaced with bugs from the reference set ( $D_{ref}$ ). This ensures the fixer is continuously exposed to realistic bug patterns, preventing overfitting to the generator's evolving synthetic distribution. The generator is not updated on these mixed episodes to avoid off-policy issues.
Similarity-Guided Generator Shaping: An auxiliary reward is added to the generator's objective to nudge it toward producing bugs similar to the reference set.
- The generator computes the unified diff between the reference solution and the generated bug.
- This diff is embedded using a code embedding model (voyage-code-3).
- The reward is based on the cosine similarity between the generated bug's embedding and the $k$ -nearest neighbors in the reference pool.
- A centered similarity reward is used: $\delta_t(b) = \text{sim}_{01}(b) - B_t$ , where $B_t$ is a running baseline. This shapes the generator toward reference-like edits without overwhelming the primary unit-test reward.

Key Contributions

Problem Formulation: The authors formalize generator–fixer self-play for code repair and identify distribution drift as a primary failure mode when relying solely on unit-test rewards.
Benchmark (BUGSOURCEBENCH): They introduce and release BUGSOURCEBENCH, a controlled benchmark for evaluating cross-source repair generalization. It spans four bug sources while holding the underlying tasks, specifications, and unit tests constant:
- HUMAN: Bugs introduced by annotators into human-written reference solutions.
- HUMAN-EDITED LM: Bugs in LM-generated code that were subsequently edited by humans.
- LM ERRORS (QWEN-7B): Errors from a weaker code model.
- LM ERRORS (GPT-OSS-20B): Errors from a stronger code model.
Method (ASP): They propose Anchored Self-Play, which combines embedding-similarity guidance for bug generation with reference-mixed fixer training to stabilize self-play and improve generalization.

Experimental Results

The authors evaluate ASP against a Base Model, a Fixer-Only baseline (frozen generator, trained fixer), and standard Self-Play on BUGSOURCEBENCH.

Overall Performance: ASP achieves the highest average fix rate across all bug sources. It improves the average fix rate by +7.0 percentage points (absolute) and 24% (relative) over standard self-play.
Cross-Source Generalization:
- LM-Generated Bugs: ASP shows massive gains, improving fix rates by +11 pp (100% relative) on average for LM-generated bugs.
- Human-Originated Bugs: ASP improves fix rates by +3.4 pp (7.1% relative) on human-authored bugs, whereas standard self-play often degrades performance on these sources.
Ablation Studies:
- Both reference mixing and similarity shaping contribute to performance, but their combination yields the largest gains, suggesting they address complementary failure modes.
- The method is sample-efficient, showing improvements with as few as 50 reference examples.
- Using a mixed reference pool (covering all bug sources) yields the best overall performance compared to pools restricted to a single source.
Robustness: ASP's gains are robust across different embedding models (voyage-code-3 vs. CodeBERT), base models (Qwen-7B vs. DeepSeek-6.7B), and task distributions (transfer to DebugBench).

Significance and Claims

The paper claims that unit-test-only self-play is insufficient for scaling code repair because it lacks an explicit signal for realism, leading to distribution drift. By introducing Anchored Self-Play, the authors demonstrate that a small reference set can effectively anchor the generation process, allowing the model to benefit from the adaptive curriculum of self-play while maintaining robustness to realistic bug sources.

The work highlights that repair capabilities are distinct from code generation; models that solve tasks from scratch may fail to repair existing solutions, and vice versa. The proposed method offers a pathway to build more reliable automated programming tools that generalize across the heterogeneous sources of errors found in real-world LM-assisted programming.

Limitations: The authors note that their experiments focus on Python function-level repair with unit-test feedback. The setting does not capture repository-level challenges (multi-file faults, dependency management) or long-horizon tool use. Additionally, the realism signal relies on a frozen embedding model and a finite reference pool, suggesting that future work could explore learned critics or preference-based rewards.