💻 computer science

Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models

This paper demonstrates that for small code models, blind resampling (retrying without feedback) outperforms or matches standard self-repair methods while using significantly fewer tokens, because conditioning on a model's own failed attempt causes it to "anchor" to the original error rather than effectively utilizing execution feedback.

Original authors: Yuvraj Verma

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuvraj Verma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write computer code. You give the robot a task, like "write a program that adds two numbers," and it tries to do it. Sometimes, the robot makes a mistake, and the code doesn't work. In the world of artificial intelligence, a popular idea is "self-repair." This is like telling the robot, "Hey, you messed up. Here is your broken code and the error message. Now, look at it, think about what went wrong, and try to fix it." It sounds like a smart way to learn from mistakes, right? But there's a catch. When we test if this "fixing" works, we often compare it to a robot that just gives up after one try. That's a bit unfair because the robot that gets a second chance has a better shot at success simply because it tried again, not necessarily because it learned from its error. To really know if "fixing" is the magic ingredient, we need to compare it to a robot that gets a second chance but doesn't see its old mistake. This paper dives into that exact question: Is it better to look back at a failure and try to fix it, or is it better to just forget the failure and try a brand-new approach from scratch?

The researchers in this paper decided to play a game of "Try Again, Don't Look Back" with some small AI models. They set up a clever experiment using a "placebo" test, which is usually something you do in medicine to see if a pill works or if it's just the hope of taking a pill that helps. Here, they compared four different ways a robot could try to fix its code:

Blind Resampling: The robot gets a second chance but is told nothing about the first try. It's like asking a student to take a math test again without showing them the first one.
The Placebo: The robot sees its broken code and a simple note saying, "This is wrong," but no details on why.
Real Feedback: The robot sees the broken code and the specific error message (like "you forgot a semicolon").
Reflection: The robot sees the error and is asked to write a paragraph explaining what went wrong before trying to fix it.

They ran this test on three different sizes of AI models (small, medium, and slightly larger) and watched how many times they got the code right.

Here is the surprising twist: Looking back at the mistake actually made the small robots worse.

For the smaller models (1.5 billion and 3 billion parameters), the "Blind Resampling" strategy was the clear winner. It was not only the most accurate but also the cheapest in terms of computer time and energy. When the researchers forced the robots to look at their own failed code (the "Placebo" or "Feedback" groups), the robots got stuck. They tended to make tiny, almost invisible changes to the broken code instead of thinking of a completely new solution. The paper calls this "anchoring." It's like if you try to draw a cat, mess it up, and then try to fix it by just erasing a few lines of the bad drawing. You end up with a weird, half-fixed cat. But if you just put the paper down and start a fresh sheet, you might draw a totally different, better cat.

The researchers found that when the robots looked at their own failed attempts, they reproduced a nearly identical program 33% to 68% of the time. In contrast, when they tried blindly (without looking), they only repeated themselves 2% to 14% of the time. The "information" in the error messages didn't help at all; a simple "this is wrong" note was just as bad as the full error report. Even asking the robot to "think out loud" about the mistake (Reflection) didn't save the day; it just made the process much more expensive without improving the results enough to matter.

However, there is a glimmer of hope for bigger brains. When they tested the largest model (7 billion parameters), the penalty for looking back almost disappeared. The "Blind Resampling" and the "Self-Repair" methods ended up in a statistical tie. This suggests that as AI gets smarter, it becomes better at not getting stuck on its own mistakes. But for the smaller, cheaper models that many people actually use, the advice is clear: Don't look back.

The paper also ruled out a few other ideas. They checked if the problem was just because the prompts got too long and confused the robots, but adding code from other successful tasks didn't hurt performance. This proved the problem wasn't about length, but specifically about the robot looking at its own failure. They also checked if the problem was just a glitch in how the computer stored the numbers (quantization), but the result stayed the same even with higher precision.

So, what's the takeaway? If you are using a small AI to write code, don't waste time showing it its mistakes and asking it to fix them. It will likely just get stuck in a loop of tiny, useless edits. Instead, just ask it to try again from scratch. It's faster, cheaper, and surprisingly, it works better. The cost of "fixing" a bad first attempt is just the cost of being stuck with a bad first attempt. Sometimes, the best way to move forward is to forget the past and start fresh.

Technical Summary: Blind Resampling Outperforms Self-Repair in Small Code Models

Problem Statement

The paper addresses a fundamental flaw in the evaluation of self-repair mechanisms in code-generating agents. Standard evaluations compare self-repair (retrying with execution feedback) against a single-shot baseline. The authors argue this comparison is confounded: it conflates the value of the feedback information with the value of the extra attempt itself. Under stochastic decoding, simply retrying a task multiple times increases the probability of success regardless of whether the retry is informed. The paper posits that without a "placebo" control that accounts for the cost of an additional attempt, the apparent benefits of self-repair may be illusory.

Methodology

The study employs a placebo-controlled experimental design on the MBPP+ benchmark (378 tasks) across three model scales (1.5B, 3B, and 7B) using Qwen2.5-Coder models.

Experimental Conditions

When a program fails a test, the system appends one of four suffixes to the original instruction to trigger a repair attempt. All conditions share the same token budget constraints and decoding parameters (temperature 0.8 for retries, greedy for the initial attempt):

Resample (Blind Control): The original prompt is re-sampled with no reference to the failed attempt. This isolates the value of an additional draw.
Placebo: The failed program is appended with a content-free notice that it was incorrect. This isolates the value of re-reading one's own attempt.
Feedback: The failed program is appended with the actual execution output (failing input, expected vs. observed values). This isolates the value of diagnostic information.
Reflect: The failed program and execution output are preceded by an instruction to diagnose the failure in prose before rewriting. This tests the value of explicit verbal reasoning.

Evaluation Metrics

Performance: pass@1 at $k=8$ iterations, measured via differential testing against reference implementations on EvalPlus.
Cost: Output token consumption.
Mechanism Analysis: Textual similarity (longest-matching-subsequence ratio) between consecutive attempts to measure "anchoring."
Statistical Rigor: McNemar's exact test for paired contrasts, bootstrap confidence intervals, and Holm correction for multiple comparisons. Experiments were pre-registered and run on a single consumer GPU.

Key Results

1. Blind Resampling Dominates Small Models

At 1.5B and 3B scales, blind resampling (re-trying without seeing the failure) achieves the highest pass rates and is the most token-efficient strategy.

1.5B: Self-repair conditions (placebo, feedback, reflect) are strictly Pareto-dominated by blind resampling, performing worse in accuracy and costing significantly more tokens. Specifically, conditioning on the model's own failed attempt (placebo) reduces accuracy by 6.1 points ( $p=0.006$ ) compared to blind resampling.
3B: Similar trends hold; the penalty for self-conditioning is 6.9 points ( $p<0.001$ ).
7B: The penalty narrows, and blind resampling becomes statistically tied with the best self-repair conditions. However, the "Reflect" condition only gains a non-significant +0.8 points over blind resampling while consuming 2.5× the tokens.

2. Execution Feedback Adds No Measurable Value

The informational content of execution feedback is indistinguishable from a content-free placebo.

The contrast between Feedback and Placebo is statistically null across all scales (differences of +0.000, +0.026, and +0.008).
Practitioners' assumption that execution feedback is the "active ingredient" in self-repair is refuted; the mere act of showing the model its own failed code is the primary driver of performance changes.

3. The Mechanism: Anchoring

The paper identifies anchoring as the mechanism causing the performance drop in self-repair.

When shown its own failed code, models reproduce near-identical programs in 33–68% of retries, compared to only 2–14% under blind resampling.
The model biases generation toward local edits of the failed program rather than exploring new regions of the solution space.
Verbal Reflection partially mitigates anchoring (reducing similarity by 0.04–0.12) but remains cost-ineffective compared to blind resampling.

4. Specificity to Self-Conditioning

Cross-Task Experience: Prepending retrieved solutions from other tasks (successful exemplars) has no effect (bounded to ±3.5 points). This confirms the harm is specific to self-conditioning (seeing one's own failure), not a generic consequence of longer context windows.
Replication: The effect persists across model families (DeepSeek-Coder) and precisions (FP16 vs. Q4 quantization), ruling out quantization artifacts or specific pre-training distributions as causes.

5. The Role of Baseline Capability

The magnitude of the anchoring penalty is strongly predicted by the model's baseline quality ( $r=0.96$ ), not just model scale or family.

The penalty is the "cost of committing to a bad first attempt."
As baseline capability improves (e.g., at 7B), the first attempt is less likely to be "bad," making the cost of anchoring negligible. The apparent scale trend is thus a capability effect.

Significance and Claims

The paper claims to fundamentally shift the understanding of iterative self-correction in code agents:

Methodological Correction: It argues that capabilities added to agent prompts must be evaluated against equally expensive, uninformative controls (like blind resampling) rather than single-shot baselines. Without this, the "benefit" of feedback is often just the benefit of persistence.
Practical Guidance for Small Models: For models below 7B, the optimal strategy is to retry blindly rather than attempt self-repair. Agents should cap iteration budgets near $k \approx 2$ and treat execution feedback as a costly resource rather than a default mechanism.
Economic Impact: Adopting blind resampling over self-repair for small models yields equal or better accuracy with significantly reduced token consumption (2.5–5.5× fewer tokens), leading to direct reductions in compute and energy costs.
Theoretical Insight: The study demonstrates that the "harm" of self-repair in small models is not a lack of reasoning ability but a cognitive bias (anchoring) that prevents the model from abandoning a failed trajectory. This bias diminishes as the model's initial generation quality improves.

The authors conclude that for small code models, "Try Again, Don't Look Back" is the superior strategy, and that the transition point where self-repair becomes viable coincides with the point where the first attempt is sufficiently high-quality that anchoring no longer incurs a penalty.