🤖 AI

How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks

This paper demonstrates that iterative self-repair significantly boosts code generation accuracy across diverse modern LLMs (including Llama 3.1/3.3/4 and Gemini 2.5 families) on HumanEval and MBPP benchmarks, revealing that most gains occur within the first two attempts, assertion errors remain the most challenging to fix, and even smaller 8B models can effectively self-correct using prompting alone without fine-tuning.

Original authors: Johin Johny Arimbur

Published 2026-04-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Johin Johny Arimbur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: "Don't Just Guess, Check Your Work"

Imagine you are asking a very smart, but slightly impatient, robot to write a recipe for you.

The Old Way (Single-Shot): You ask the robot for a recipe, it gives you one, and you have to take it or leave it. If the recipe says "add 50 cups of salt," you're stuck with a salty disaster. Most previous studies only looked at this "one-and-done" attempt.
The New Way (Self-Repair): You ask the robot for a recipe. It writes one. You try to cook it, and it tastes terrible. Instead of giving up, you tell the robot, "Hey, this is too salty." The robot reads your complaint, thinks, "Oh, I meant 1 teaspoon," and writes a new recipe. You try again. If it's still wrong, you complain again, and it fixes it a third time.

This paper asks: Does this "try, fail, fix, try again" loop actually work for modern AI models? And does it work for all models, even the smaller, cheaper ones?

The Cast of Characters (The Models)

The researchers tested seven different AI models (the "robots"). Think of them as different types of students:

The Small Students: Like an 8-billion-parameter model (Llama 3.1 8B). They are fast and cheap but make more mistakes.
The Big Students: Like the 70-billion-parameter models (Llama 3.3 70B, Gemini 2.5 Pro). They are slow and expensive but usually get it right the first time.
The Specialized Students: Models like "Scout" and "Maverick" that use a "Mixture of Experts" (MoE) architecture. Imagine a student who has 16 or 128 different "mini-brains" inside them, only waking up the specific one needed for the task.
The Speedsters: Like Gemini 2.5 Flash, designed to be incredibly fast.

They tested these students on two types of homework:

HumanEval: Short, standard coding puzzles (like a pop quiz).
MBPP: A slightly larger set of basic programming problems.

The Main Findings: The "Magic" of Feedback

The results were surprisingly positive. Here is what they found, using our kitchen analogy:

1. Everyone Gets Better with Feedback
In the past, researchers thought small or "weaker" AI models would just get more confused if you told them they were wrong. They thought the robot would just make up a new, worse recipe.

The Discovery: That's not true anymore! Every single model, from the tiny 8B one to the massive 70B one, got significantly better when allowed to fix its own mistakes. Even the small models improved by a huge margin. It's like a student who, when told "check your math," actually learns to check their work and gets an A.

2. The "First Two Tries" Rule
You might think you need to let the robot try 10 times to get it right.

The Discovery: You don't. 90% of the improvement happens in the first two tries.
- Try 1: The robot writes code. (Maybe it fails).
- Try 2: The robot fixes the obvious errors. (Huge improvement!).
- Try 3 & 4: The robot fixes the tiny, tricky bugs. (Small improvements).
- Try 5: The robot is just spinning its wheels.
- Lesson: If you are building an app, just let the AI try twice. It's the sweet spot between cost and quality.

3. Not All Mistakes Are Created Equal
The researchers looked at why the robots failed.

The Easy Fixes: If the robot forgot to define a variable (a "Name Error") or messed up the punctuation (a "Syntax Error"), it fixed these almost instantly. It's like the robot saying, "Oh, I forgot the comma! My bad."
The Hard Fixes: If the robot wrote code that ran perfectly but gave the wrong answer (an "Assertion Error"), it struggled. This is like the robot writing a recipe that looks perfect but results in a burnt cake. The robot can't "see" the logic error as easily as it sees a missing comma.
- Success Rate: It fixed "Name Errors" about 77% of the time, but only fixed "Logic Errors" about 45% of the time.

4. The "Chain of Thought" Superpower
The researchers tried a trick: instead of just saying "Fix this," they asked the robot to explain its thinking first before writing the code.

The Discovery: For the smarter models, this "explain your work" step was a game-changer. It was like asking a student to show their work on a math test. The smarter the model, the more it benefited from this. The tiny models didn't gain much, but the big ones jumped ahead significantly.

5. Repair vs. Resampling (The "Lottery" vs. The "Tutor")
There are two ways to get a better answer from an AI:

Resampling (The Lottery): Ask the AI to write the code 5 different times at once and pick the best one. This uses a lot of computer power (tokens).
Self-Repair (The Tutor): Ask the AI to write it once, see the error, and fix it.
The Discovery: For smart models, Self-Repair is much cheaper and often better. It's like hiring a tutor to fix one bad essay vs. hiring 5 different students to write 5 different essays. The "Tutor" approach (Self-Repair) saved a massive amount of money (computer tokens) while getting equal or better results.

The "MoE" Mystery

The paper also compared "Dense" models (one big brain) vs. "MoE" models (many small experts).

The Result: The "MoE" models (like Llama 4 Scout) were surprisingly good at fixing their own mistakes, sometimes beating the much larger "Dense" models. It suggests that having specialized "experts" inside the model helps it diagnose errors better.

The Bottom Line for Humans

If you are a developer or a business owner using AI to write code:

Don't settle for the first try. Always let the AI fix its own errors.
Stop at two. Letting it try more than twice usually isn't worth the extra cost.
Ask it to think. If you have a powerful model, tell it to "explain the bug before fixing it."
It works for everyone. You don't need a super-expensive, massive model to get good results; even smaller, cheaper models can learn from their mistakes if you give them the chance.

In short: AI isn't just a "guess and check" machine anymore. It's becoming a "try, learn, and improve" machine, and we don't need to teach it how to do that—it just needs the right prompt to start the conversation.

1. Problem Statement

Large Language Models (LLMs) frequently fail to generate correct code on their first attempt. However, standard benchmarks (e.g., HumanEval, MBPP) typically evaluate models using a single-shot metric (pass@1), which diverges from real-world software development where programmers iteratively debug and refine code.

Previous research (e.g., Olausson et al., 2023) suggested that iterative self-repair (feeding execution errors back to the model) was not a "silver bullet," particularly for smaller or weaker models, which often degraded in performance or required fine-tuning to succeed. With the emergence of powerful 2024–2025 instruction-tuned models and new architectures (like Mixture-of-Experts, or MoE), the authors investigate whether self-repair is now universally effective without fine-tuning, how different architectures compare, and what the practical limits and costs are.

2. Methodology

The study employs a systematic, empirical evaluation across seven models from three families (Meta, Alibaba, Google) and two benchmarks.

Models Evaluated:
- Open-Weight (via Groq API): Llama 3.1 8B (Dense), Llama 3.3 70B (Dense), Llama 4 Scout (MoE, 16 experts), Llama 4 Maverick (MoE, 128 experts), Qwen3 32B (Dense).
- Proprietary (via Vertex AI): Gemini 2.5 Flash, Gemini 2.5 Pro.
Benchmarks:
- HumanEval: 164 Python programming problems.
- MBPP Sanitized: 257 Python programming problems.
Self-Repair Protocol:
- Round 0 (R0): Initial code generation using greedy decoding ( $T=0$ ).
- Repair Rounds (R1–R4): If execution fails, the model receives the original problem, its previous code, and the error traceback. It is asked to generate a corrected solution.
- Termination: The process stops if the code passes all tests or after 5 total attempts.
Metrics:
- Cumulative Pass@1: The fraction of problems solved by round $R_i$ .
- Self-Repair Gain ( $\Delta$ ): The difference between final pass rate ( $R4$ ) and initial pass rate ( $R0$ ).
- Token Efficiency: Comparison of token usage between self-repair and independent resampling.
Ablation Studies:
- Prompt Strategies: Minimal (error only) vs. Explain-then-fix vs. Chain-of-Thought (CoT).
- Error Analysis: Categorization of failures (Syntax, Name, Assertion, etc.) to determine repair success rates.

3. Key Contributions

Universal Effectiveness of Prompt-Based Repair: The study demonstrates that modern instruction-tuned models (even at 8B scale) benefit from self-repair using prompting alone, without the need for fine-tuning. This updates prior findings that weaker models are harmed by self-repair.
Dense vs. MoE Comparison: It provides the first direct comparison of dense and Mixture-of-Experts architectures for self-repair, revealing distinct interaction patterns (e.g., MoE models like Scout showing high repair gains).
Error-Type Granularity: The paper identifies that Assertion Errors (logical mistakes) are the hardest to repair (~~45% success rate), while Name Errors are the easiest (~~77% success rate).
Diminishing Returns & Deployment Guidelines: It quantifies that the majority of gains (76–95%) are captured within the first two repair rounds, offering clear guidelines for cost-effective deployment.
Prompt Ablation: It shows that Chain-of-Thought (CoT) prompting yields significant additional gains (up to +5.5 pp) for capable models compared to minimal prompting.
Repair vs. Resampling: It extends the exploration-exploitation tradeoff analysis, showing that for capable models, self-repair is more token-efficient than generating multiple independent samples.

4. Key Results

Performance Gains

Universal Improvement: All seven models improved on both benchmarks.
- HumanEval: Gains ranged from +4.9 pp (Qwen3 32B) to +17.1 pp (Gemini 2.5 Pro).
- MBPP: Gains were even larger, ranging from +16.0 pp to +30.0 pp (Gemini 2.5 Flash).
Top Performers:
- Gemini 2.5 Flash achieved the highest final pass rates: 96.3% on HumanEval and 93.8% on MBPP.
- Gemini 2.5 Pro showed the largest gain (+17.1 pp on HumanEval) but started from a lower baseline due to reasoning overhead interfering with code extraction.
- Llama 4 Scout (MoE) achieved the highest gain among open-weight models on HumanEval (+14.0 pp).

Error Analysis

Assertion Errors: The dominant failure mode. Repair success is low (~45% on HumanEval) because the model must re-evaluate logic without explicit code errors.
Name/Syntax Errors: High repair success rates (~77% and ~66% respectively) due to clear diagnostic signals.
MBPP vs. HumanEval: MBPP problems (shorter, focused) yielded higher repair success rates for assertion errors (~~63%) compared to HumanEval (~~45%).

Prompt Ablation (CoT vs. Minimal)

Chain-of-Thought (CoT) consistently outperformed minimal prompting.
For Llama 3.3 70B, CoT provided an additional +5.5 pp gain over minimal prompting.
For Qwen3 32B, CoT achieved the highest absolute pass rate in the ablation (97.0% at R2).

Token Efficiency (Repair vs. Resampling)

Self-repair is more token-efficient for capable models.
Gemini 2.5 Flash and Llama 3.3 70B achieved higher pass rates with self-repair using less than half the tokens compared to independent resampling (5 samples).
Llama 3.1 8B was the exception where resampling slightly outperformed repair, suggesting that for very weak models, diversity of samples is more valuable than error feedback.

5. Significance and Implications

Paradigm Shift: The paper challenges the notion that self-repair is ineffective for smaller models. It establishes that the 2024–2025 generation of instruction-tuned models has closed the gap, making self-repair a viable, training-free strategy even for 8B models.
Practical Deployment:
- Two Rounds are Enough: For most applications, running two repair rounds captures the vast majority of benefits, optimizing the cost-performance tradeoff.
- Prompt Engineering: For capable models (≥32B), using CoT prompts during repair is highly recommended.
- Architecture Choice: MoE architectures (like Scout) show strong potential for repair tasks, while proprietary models (Gemini Flash) currently set the performance ceiling.
Limitations of Self-Correction: The study reinforces that LLMs struggle to self-correct logical reasoning (assertion errors) without external feedback, highlighting a fundamental limit in current self-debugging capabilities.

In conclusion, this paper provides a comprehensive roadmap for deploying iterative self-repair in production LLM code generation systems, demonstrating that it is a robust, universally beneficial technique that significantly outperforms single-shot generation and is increasingly efficient compared to resampling strategies.