💻 computer science

Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation

The paper demonstrates that for 1-3B parameter code generation, incorporating execution feedback through a simple self-refinement loop provides significantly more performance gains than increasing pipeline complexity or architectural topology.

Original authors: Charles Junichi McAndrews

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Charles Junichi McAndrews

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Smart Assistant" Problem: Why a Good Editor is Better Than a Fancy Office

Imagine you have a very talented but slightly scatterbrained junior intern. This intern is great at following instructions, but they often make silly mistakes—like forgetting to put a lid on a jar or misspelling a client's name.

Now, you have two ways to help them:

The Fancy Office Approach: You build them a massive, high-tech office with complex filing systems, multiple meeting rooms, and a complicated workflow of different departments.
The "Check Your Work" Approach: You just give them a red pen and a way to test if their work actually works (like a "test kitchen" where they can taste the soup before serving it).

This paper, written by Charles Junichi McAndrews, proves that for small, "junior-level" AI models, the red pen is much more important than the fancy office.

The Core Discovery: Feedback is the Secret Sauce

The researcher looked at "Small Language Models" (the AI equivalent of a junior intern). On their own, these models struggle with complex coding tasks. The big question was: Can we make them smarter by linking them together in a complex "pipeline" (a chain of command)?

The researcher used an evolutionary algorithm (kind of like digital Darwinism) to try and "evolve" the perfect chain of command. They tried adding different models, different steps, and different "bosses."

The result? The most complex, evolved "offices" weren't much better than a very simple loop:
Write Code $\rightarrow$ Run Code $\rightarrow$ If it breaks, fix it.

The "secret sauce" wasn't the structure of the team; it was the Execution Feedback.

1. The "Taste Test" (Execution Feedback)

If the AI writes code and it fails, the computer gives it a "traceback"—a specific error message that says, "Hey, you forgot to define the variable 'X' on line 5."

This is like a chef being told, "This soup is too salty." They know exactly what to do! The paper found that this feedback allows small models to fix "runtime errors" (the silly mistakes) almost perfectly.

However, if the error is a "logic error" (the soup is perfectly salted, but it's actually tomato soup and the customer wanted onion), the AI struggles. It knows the customer is unhappy, but it doesn't know why.

2. The "Editor vs. The Writer" (Refiner vs. Generator)

The study found something surprising: The person fixing the mistake is more important than the person who made it.

If you pair a "weak" writer (a 1.5B parameter model) with a "strong" editor (a 3B parameter model), they perform just as well as a single strong model doing both jobs. In the world of small AI, a great editor can save a mediocre writer, but a great writer can't always save a bad editor.

3. The "Don't Touch It If It's Working" Rule (Early Stopping)

This is a crucial warning. The researcher found that if you force the AI to keep "improving" its code even after it has already passed the tests, it actually starts making mistakes.

It’s like an editor who is so eager to work that they start "fixing" a sentence that was already perfect, eventually turning it into gibberish. The paper calls this the "Early Stopping Paradox": you must tell the AI to stop the moment it gets it right, or it will break its own success.

The "Too Long; Didn't Read" Summary

If you are building small, local AI to help you code:

Don't waste time building a massive, complicated web of different AI agents.
Do give the AI a "test kitchen" where it can run its code and see exactly where it failed.
Invest in a strong "Editor" model to review the work.
Tell the AI to shut up the moment the code actually works.

The takeaway: In the world of small AI, a simple loop with a "reality check" beats a complex hierarchy every single time.

Technical Summary: Feedback Over Form

Title: Feedback Over Form: Why Execution Feedback Matters More Than Pipeline Topology in 1-3B Code Generation
Author: Charles Junichi McAndrews (April 2026)

1. Problem Statement

Small Language Models (SLMs) in the 1–3B parameter range are highly practical for local, on-device deployment but lack the reasoning capabilities of larger models (70B+) for complex tasks like code generation. While research in larger models suggests that composing multiple models into "pipelines" or "agents" improves performance, it is unclear if this composition holds at the small scale. Specifically, the paper investigates whether increasing the complexity of the pipeline architecture (topology) or providing external execution feedback is the primary driver of performance gains for SLMs.

2. Methodology

The study employs a systematic approach to testing model composition for Python code generation on two benchmarks: HumanEval and Sanitized MBPP.

Pipeline Architecture: The researchers designed a constrained linear pipeline consisting of:
- Generator: Produces initial code.
- Executor: Runs code in a sandbox and returns pass/fail status and tracebacks.
- Analyzer (Optional): Diagnoses errors using natural language.
- Refiner: Attempts to fix the code based on feedback.
Evolutionary Search (NEAT-inspired): Instead of manual design, the authors used a modified NeuroEvolution of Augmenting Topologies (NEAT) algorithm to evolve pipeline configurations. The search space included model selection (from a pool of Gemma3:1B, Qwen2.5:1.5B, and Llama3.2:3B), number of refinement stages, presence of an analyzer, prompt selection, temperature, and iteration counts.
Evaluation Protocol: To account for the stochastic nature of LLMs, results were reported as 5-run means to ensure statistical significance. The authors also implemented early stopping, halting the pipeline immediately once code passes tests.

3. Key Contributions

Identification of the "Load-Bearing Mechanism": The paper proves that execution feedback (tracebacks) is the essential component that enables small-model composition to succeed.
The Early Stopping Paradox: The study identifies that without early stopping, refinement is "net-negative" because refiners frequently break code that is already correct.
Topology vs. Capability: The research demonstrates that at the 1–3B scale, the specific arrangement of nodes (topology) is less important than the capability of the model acting as the refiner.
Quantification of Evaluation Noise: The authors provide a critical warning for LLM research, showing that single-run evaluations can inflate results by 5–7%, potentially leading evolutionary searches to select "lucky" rather than "good" configurations.

4. Key Results

Significant Gains via Feedback: Self-refinement with execution feedback improved performance by >4σ (standard deviations) on both benchmarks compared to solo generation.
Error Taxonomy (The "Fixability" Gap):
- Runtime Errors (e.g., NameError, SyntaxError) have high fix rates because tracebacks provide explicit, actionable signals.
- Logic Errors (e.g., AssertionError) have low fix rates because the feedback only reports the symptom (wrong answer) without explaining the underlying cause.
Refiner > Generator: In the general-purpose pool, a 1.5B generator paired with a 3B refiner performed as well as a 3B model performing both roles, suggesting the refiner's capability is the bottleneck.
Evolutionary Convergence: The NEAT search mostly rediscovered simple "Generate $\rightarrow$ Execute $\rightarrow$ Refine" loops. The "NEAT Champion" provided only a marginal (non-significant) improvement over manual baselines, suggesting that complex topologies do not compensate for model limitations at this scale.
Specialization Wins: Code-specialized models (e.g., Qwen2.5-Coder) significantly outperformed all general-purpose pipeline configurations, indicating that model training/specialization is more impactful than architectural complexity.

5. Significance

This paper shifts the focus of SLM research from architectural complexity to signal quality. It concludes that for small models, the most effective way to boost performance is not to build elaborate multi-agent workflows, but to ensure the model has access to high-quality, explicit, and machine-verifiable feedback (like compiler or test tracebacks). This provides a clear roadmap for developers building local, resource-constrained AI: prioritize specialized models and robust execution-feedback loops over complex agentic topologies.