🤖 AI

Steer, Don't Solve: Training Small Critic Models for Large Code Agents

This paper proposes training small, supervised fine-tuned critic models to provide intra-trajectory steering for large code agents, demonstrating that this approach significantly improves performance on benchmarks like SWE-bench Verified while reducing both computational costs and trajectory lengths compared to end-to-end training or post-hoc scoring.

Original authors: Shubham Gandhi, Yiqing Xie, Atharva Naik, Ruichen Zhu, Carolyn Rose

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shubham Gandhi, Yiqing Xie, Atharva Naik, Ruichen Zhu, Carolyn Rose

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The GPS vs. The Driver

Imagine you are trying to drive a car from New York to Los Angeles. You have a very skilled driver (the Code Agent) who knows how to turn the wheel, press the gas, and change gears perfectly. However, this driver sometimes gets confused about the big picture. They might drive in circles, take a wrong turn because they misread a sign, or keep trying to drive through a wall because they think it's a door.

Usually, to fix this, people try to retrain the driver from scratch. But that's expensive, takes a long time, and the driver might still get stuck on the same confusing roads.

This paper proposes a different solution: Instead of retraining the driver, we hire a small, smart GPS navigator (the Critic Model).

The Driver (Agent): Does all the actual work (writing code, running commands). They stay exactly the same; we don't change their brain.
The GPS (Critic): Doesn't touch the steering wheel. Instead, it watches the driver every few minutes. If the driver starts going in circles or heading toward a cliff, the GPS says, "Hey, you're looping! You're trying to fix the wrong file. Stop and rethink your strategy."

The paper shows that this small GPS is much cheaper to hire than a super-smart human navigator, and it helps the driver get to the destination much faster and more often.

The Problem: Why Just "Training" Isn't Enough

The authors noticed that when you train big AI coding agents to solve software bugs, they get really good at the mechanics (typing code, running tools). But they often fail at the strategy (figuring out the right plan).

It's like teaching a student to write essays. You can teach them perfect grammar and spelling (the mechanics), but if they don't know how to organize their thoughts or stay on topic (the strategy), the essay will still be a mess.

When you try to train the AI to do both at once, it hits a "ceiling." It gets stuck. The authors found that about 65% of failures weren't because the AI wrote bad code, but because it had flawed reasoning.

The Solution: A "Small Critic"

The team built a system where a small AI model (the Critic) sits next to the big coding agent.

The Setup: The big agent tries to solve a problem. Every 5 or 10 steps, it pauses.
The Check-in: The small Critic looks at what the agent has done so far.
The Advice: The Critic gives a short, high-level tip.
- Bad advice: "Delete line 42 and write a loop here." (This is "backseat driving" and the paper says this is bad because the small model isn't smart enough to know the code better than the big one).
- Good advice: "You are repeating the same command three times. You are stuck in a loop. Try a different approach." (This is "steering").
The Resume: The big agent reads the advice and continues working.

How They Trained the GPS

To teach the small Critic how to give good advice, they didn't just guess. They used a "Teacher" (a massive, very expensive AI called Claude-Opus-4.6) to generate the advice first.

The Teacher: The expensive AI watched the agent work and wrote down perfect, high-level advice.
The Student: The small, cheap AI (Qwen3-8B) studied these notes. It learned to mimic the style of the advice (focusing on strategy, not specific code lines) without needing to be as smart as the Teacher.

Crucial Discovery: The paper found that the type of advice matters most.

If the Teacher gave specific code instructions, the small student couldn't learn it well.
If the Teacher gave high-level strategy (e.g., "You're stuck," "Check your assumptions"), the small student learned it perfectly.

The Results: Faster, Cheaper, and Smarter

The team tested this on a famous benchmark called SWE-bench, which is like a test of 500 real-world software bugs.

It Works on Different Drivers: They trained the Critic using one type of coding agent (CWM-32B). Then, they used that same Critic to help two different coding agents (Qwen models) that it had never seen before.
- Result: The new agents got significantly better at solving bugs just by listening to the Critic.
It Saves Money: The "Teacher" (Claude-Opus) is very expensive to run. The "Student" (Qwen3-8B) is tiny and cheap.
- Result: The system using the small Critic was 30 to 92 times cheaper than using the expensive Teacher.
It Actually Saves Time: On one of the big agents, the Critic didn't just make the agent smarter; it made the agent stop wasting time. The agent finished tasks faster, which saved even more money. In fact, the whole system (Agent + Critic) was cheaper than the Agent working alone.

The "Secret Sauce"

The paper highlights two main reasons why this worked:

Separation of Concerns: The Agent focuses on doing (coding), and the Critic focuses on thinking (strategy). They don't fight over who is in charge.
No Backseat Driving: The Critic is strictly forbidden from saying "Write this specific line of code." It only says "You are going the wrong way." This prevents the small model from giving bad, specific instructions that confuse the big model.

Summary

Think of this paper as a blueprint for building a team rather than a super-hero. Instead of trying to make one giant, perfect AI that does everything (which is hard and expensive), they built a system where a small, specialized coach guides a powerful worker.

The coach doesn't do the work, but by giving the right high-level nudges, the worker solves problems they couldn't solve alone, and the whole team finishes the job faster and for less money.

Technical Summary: Steer, Don't Solve

Problem Statement

End-to-end training of code agents is resource-intensive and often plateaus regarding the strategy-level reasoning required to resolve complex code issues. Empirical evidence suggests that while agents can learn low-level implementation skills (executing tool calls, editing code), their high-level reasoning (troubleshooting, planning, search) remains underdeveloped when jointly optimized with execution. Approximately 65% of code agent failures stem from flawed reasoning rather than incorrect actions. Current post-training methods that couple strategy-level reasoning with implementation behavior tend to saturate quickly, leaving the agent unable to generalize patterns effectively. Furthermore, existing critic models are typically post-hoc, scoring completed trajectories rather than providing intra-trajectory guidance to steer an agent away from failure during execution.

Methodology

The authors propose a "Steer, Don't Solve" paradigm: freezing the code agent and training a separate, small critic model to provide high-level, strategy-level feedback during the agent's execution.

Core Architecture

Frozen Agent ( $\pi_A$ ): The code agent (e.g., CWM-32B, Qwen3 variants) remains frozen throughout the process. It is responsible for all concrete actions, tool calls, and code edits.
Critic Intervention: Every $k$ steps (e.g., $k=5$ or $10$), a critic model $C$ inspects the partial trajectory $\tau_{1:t}$ and the agent's history.
Feedback Loop: The critic generates a short textual critique $f_t$ which is appended to the agent's context before its next step. The critic does not interact with the environment directly.

Critic Design & Training

Teacher Distillation: A frontier model (Claude-Opus-4.6) acts as a teacher critic ( $C_T$ ) to generate training data. The student critic ( $C_\theta$ ) is a smaller model (Qwen3-8B) trained via Supervised Fine-Tuning (SFT) on these critiques.
Concise vs. Detailed Prompts: The paper identifies a critical design choice regarding the feedback format.
- Detailed Prompt: Allows the teacher to provide specific, action-level instructions (e.g., "run this command," "edit this file"). While effective for the strong teacher, this leads to "backseat driving" where the agent simply copies the teacher's solution rather than reasoning.
- Concise Prompt: Constrains the teacher to high-level feedback only. It forbids specific code changes or technical approaches, focusing instead on error taxonomy (e.g., "Step Repetition," "Task Derailment") and strategic redirection. The trained student critic is distilled from this concise prompt to ensure it learns to steer rather than dictate.
Budget Awareness: During inference, once the agent approaches its step limit (e.g., step 100 of 150), budget-aware instructions are appended to the critic's input, prompting it to prioritize recommending submission if the agent is stuck or has a fix.
Training Data: Trajectories are collected by running the frozen agent with the teacher critic intervening. The training corpus uses a "concise" supervision signal. To ensure transferability, the authors collect data from one agent (CWM-32B) and test on others, or mix trajectories from target agents into the corpus.

Evaluation Setup

Benchmark: SWE-bench Verified (500 human-verified GitHub issues).
Agents Tested: CWM-32B (dense), Qwen3-32B (dense), and Qwen3-Next-80B-A3B (MoE).
Metrics: Resolve Rate (passing hidden tests), Localized Rate, Non-Empty Rate, and Cost (USD per instance).
Fallback Mechanism: If a critic-guided run fails to submit within the budget, a base run (no critic) is executed as a fallback to ensure fair cost comparison.

Key Results

Performance Gains: The trained critic significantly improves the resolve rate of frozen agents.
- On CWM-32B, a critic trained on CWM-32B trajectories alone improved resolve rate by +3.8 points (29.2% $\to$ 33.0%).
- On Qwen3-32B, the same critic (trained only on CWM-32B data) improved resolve rate by +3.4 points (9.6% $\to$ 13.0%).
- On Qwen3-Next-80B-A3B, the gain was +3.0 points (20.8% $\to$ 23.8%).
- Adding target-agent trajectories to the training corpus further increased gains, reaching +5.2 points on Qwen3-32B and +4.4 points on Qwen3-Next-80B-A3B.
Transferability: A critic trained on a single agent's trajectories transfers effectively to unseen agents with different architectures (dense vs. MoE) and sizes, suggesting the learned steering signal is agent-agnostic.
Cost Efficiency:
- The trained 8B critic costs 30–92 $\times$ less than the Claude-Opus-4.6 teacher.
- Crucially, on the Qwen3-Next-80B-A3B agent, the critic-guided system is cheaper end-to-end ($0.04 vs. $0.11) than the unguided agent. This is because the critic shortens the agent's trajectories (reducing agent compute) enough to offset the cost of the critic calls.
Ablation Findings:
- Training is essential: An off-the-shelf (untrained) 8B critic provided no performance gain over the baseline.
- Prompt format matters: While the teacher performed better with detailed prompts, the trained student performed better with concise prompts. The detailed prompt caused the teacher to "backseat drive," a behavior the 8B student could not faithfully replicate or generalize, leading to lower performance when distilled.

Significance and Claims

The paper claims to be the first study of training critic models specifically for code agents as a separation-of-concerns alternative to scaling the agent itself.

Practical Complement: The results demonstrate that a small, well-trained critic is a practical and cost-effective complement to scaling agent training. It recovers strategy-level reasoning capabilities that end-to-end training leaves underdeveloped.
Separation of Concerns: By decoupling strategy (critic) from execution (agent), the system learns simpler, more generalizable patterns. The critic handles high-level diagnosis and redirection, while the agent handles implementation.
Efficiency: The approach achieves Pareto-optimal improvements, offering higher accuracy at lower costs compared to both unguided agents and systems using expensive frontier models as critics.

The authors conclude that training critic models is a viable path forward for improving code agents without the prohibitive costs of retraining the agents themselves or relying on expensive, black-box teachers at inference time.