🤖 machine learning

Probe-and-Refine Tuning of Repository Guidance for Coding Agents

This paper introduces "probe-and-refine tuning," a method that iteratively optimizes repository guidance files (AGENTS.md) using synthetic bug-fix probes to significantly improve coding agent performance on SWE-bench Verified by expanding the coverage of evaluable patches rather than increasing per-patch precision.

Original authors: Asa Shepard, Jeannie Albrecht

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Asa Shepard, Jeannie Albrecht

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant but inexperienced apprentice to fix a massive, ancient library (the code repository). The apprentice is smart enough to read books and fix broken pages, but they don't know the library's secret rules: which shelf holds the rare maps, how to ask the librarian for help without waking the sleeping cats, or which ladders are safe to climb.

Without this "local knowledge," the apprentice wanders aimlessly, climbs the wrong ladder, or tries to fix a book in the wrong section, often breaking things in the process.

This paper introduces a method called "Probe-and-Refine Tuning" to solve this problem. Here is how it works, using simple analogies:

The Problem: The "One-Size-Fits-All" Manual

Engineers often write a "cheat sheet" (like an AGENTS.md file) for their AI coding agents.

The Old Way (Static Knowledge): They ask a smart AI to write a generic manual: "Always check the index before fixing," or "Look for the main entry point." This is like giving the apprentice a map of every library in the world. It's helpful, but it doesn't tell them where the specific library's secret stash is hidden.
The Result: The apprentice does okay, but they still get lost often.

The Solution: The "Trial-and-Error" Coach

The authors created a new process called Probe-and-Refine. Instead of just writing a manual once, they treat the manual like a living document that gets trained by making mistakes.

Think of it like a coach training a new player:

The Probes (The Drills): The coach generates 10 fake, made-up problems (probes) specific to this library.
The Attempt: The apprentice tries to solve these fake problems using the current manual.
The Diagnosis: The coach watches the attempt. "Oh, you tried to fix the plumbing in the kitchen, but the pipes are actually in the basement. And you forgot to check the blueprints first."
The Refinement: The coach updates the manual immediately with a specific rule: "For this library, plumbing is in the basement, and always check the blueprints first."
Repeat: They do this 3 to 5 times. The manual evolves from a generic guide into a hyper-specific, "insider" guide.

Crucially, this whole training process happens without the agent actually fixing real bugs. It's a simulation loop where the AI writes the manual, tests it on fake problems, and fixes the manual based on the results.

What They Found

The researchers tested this on a famous coding benchmark (SWE-bench) with four different runs. Here is what happened:

No Guide: The apprentice solved 25.5% of the problems.
Generic Guide: The apprentice solved 28.3% of the problems.
Probe-and-Refine Guide: The apprentice solved 33.0% of the problems.

The Big Secret: It's About Finding the Right Door, Not Fixing Better
You might think the improved manual made the apprentice smarter or better at fixing things. It didn't.

When the apprentice did fix something, the quality of the fix was exactly the same (about 59% success rate) regardless of which manual they used.
The real magic was "Coverage." The improved manual helped the apprentice find the right file to fix much more often.
- Analogy: The generic manual made the apprentice knock on 100 doors, but only 40 were the right ones. The refined manual made them knock on 100 doors, and 56 were the right ones. Once they found the right door, their ability to fix the lock was the same.

The "Step Budget" Trap

The paper also discovered that the guide only works if you give the apprentice enough time.

If you give the apprentice only 25 steps to solve a problem, the fancy manual doesn't help. They don't have time to follow the complex instructions.
If you give them 200 steps, the fancy manual shines. It tells them a workflow (Reproduce -> Trace -> Fix) that takes time but works. Without the manual, they just rush and fail.
Analogy: If you tell someone, "Take a scenic route to avoid traffic," but you only give them 5 minutes to get to work, they will just get lost. You need to give them enough time to actually take the scenic route.

The "Model Mismatch" Warning

The most surprising finding was that this guide is not universal.

They tried using the guide trained on one AI model (Qwen) with a different, slightly weaker AI model (Nemotron).
Result: The second model crashed. It got so confused by the specific instructions that it stopped working entirely.
Analogy: It's like giving a detailed, high-speed racing manual to a driver in a slow, old sedan. The instructions are too complex for the car, and the driver freezes up. The guide must be "tuned" specifically for the brain (model) that will read it.

Summary

This paper proves that how you write the instructions matters more than just having instructions.
By using a simple loop of "make a fake problem, try to solve it, and fix the instructions based on the failure," you can turn a generic guide into a specialized expert manual. This doesn't make the AI smarter at fixing code; it just stops it from looking in the wrong place, allowing it to use its full potential to solve more problems.

Technical Summary: Probe-and-Refine Tuning of Repository Guidance for Coding Agents

Problem Statement

LLM-based coding agents often fail to resolve software engineering tasks not due to a lack of reasoning capability, but because they lack high-level operational knowledge about a specific code repository. This implicit knowledge—such as subsystem boundaries, test execution workflows, and historical debugging patterns—is rarely documented in the code itself. While engineers maintain context files (e.g., AGENTS.md) to supply this information, recent literature presents conflicting evidence on their efficacy: some studies suggest they improve efficiency, while others indicate they can reduce resolve rates if the guidance is generic or counterproductive. The core problem is that it remains unclear whether the failure lies in the concept of guidance itself or in how such guidance is produced. Current methods often rely on single-pass LLM generation, which yields generic advice rather than the specific, failure-informed operational strategies required by agents.

Methodology

The paper introduces Probe-and-Refine Tuning, a lightweight procedure designed to iteratively diagnose and patch a repository's guidance file using synthetic bug-fix probes. The procedure is distinct from standard agent loops as it relies entirely on single-shot LLM calls with no tool use or reinforcement learning during the tuning phase.

The Tuning Loop

The process transforms a static, generic knowledge base into specialized operational guidance through 3–5 iterations:

Generate Probes: The system generates a batch of 10 synthetic bug-fix tasks from the repository's codebase, targeting diverse subsystems.
Attempt Fix: For each probe, a single-shot LLM call attempts to generate a patch using the current guidance artifact.
Judge Attempt: A separate single-shot call assesses the attempted solution against expected behaviors (e.g., strong, partial, or missing fixes) and proposes specific edits to the guidance where shortcomings are identified.
Aggregate and Apply: A final call aggregates diagnostics from all probes and proposes a consolidated set of edits. These edits are mechanically applied (inserting, modifying, or strengthening sections) to the guidance file, capped at 3,000 characters.

The loop terminates early if the guidance stabilizes (no changes between consecutive iterations). The resulting artifact is then used to guide a standard ReAct-style coding agent during the evaluation phase on SWE-bench Verified.

Experimental Setup

The authors evaluated three conditions on 500 SWE-bench Verified instances across four independent trials using the Qwen3.5-35B-A3B model:

No Context: The agent operates with no repository-specific guidance.
Static KB: The agent uses a two-layer knowledge base containing structural data (parsed via tree-sitter) and generic, repository-agnostic best practices.
Probe-and-Refine: The agent uses guidance produced by the iterative tuning procedure described above.

A secondary experiment substituted the model with NVIDIA-Nemotron-3-Nano-30B-A3B to test cross-model generalization, and a step-budget sweep (25, 50, 100, 200 steps) was conducted to analyze the interaction between guidance and agent budget.

Key Results

Performance Gains

Across four trials, Probe-and-Refine achieved a mean resolve rate of 33.0%, significantly outperforming the Static KB (28.3%) and the No Context baseline (25.5%). The improvement was statistically significant ( $p < 0.001$ ) for both contrasts. The procedure also yielded 31 consistently unique solves (resolved in $\ge$ 3 of 4 trials) that neither of the other conditions could resolve.

Mechanism: Coverage vs. Precision

The paper decomposes the performance gain to determine how the guidance helps:

Coverage: The primary driver of improvement is evaluation coverage. Probe-and-Refine guidance enabled the agent to produce well-formed, evaluable patches for 56.2% of instances, compared to 41.7% for the unguided baseline (a 14.5 percentage point gap).
Precision: The correctness of the patches, conditional on them being evaluable, remained statistically constant across all conditions ( $\sim$ 59%, $p = 0.119$ ).
Conclusion: The guidance does not improve the quality of the code changes the agent makes; rather, it helps the agent reach the correct file and produce a patch that the evaluation harness can accept. The guidance effectively reduces the rate of malformed patches or patches that crash the test suite before assertions can be checked.

Step Budget and Workflow

The study found that guidance acts as a moderator for step budget utility.

Unguided agents saturate quickly; increasing the step budget from 25 to 200 steps did not improve their resolve rate, as they lack a structured plan to convert extra steps into better patches.
Probe-and-Refine agents utilize larger budgets productively. The prescribed "reproduce-trace-patch" workflow requires more steps to complete. Consequently, the probe-and-refine condition showed significant gains at higher budgets (100 and 200 steps) where simpler guidance (Static KB) plateaued.

Cross-Model Generalization

When applied to the Nemotron model, the results highlighted a critical model-fit dependency:

Tuning Failure: The tuning loop degraded on Nemotron because the model could not generate sufficiently diagnostic output for the "Judge" step, causing the loop to stall.
Negative Transfer: Guidance tuned on Qwen, when applied to Nemotron, actively collapsed agent performance (13.2% resolve rate vs. 27.0% for Nemotron's own baseline). The detailed, prescriptive guidance caused Nemotron to engage in "compliance by analysis" (writing prose about what it would do) rather than executing commands, triggering a cascade of fallback failures.
Precision Constancy: Even in the cross-model collapse, the precision of the few patches that were produced remained constant, reinforcing that the failure was entirely in coverage (the agent failing to produce a patch), not in the quality of the patch itself.

Significance and Claims

The paper claims that instruction quality is a first-order variable in coding agent reliability, comparable in magnitude to model capability or step budget choices.

Operational Knowledge is Learnable: The authors demonstrate that operational knowledge (which files house which subsystems, how to run tests) can be distilled from synthetic failures into a compact text artifact without complex agent loops or tool use.
The Bottleneck is Guidance, Not Reasoning: The results suggest that for many "hard" instances, the agent's failure is not a lack of reasoning power but a failure to localize the problem to the correct file. The guidance acts as a localization aid, enabling the agent to navigate the repository structure effectively.
Model-Specific Calibration: Guidance is not transferable repository knowledge; it is model-specific behavioral calibration. Guidance tuned for one model's failure modes can actively harm a different model with a different behavioral profile.
Reconciling Prior Disagreements: The paper reconciles conflicting literature on AGENTS.md files by showing that the efficacy of guidance depends on how it is produced (iterative refinement vs. single-pass generation) and the step budget available. Static, generic guidance may be neutral or harmful, while iteratively refined, workflow-specific guidance is beneficial, provided the budget allows the workflow to complete.

The authors conclude that for practitioners, investing in iteratively refined guidance is valuable for repositories with high issue volumes, provided the guidance is tuned on the specific model that will consume it and the step budget is sufficient to accommodate the prescribed workflow.