🤖 AI

Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair

This paper introduces a dual-LLM policy framework combining bug abstention and patch validation to significantly reduce noise and improve success rates in industrial-scale agentic Automated Program Repair by filtering out unlikely-to-be-fixed bugs and suboptimal patches before human review.

Original authors: José Cambronero, Michele Tufano, Sherry Shi, Renyao Wei, Grant Uy, Runxiang Cheng, Chin-Jung Liu, Shiying Pan, Satish Chandra, Pat Rondon

Published 2026-01-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: José Cambronero, Michele Tufano, Sherry Shi, Renyao Wei, Grant Uy, Runxiang Cheng, Chin-Jung Liu, Shiying Pan, Satish Chandra, Pat Rondon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented, but slightly over-eager, robot assistant named "Repair-Bot." Your job is to fix thousands of broken things in a giant warehouse (the company's codebase). Repair-Bot is great at trying to fix things, but it's also prone to making mistakes, wasting time, and suggesting fixes that don't actually work.

If you let Repair-Bot show you every single thing it tries to fix, you would be overwhelmed. You'd spend all day looking at broken suggestions, getting frustrated, and eventually stop trusting the robot.

This paper introduces a two-step "Gatekeeper" system to stop the noise before it reaches your desk. Think of it as a bouncer at a club and a quality inspector at a factory.

The Problem: Too Much Noise

In the world of software, "bugs" are mistakes in the code. Automated systems (called Agentic APR) try to fix them. But these systems often try to fix bugs that are too hard for them, or they generate "patches" (code changes) that look okay but are actually wrong.

The Result: Developers waste time reviewing bad fixes.
The Goal: Show the developer only the fixes that are likely to work.

The Solution: The Two-Gate System

The authors propose two policies that act like filters in a funnel.

Gate 1: The "Bug Abstention" (The Bouncer)

What it does: Before the robot even tries to fix a bug, this gate looks at the bug report (the description of the problem).
The Analogy: Imagine you are a bouncer at a club. You look at the person trying to get in (the bug). If the person looks too confused, the description is too vague, or the problem seems impossible to solve with the tools you have, you say, "Sorry, you can't come in."
How it works: An AI model reads the bug report and asks, "Can our robot actually fix this?" If the answer is "probably not," the system abstains (refuses to try). It saves the robot's energy and, more importantly, saves the human developer from seeing a failed attempt.
The Paper's Claim: By acting as a bouncer, this gate filters out the "impossible" bugs. It raises the success rate of the bugs that do get through from about 11% to 21% (using a specific set of guidelines).

Gate 2: The "Patch Validation" (The Quality Inspector)

What it does: If the robot does try to fix a bug and produces a patch, this gate checks the result.
The Analogy: Imagine the robot has built a new engine part. Before you install it, a quality inspector looks at the blueprint. The inspector doesn't just look at the part; they first write down a "recipe" for what a perfect part should look like based on the problem description. Then, they compare the robot's part against that recipe.
How it works:
1. The AI writes a "specification" (a set of rules for a correct fix) based on the bug report.
2. A second AI looks at the robot's actual code change and the specification.
3. It gives a score: "This looks good," "This looks bad," or "I'm not sure."
4. If the score is too low, the patch is rejected.
The Paper's Claim: This gate catches the robot's mistakes. Even if the robot tries to fix a bug, this gate can say, "No, that's not the right fix." This alone can raise the success rate of shown patches to about 29%.

The Magic Combo: The Funnel

When you use both gates together, they work like a highly efficient funnel.

Gate 1 stops the impossible problems from entering the system.
Gate 2 stops the bad solutions from leaving the system.

The Results:

Baseline: Without any gates, developers see a working fix only 1 out of 10 times (11%).
With Both Gates: If you set the gates to be strict (only showing the very best candidates), developers see a working fix 1 out of 2 times (53%).
The Trade-off: You have to be okay with showing fewer bugs overall. The system filters out many bugs to ensure that the few it does show are high-quality.

What About Machine-Generated Bugs?

The paper also tested this on bugs found automatically by computers (like "Null Pointer Exceptions" or memory errors). These bugs usually come with clear instructions (like a stack trace).

For these, they didn't need the "Bouncer" (Gate 1) because the computer already knows these are fixable.
However, the "Quality Inspector" (Gate 2) still helped, raising the success rate of accepted patches significantly.

Summary

This paper doesn't claim to make the robot smarter at fixing bugs. Instead, it claims to make the process of showing fixes to humans much smarter. By using two AI "gatekeepers" to filter out the noise, developers stop wasting time on bad ideas and start trusting the automated system again.

In short: It's not about fixing more bugs; it's about fixing the right bugs so humans don't get tired of looking at the wrong ones.

Technical Summary: Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair

Problem Statement

Agentic Automated Program Repair (APR) systems have demonstrated the ability to resolve complex, repository-level bugs, achieving high pass@1 rates on benchmarks like SWE-Bench-Verified. However, deploying these systems in industrial settings presents a critical challenge: the "noise" problem. Even with improving resolution rates, a significant portion of generated patches are unlikely to be accepted by human developers. Presenting these low-quality or intractable patches wastes valuable developer time, erodes trust in automated tools, and incurs unnecessary computational costs. The core problem addressed is how to filter out bugs that an APR system is unlikely to fix before repair attempts, and how to reject patches that are unlikely to be correct after generation, thereby optimizing the signal-to-noise ratio for human reviewers.

Methodology

The authors propose a dual-LLM policy framework consisting of two complementary stages: Bug Abstention and Patch Validation.

1. Bug Abstention Policy

This policy operates before the repair agent attempts a fix. It functions as a pre-filter to exclude bugs deemed unlikely to be resolved by the system.

Input: Only the bug report (title, description, metadata). The policy does not access the codebase or the agent's trajectory to maintain efficiency.
Mechanism: An LLM predicts the probability of a successful repair attempt. The authors formalize this as estimating the empirical success probability ( $\hat{\pi}_b$ ) of a bug $b$ .
Representations: Two prompt strategies were evaluated:
- Instruction Prompting without Guidelines: Basic bug report input.
- Instruction Prompting with Guidelines: Augmented with human-derived criteria (e.g., "Clear Problem & Action," "Precise Code Localization") to guide the LLM's assessment of difficulty.
Decision: If the predicted probability $P_a(b)$ falls below a configurable threshold $\tau$ , the system "abstains" from attempting repair, saving compute and preventing the generation of low-quality patches.

2. Patch Validation Policy

This policy operates after the agent generates a repair trajectory. It assesses whether a specific patch is a correct fix.

Input: The bug report, the agent's trajectory (thoughts and actions), the generated patch (diff), and test results.
Mechanism: A multi-stage approach:
1. Deterministic Checks: Build and test regression checks to remove patches with obvious failures.
2. Fix Specification Generation: An LLM generates a "fix specification"—a set of criteria for a correct fix based only on the bug report and original source files (without ground-truth knowledge).
3. LLM Judgment: A second LLM call evaluates the agent's attempt against the generated specification, the patch, and test results.
Output: A triplet consisting of a binary judgment (fixed/not fixed), a natural language explanation, and a confidence score derived from token-level log probabilities.
Decision: Patches can be filtered based on the binary judgment or a percentile-based cutoff of the confidence score, allowing for adjustable strictness.

3. Combined Framework

The two policies are composed sequentially: Bug Abstention filters the initial bug set, the APR system attempts repairs on the remaining bugs, and Patch Validation filters the resulting patches. This creates a configurable funnel for developer review.

Evaluation and Results

The policies were evaluated on three industrial datasets from Google's codebase:

Human-Reported Bugs (174 bugs): Known ground-truth fixes and fail-to-pass reproduction tests.
Machine-Reported NPE Bugs (198 bugs): Null pointer exceptions from live deployment with stack traces.
Sanitizer-Reported Bugs (50 bugs): Memory errors and data races with known reproduction tests.

Key Findings:

Human-Reported Bugs:
- Baseline: The unfiltered system achieved a fail-to-pass@1 rate of 11.29%.
- Bug Abstention: Using guidelines ( $r_2$ ), the filtered fail-to-pass@1 rate increased to 21.05%.
- Patch Validation: Using a strict percentile-based validation (90th percentile), the filtered fail-to-pass@1 rate increased to 29% (a 15 percentage point gain over baseline).
- Combined Policy: The dual-policy approach achieved a filtered fail-to-pass@1 of 53% (a 39 percentage point gain) with the strictest configuration (combined-p90), effectively raising the success rate to 1-in-2 for the patches shown to developers.
Machine-Reported Bugs:
- Patch validation significantly improved filtered accept@1 rates for NPE bugs (from 0.38 to 0.62) and sanitizer bugs (up to 15 percentage points over baseline).
Trade-offs: The study highlights a trade-off between the volume of bugs attempted and the precision of patches shown. Stricter filtering (e.g., 90th percentile) drastically reduces the number of bugs presented to developers (e.g., from 174 down to 12) but ensures a much higher success rate for those that remain.

Key Contributions

Bug Abstention Policy: A novel pre-repair filtering mechanism that uses LLMs to predict repair feasibility based solely on bug reports, preventing futile repair attempts.
Patch Validation Policy: A multi-stage post-repair assessment framework that utilizes "fix specifications" generated without ground-truth knowledge to filter candidate patches.
Extensive Industrial Evaluation: A comprehensive study across three distinct bug categories (human-reported, NPE, and sanitizer) demonstrating the complementary nature of the two policies.
Metric Innovation: Introduction of "filtered success@k," a metric that evaluates success rates only on the subset of bugs and patches allowed by the policies, providing a more realistic measure of developer experience.

Significance and Claims

The paper claims that this two-policy approach provides a practical path to the reliable, industrial-scale deployment of agentic APR systems. By significantly reducing the noise of unproductive repair attempts, the framework addresses the critical bottleneck of developer trust and time management.

The authors emphasize that while the overall bug resolution rate may decrease due to aggressive filtering (as some fixable bugs are rejected), the quality of the patches presented to developers improves dramatically. In practical terms, this transforms the developer experience from reviewing a patch with a ~10% chance of success to reviewing a patch with a ~50% chance of success (under strict settings) or ~33% (under permissive settings). This shift is argued to be essential for the adoption of AI tools in software engineering, as it aligns the output of automated systems with the limited capacity of human reviewers. The work does not claim to solve the fundamental difficulty of repairing all bugs, but rather to optimize the interaction between agentic systems and human developers by ensuring that only the most promising candidates reach the human stage.