💻 computer science

Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback

This paper introduces PAIR-Bench, a progressive and adaptive benchmark that evaluates large language models' code improvement capabilities by measuring their ability to refine programs through structured, multi-level feedback rather than relying solely on binary pass/fail outcomes.

Original authors: Cuong Chi Le, Aashish Yadavally, Minh Le-Anh, Tien N. Nguyen

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cuong Chi Le, Aashish Yadavally, Minh Le-Anh, Tien N. Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to fix a broken toaster.

The Old Way (Binary Pass/Fail):
In the past, researchers would give the robot a broken toaster and a list of tests (e.g., "Does it toast bread?", "Does it pop up?"). If the robot fixed the toaster perfectly, it got a gold star. If it failed even one test, it got a zero.

The Problem: This is like grading a student who gets 99% of a math test right but misses one tiny detail. They get an "F." Conversely, a student who guesses the right answer by luck gets an "A," even if they don't understand why it works. It ignores the journey of learning and the fact that the robot might have fixed 90% of the problem but got stuck on the last 10%.

The New Way (PAIR-BENCH):
The authors of this paper, Cuong Chi Le and colleagues, created a new way to test robots (specifically, Large Language Models or LLMs) called PAIR-BENCH. Instead of just looking at the final result, they watch the entire process of how the robot learns to fix the code.

Think of PAIR-BENCH as a video game with a helpful coach rather than a final exam.

How It Works: The Two "Knobs"

The system uses a "coach" (a feedback model) to guide the "player" (the robot trying to fix the code). The coach has two special knobs to control the hints:

The "Where" Knob (Failure-Region Control):
Imagine the broken toaster has three problems: a burnt wire, a stuck spring, and a loose plug.
- Old way: The coach might randomly shout, "It's broken!" without saying where.
- New way: The coach picks one specific problem to focus on first, like "Let's look at the burnt wire." Once the robot fixes that, the coach moves to the next problem. This ensures the robot is actually fixing specific issues, not just guessing.
The "How Much" Knob (Hint-Depth Control):
This is like adjusting how much help the robot gets, similar to a teacher helping a student.
- Level 1 (Symptom): "The toaster is smoking." (Very vague).
- Level 2 (Pattern): "It smokes when you put in thick bread." (Better).
- Level 3 (State): "You aren't tracking how long the bread has been in there." (Getting closer).
- Level 6 (Direction): "Change the timer logic to count seconds instead of loops." (Almost the answer).
- The Magic: If the robot fixes the problem with just a Level 1 hint, it's a genius. If it needs a Level 6 hint to fix it, it's struggling. The system measures how much help the robot needed to succeed.

What They Discovered

The authors tested this new system on several top-tier AI models (like DeepSeek, Gemini, and GPT-4o-mini) using real coding problems. Here is what they found, using simple terms:

Some models are "Self-Starters": One model (DeepSeek) could often fix problems with very vague hints (Level 1 or 2). It didn't need the coach to hold its hand.
Some models need "Hand-Holding": Other models could eventually fix the problem, but they needed very specific, detailed instructions (Level 5 or 6). They couldn't figure it out on their own.
Stability Matters: Some models would fix one part of the code but accidentally break another part they had already fixed. The new system catches this "regression" (going backward), which the old "pass/fail" system missed.
Consistency: When they ran the test multiple times, the new system gave very consistent results. The old way was like rolling dice; sometimes a model got lucky with a vague hint, and sometimes it got unlucky. The new system is fair and stable.

The Big Takeaway

The paper argues that we shouldn't just ask, "Did the robot fix the code?" We should ask:

"How much help did it need?"
"Did it get stuck on one thing and ignore the rest?"
"Did it fix things without breaking what was already working?"

By measuring the journey (the trajectory) rather than just the destination (the final pass/fail), PAIR-BENCH gives us a much clearer, fairer picture of how smart and capable these AI models really are at improving code. It's the difference between saying "He passed the test" and "He learned the material, needed a little nudge on the hard parts, and didn't forget what he already knew."

Technical Summary: Benchmarking Code Improvement with Progressive, Adaptive, and Interactive Feedback

1. Problem Statement

Current Large Language Model (LLM) evaluation for code generation and automated program repair (APR) relies heavily on binary pass/fail metrics determined by test suites. The authors argue this approach is coarse and misleading for several reasons:

Sparsity of Information: A binary outcome ignores partial progress, treating a solution that fixes one edge case the same as one that fails all tests.
Inflation of Performance: Weak test suites may allow models to pass by making shallow edits that fix reported failures but introduce subtle regressions.
Neglect of Refinement Trajectories: In iterative workflows where models refine code based on feedback (compiler errors, execution traces, or diagnostics), evaluation typically only considers the final outcome. This fails to capture whether a model effectively utilizes feedback, generalizes repairs, or maintains previously correct behavior.
Uncontrolled Feedback Variability: Existing interactive benchmarks often lack control over the feedback provided. The informativeness of feedback (e.g., revealing the root cause vs. a vague symptom) can vary wildly between runs, making it difficult to distinguish between model capability and feedback quality.

The paper posits that benchmarking a model's ability to make progress through feedback-guided refinement offers a more fine-grained and faithful measure of real-world utility than static pass/fail rates.

2. Methodology: PAIR-BENCH

To address these limitations, the authors introduce PAIR-BENCH, a progressive and adaptive benchmark for evaluating code improvement. The core innovation is a Progressive Hinting protocol that controls feedback along two dimensions:

A. Core Components

Scenario Constructor (Failure-Region Control):
- Hidden failing test cases are grouped into failure scenarios based on a "failure signature" comprising:
  - Reference execution trace (source lines executed).
  - Expected output shape (structural form).
  - Candidate failure type (e.g., timeout, wrong value).
- This groups related incorrect behaviors, allowing the benchmark to target specific regions of the failure space rather than arbitrary test cases.
Hint Generator (Hint-Depth Control):
- Inspired by Item Response Theory (IRT) and educational scaffolding, hints are generated at one of six levels of increasing informativeness:
  - L1 (Symptom): Only the observed incorrect behavior.
  - L2 (Input Pattern): The input condition exposing the failure.
  - L3 (State Tracking): Missing state or logic the solution fails to preserve.
  - L4 (Fault Location): Suspicious code regions.
  - L5 (Conceptual Correction): Missing invariants or reasoning.
  - L6 (Repair Direction): Concrete implementation guidance without code.
- A feedback model (separate from the candidate model) generates hints constrained by these levels to prevent information leakage.
Adaptive Policy:
- Progressive: The benchmark iterates through distinct failure scenarios, prioritizing those with the highest coverage of failing tests.
- Adaptive: The hint depth adjusts based on the model's performance. If a model repairs a scenario using a shallow hint, the next scenario starts with a weaker hint (fading assistance). If the model fails, the hint depth increases (escalating guidance).
Trajectory Evaluator:
- Instead of a single binary score, the benchmark tracks the repair trajectory across turns, measuring progress at every step.

B. Evaluation Metrics

The paper proposes a suite of progress-centric metrics:

Initial Fix Rate: Unaided repair capability.
Targeted Repair Success (TRS): Ability to fix the specific scenario described by the hint.
Broader Repair Gain (BRG): Generalization of the fix to other, unhinted failures.
Behavior Preservation (BP): Ability to avoid introducing regressions in previously passing tests.
Progress Monotonicity Rate (PMR): Frequency of non-decreasing pass rates across turns.
Hint Efficiency (HE): The level of hint required to close a scenario (lower levels indicate stronger diagnostic ability).
Gap Closure: The fraction of the remaining repair opportunity closed.
Turns to Fix: Convergence speed among successfully repaired instances.

3. Empirical Evaluation

The authors instantiated PAIR-BENCH using 440 code-improvement instances derived from Codeforces Python submissions (specifically "Wrong Answer" verdicts). They evaluated eight state-of-the-art LLMs (including DeepSeek V3.2, Gemini 2.5 Flash Lite, Qwen3, GPT-4o-mini, Llama 3.3, etc.) using a controlled feedback model (GPT-OSS 120B).

Key Findings

Model Differentiation: Binary metrics often fail to distinguish models effectively. PAIR-BENCH revealed significant differences in how models utilize feedback. DeepSeek V3.2 emerged as the strongest model, ranking first in most metrics, including high Initial Fix Rate, Targeted Repair Success, and Hint Efficiency.
Feedback Utility: Models benefit significantly from controlled hints. For example, Gemini 2.5 Flash Lite improved its Final Fix Rate from ~78% (no-hint multi-turn) to ~95% (controlled feedback), demonstrating that gains are driven by repair-relevant hints, not just repeated prompting.
Diagnostic vs. Dependent Repair: Stronger models (e.g., DeepSeek) often solved scenarios with shallow hints (L1–L2), indicating strong diagnostic abilities. Weaker models required deeper hints (L5–L6) or failed to resolve scenarios, highlighting a dependence on explicit guidance.
Stability: Controlled feedback produced significantly more stable evaluation results than "Vanilla" feedback (uncontrolled). The standard deviation in repair rates and final fix rates decreased by approximately 78–84% when using the controlled protocol, and ranking stability (Kendall's $\tau$ ) improved from 0.905 to 0.982.
Trade-offs: High Hint Efficiency (needing less help) did not always correlate with high Final Fix Rate or stability. For instance, Qwen showed strong preservation and monotonicity but required more turns to converge compared to Gemini.

4. Key Contributions

PAIR-BENCH: A new progressive and adaptive benchmark paradigm for evaluating code improvement, available with data and code.
Progress-Centric Metrics: A suite of metrics designed to measure interactive code improvement beyond final correctness, capturing trajectory, stability, and generalization.
Controlled Interactive Feedback Protocol: The introduction of Progressive Hinting, a structured protocol that jointly controls failure regions (via scenario grouping) and hint depth (via a 6-level scale) to calibrate feedback difficulty to model capabilities.
Empirical Study: A comprehensive evaluation of multiple SOTA LLMs demonstrating that model rankings and capabilities vary significantly depending on the dimension of code improvement being measured.

5. Significance and Claims

The paper claims that PAIR-BENCH provides a more faithful assessment of LLM code-improvement capabilities by moving beyond sparse binary outcomes. Its significance lies in:

Revealing Hidden Capabilities: It distinguishes between models that merely "pass" via shallow edits and those that demonstrate genuine diagnostic reasoning and stable refinement.
Standardizing Feedback: By controlling the failure region and hint depth, it removes the variability of unstructured feedback, allowing for fairer comparisons of model reasoning capabilities.
Capturing the Refinement Trajectory: It acknowledges that code improvement is an iterative process, measuring not just the destination (final pass) but the path taken (progress, stability, and efficiency).

The authors conclude that effective interactive repair depends on multiple dimensions—final correctness, targeted repair, progressive improvement, stability, and hint efficiency—and that PAIR-BENCH is necessary to evaluate these dimensions holistically. Future work is noted to include extending the adaptive process to the question level (varying problem difficulty).