🤖 AI

FLARE: Fine-Grained Diagnostic Feedback for LLM Code Refinement

FLARE is an iterative framework that enhances large language model code refinement by employing a lightweight diagnostic model to generate fine-grained, line-level bug localization signals, which are further optimized through a top-k candidate search strategy to achieve significant performance improvements over existing baselines.

Original authors: Yinsheng Yao, Hongxiang Zhang, Weixi Tong, Tianyi Zhang

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yinsheng Yao, Hongxiang Zhang, Weixi Tong, Tianyi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you ask a very talented but slightly scatterbrained robot to write a computer program for you. The robot does a great job most of the time, but occasionally, it sneaks in a tiny mistake—a "bug"—that makes the program crash or give the wrong answer.

Usually, when this happens, we tell the robot, "Hey, it didn't work," and ask it to try again. But that's like telling a student, "You got the math problem wrong," without pointing out which number they added incorrectly. The robot might just guess randomly where the error is, wasting time and energy.

This paper introduces FLARE, a new system designed to be a much better "debugging coach" for these AI robots. Here is how it works, broken down into simple concepts:

1. The "X-Ray Vision" Coach

Most current methods just look at the final result (the "crash") and guess. FLARE is different. It has a special, lightweight "diagnostic model" that acts like an X-ray machine.

Instead of just looking at the finished code, FLARE peeks inside the robot's brain while it was writing the code. It looks at the robot's "confidence levels" (how sure the robot was about every single word it typed).

The Analogy: Imagine a student taking a test. If they hesitate, erase, or look unsure about a specific line of math, a teacher knows that's where the mistake likely is. FLARE does this automatically. It scans the code and says, "I'm 90% sure the error is hiding in this specific line," rather than just saying, "The whole program is broken."

2. The "Top-K" Safety Net

Even with X-ray vision, the coach can't be 100% perfect. Sometimes the robot's hesitation might be a false alarm. To handle this uncertainty, FLARE doesn't just pick the one most suspicious line.

The Analogy: Imagine a detective looking for a lost key. A normal detective might say, "It's definitely in the kitchen." FLARE is like a detective who says, "It's probably in the kitchen, but let's also check the living room and the bedroom just in case."
FLARE picks the top 10 most suspicious spots (or however many you tell it to) and asks the robot to fix all of them. It then runs all 10 versions to see which one actually works. This ensures that even if the coach guesses wrong on the #1 spot, they still have a backup plan.

3. The Results: Faster and Smarter

The authors tested FLARE on two big sets of coding puzzles (LiveCodeBench and BigCodeBench) using five different AI models.

The Win: Even when FLARE only looked at the single most suspicious line (no backup plan), it fixed more bugs than the best existing methods.
The Big Win: When FLARE used its "Top-K" safety net (checking 10 possibilities), it improved the success rate by an average of 8.5%.
Efficiency: Because FLARE uses a tiny, fast "coach" to find the bug, it doesn't need to waste time asking the big AI to guess where the bug is. It saves time compared to other methods that rely on the AI just "talking to itself" to find errors.

4. Where It Stumbles

The paper is honest about where FLARE isn't perfect. It found three main types of failures:

The "Right Spot, Wrong Fix" Problem: FLARE correctly points to the suspicious line, but the AI robot still doesn't know how to fix the logic. It's like pointing to the broken engine part, but the mechanic doesn't know how to replace it.
The "Misunderstanding the Job" Problem: Sometimes the robot writes code that looks perfect but solves the wrong problem entirely. FLARE can't fix this because the code isn't "buggy" in a technical sense; the robot just misunderstood the instructions.
The "Long Story" Problem: If the bug involves a complex chain of events happening over a long time (like a simulation), FLARE might miss the connection between the start and the end.

Summary

FLARE is a system that helps AI code better by giving it a precise, line-by-line map of where it likely made a mistake, rather than just a vague "you're wrong" signal. By checking a few top guesses at once, it dramatically increases the chances of getting the code right on the first few tries, making AI coding assistants more reliable and efficient.

Technical Summary: FLARE – Fine-Grained Diagnostic Feedback for LLM Code Refinement

Problem Statement

Large Language Models (LLMs) demonstrate significant capabilities in code generation but frequently produce code containing bugs. Current iterative refinement methods attempt to correct these errors by generating an initial program, receiving feedback, and refining the code over multiple rounds. However, existing approaches rely on feedback signals that are often too coarse-grained or high-level. Common signals include execution failures (e.g., test case errors) and self-critiques generated by the LLM itself. These signals indicate that a bug exists but fail to provide sufficient guidance on where the bug resides within the code, limiting the model's ability to perform targeted fixes.

Methodology

The authors propose FLARE, an iterative framework designed to provide fine-grained, line-level diagnostic feedback to guide code refinement. The framework consists of two primary components: a lightweight diagnostic model and a candidate search procedure.

1. Fine-Grained Diagnostic Feedback

FLARE introduces a diagnostic model trained to predict line-level suspiciousness scores for buggy code.

Input Alignment: The model aligns the LLM's internal token-level probability signals with the code's lexical units (identifiers, operators, keywords). Since standard tokenization (e.g., Byte-Pair Encoding) often splits lexical units into subwords, FLARE aggregates the probabilities of subword tokens to compute a confidence score for each lexical unit.
Feature Representation: For each lexical unit, the model constructs a feature tuple comprising the token embedding, its syntactic type (e.g., Keyword, Identifier), and the aggregated confidence score.
Architecture: A bidirectional Transformer encoder processes the sequence of lexical units, allowing each unit to attend to the full program context. A linear classifier then outputs a suspiciousness score for each unit.
Aggregation: To generate line-level feedback, the model aggregates the scores of all lexical units within a line using max pooling, selecting the highest suspiciousness score as the line's score.
Training: The model is trained on the Collu-Bench dataset, which contains buggy code solutions paired with ground-truth bug locations and the generator LLM's token probabilities. The objective is to predict the position of the ground-truth buggy lexical unit.

2. Top-k Candidate Search

Recognizing the inherent uncertainty in diagnostic predictions, FLARE does not commit to a single predicted fault location. Instead, it employs a search strategy:

Selection: At each iteration, the diagnostic model ranks lines by suspiciousness. FLARE selects the top- $k$ suspicious lines.
Refinement: For each of the top- $k$ lines, a targeted refinement prompt is constructed containing the current program, execution feedback, and the specific suspicious line. The LLM generates a candidate revision for each prompt.
Evaluation: All $k$ candidates are executed against test cases. The candidate that passes the most tests (or resolves the most failures) is selected for the next iteration. If a candidate passes all tests, the process terminates.

Key Contributions

Fine-Grained Diagnostic Model: The paper presents a lightweight diagnostic model that leverages LLM internal signals (confidence and syntax) to localize bugs at the line level, addressing the granularity limitations of existing feedback methods.
Iterative Framework with Search: FLARE combines this diagnostic signal with execution feedback and a top- $k$ candidate search procedure to mitigate the uncertainty of fault localization.
Comprehensive Evaluation: The framework is evaluated on LiveCodeBench and BigCodeBench using five diverse base LLMs (including Qwen, Llama, and GPT-4o-mini) and compared against strong baselines like Self-Debugging, Self-Refine, and NL-Debugging.

Experimental Results

Performance Gains: Even without candidate search ( $k=1$ ), FLARE outperforms the strongest baseline (NL-Debugging) in 9 out of 10 settings, achieving absolute improvements in Pass@1 ranging from 1.72% to 7.42%.
Impact of Search: Expanding the search to $k=10$ candidates yields an average improvement of 8.50% over the $k=1$ variant, demonstrating that exploring multiple plausible fault regions significantly enhances refinement success.
Diagnostic Accuracy: When evaluated in isolation on fault localization tasks, FLARE's diagnostic model achieves 67% Top-1 and 89% Top-10 accuracy, outperforming recent fault localization methods such as FlexFL, LLMAO, and BAP.
Efficiency: The $k=1$ variant of FLARE is more efficient than Self-Debugging and Self-Refine, reducing average runtime per task from ~95 seconds to ~73 seconds, as it avoids the overhead of generating self-debugging text. The $k=10$ variant incurs higher computational costs due to multiple candidate generations but provides the highest accuracy.

Significance and Claims

The paper claims that fine-grained localization signals serve as reliable guidance for code refinement. By moving beyond coarse execution feedback or high-level self-critiques, FLARE provides the LLM with specific targets for correction. The authors emphasize that the combination of precise diagnostic localization and expanded candidate search plays a complementary role: localization identifies likely regions, while search mitigates the uncertainty of the prediction.

The study acknowledges limitations, noting that the framework's success is still bounded by the reasoning capabilities of the underlying LLM. Failures often occur not because the bug location is missed, but because the LLM lacks the algorithmic insight to fix the identified issue, misinterprets the task requirements, or fails to manage complex long-range state dependencies. Nevertheless, FLARE represents a significant step toward more effective, targeted iterative code refinement.