💻 computer science

GapForge: Directed Compiler Fuzzing via Coverage-Gap Analysis

GapForge is a targeted LLM-based compiler fuzzing technique that systematically identifies and bridges long-tail coverage gaps by prioritizing undercovered files, inferring fine-grained triggering requirements through path-difference analysis, and iteratively synthesizing prompts to significantly outperform existing methods in coverage and bug discovery on GCC and LLVM.

Original authors: Mingxuan Zhu, Qingyuan Liang, Junjie Chen, Zhihong Xue, Dan Hao

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingxuan Zhu, Qingyuan Liang, Junjie Chen, Zhihong Xue, Dan Hao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the software that runs your phone, your car, or the internet as a giant, invisible factory. Inside this factory, there are millions of tiny workers (lines of code) that take your high-level instructions and turn them into machine language the computer understands. This factory is called a compiler. If even one worker is lazy, confused, or breaks a rule, the whole factory can produce garbage products—programs that crash, freeze, or secretly do the wrong thing. Because these factories are so huge and complex, it's incredibly hard to make sure every single worker is being watched and tested.

Enter the world of software testing. Think of this as sending in a team of inspectors to see if the factory is working right. For years, inspectors have used two main tricks: either they throw random darts at the factory (random testing) or they try to break things by twisting existing instructions (mutation testing). But here's the problem: the inspectors keep hitting the same easy-to-reach corners of the factory, leaving the dusty, dark, hard-to-reach backrooms completely ignored. These "blind spots" are where the most dangerous bugs hide, waiting to cause trouble. Recently, scientists started using Large Language Models (LLMs)—super-smart AI that can write code—to help with inspections. But even these AI helpers often wander aimlessly, not knowing exactly which dark corners need the most light.

This is where a new technique called GapForge comes in, acting like a detective with a magical map. Instead of just throwing darts or guessing, GapForge looks at a map of the factory to see exactly which rooms have never been visited. It then uses its AI brain to figure out the specific "secret handshake" (a specific type of code and a special setting) needed to unlock those locked doors. The researchers tested this on two of the world's biggest compiler factories, GCC and LLVM. They found that GapForge is a master at finding these hidden rooms. In just 72 hours, it covered 68.13% of the core GCC code and 69.11% of the LLVM code. That might not sound like 100%, but remember, the previous best AI method only managed about 64.62% and 65.02%. GapForge didn't just nudge the numbers up; it found 24,736 more lines of code in GCC and 19,798 more in LLVM that the other methods completely missed.

How does it do this? Think of GapForge as a three-step process. First, it scans the factory map and picks the "dirtiest" room—the one with the most untested code. Second, instead of just looking at the whole room, it zooms in on the specific dusty spot and asks the AI, "What kind of key do we need to open this?" The AI analyzes the surrounding clean areas to guess the exact combination of code structures and factory settings (like turning on a specific switch) required to reach the spot. Third, if the AI tries a key and it doesn't work, GapForge remembers that failure. It tells the AI, "Don't try that again; try something different," and sends it back with a better plan. This cycle repeats, slowly but surely lighting up every dark corner.

The results were impressive. Not only did GapForge cover more ground than eight other top-tier techniques, but it also found 12 real-world bugs that had been hiding in the shadows. These included 8 crashes (where the compiler just gave up and stopped) and 4 miscompilations (where the compiler silently built a broken program that looked perfect). The researchers showed that every part of GapForge's process mattered; if they removed the "map reading," the "key guessing," or the "learning from failure," the results got significantly worse. While other methods were busy generating thousands of test programs, GapForge generated fewer but much smarter ones, proving that in the world of software testing, quality and direction often beat sheer quantity.

Technical Summary: GapForge

Problem Statement

Modern compiler codebases, such as GCC and LLVM, are characterized by immense scale and complexity, making it difficult to achieve comprehensive code coverage. Existing test generation techniques—ranging from grammar-aided and mutation-based methods to recent Large Language Model (LLM) approaches—primarily operate in a "program-driven" manner. They focus on generating diverse inputs or triggering bugs without explicitly modeling which specific compiler regions they exercise. Consequently, these methods tend to repeatedly exercise frequently triggered "hotspot" paths, leaving substantial "long-tail" coverage gaps in under-tested files and hard-to-reach edge regions. While white-box techniques like WhiteFox have attempted to address this by summarizing optimization code, they often rely on coarse, file-level summarization that fails to provide the fine-grained guidance necessary to reach specific uncovered regions, particularly those requiring specific compilation options or program structures.

Methodology: GapForge

GapForge is a targeted, LLM-based compiler test generation technique designed to systematically improve source-code coverage by reasoning about coverage gaps. Unlike general fuzzers, GapForge treats uncovered code regions as explicit targets and iterates through a three-step process to generate test programs that exercise these specific gaps.

1. Coverage-Driven Target Selection

To address the uneven coverage potential across thousands of source files, GapForge employs a probabilistic selection mechanism rather than random sampling.

Exploration Score: Each file $f$ is assigned a score $S_f = L_f \times (1 - C_f)^2$ , where $L_f$ is the line count and $C_f$ is the current coverage ratio. The quadratic term amplifies the priority of files with low coverage.
Selection Probability: To prevent high-scoring files from dominating the selection process indefinitely, the scores are normalized and transformed into a selection probability $P_f = 1 - (1 - W_f)^k$ . This non-linear transformation ensures that while high-potential files are favored, lower-potential files still receive a chance to be selected, balancing exploration and exploitation.

2. Targeted Summarization

Instead of summarizing an entire file, GapForge performs fine-grained analysis on specific uncovered line spans.

Context Pairing: For each uncovered region, the system pairs it with its immediately enclosing "covered context" (the code surrounding the gap).
Path-Difference Analysis: An LLM analyzes the contrast between the covered context and the uncovered target. It infers:
- Functional Role: The high-level purpose of the file within the compiler pipeline.
- Triggering Requirements: Specific program structures (e.g., data types, control flow patterns) needed to steer execution into the uncovered block.
- Compilation Options: Specific flags (e.g., -fsanitize, -fdump-ada-spec) required to activate the code path.
Output: This process yields structured "Target Requirements" that serve as precise instructions for the next generation step.

3. Prompt Synthesis with Failure Reflection

GapForge synthesizes a structured prompt for the test generation LLM by integrating three components:

General Constraints: Rules ensuring generated programs are valid, deterministic, and structurally complex enough to trigger deep compiler paths.
Target Requirements: The specific goals and constraints derived from the targeted summarization step.
Failure Reflection: A per-file log of previously failed prompts (those that resulted in compilation errors or failed to cover new regions). The system retrieves a recent failure case and instructs the LLM to avoid structurally similar programs, creating an iterative refinement loop that steers generation away from unproductive strategies.

Key Contributions

GapForge Framework: An LLM-based technique that explicitly targets coverage gaps through a three-step workflow: coverage-driven file selection, fine-grained region analysis (combining uncovered code with covered context), and failure-aware prompt synthesis.
Fine-Grained Reasoning: Unlike prior white-box approaches that summarize files holistically, GapForge infers specific triggering requirements (program structures and compilation options) for individual uncovered basic blocks.
Empirical Validation: Extensive experiments on GCC 14.3.0 and LLVM 19.1.0 demonstrating significant improvements over state-of-the-art baselines.
Reproducibility: A public package containing the source code and data for the technique.

Experimental Results

The authors evaluated GapForge against eight state-of-the-art techniques (including Csmith, WhiteFox, LegoFuzz, and Fuzz4All) on GCC and LLVM over a 72-hour period.

Coverage Improvement: GapForge achieved 68.13% coverage on core GCC modules and 69.11% on core LLVM modules. This surpassed the previous state-of-the-art white-box technique (WhiteFox) by covering an additional 24,736 lines in GCC and 19,798 lines in LLVM.
Incremental Gains: GapForge improved the compilers' own official test suites by adding 3,452 new covered lines in GCC and 531 in LLVM, whereas several baselines added zero new lines to the official suites.
Efficiency: GapForge achieved these results with significantly lower token consumption (976K tokens for GCC) compared to WhiteFox (2.30M tokens) and LegoFuzz (3.63M tokens).
Bug Discovery: The technique discovered 12 real-world compiler failures (5 in GCC, 7 in LLVM), including 8 crashes and 4 miscompilations.
Ablation Studies: Removing any of the three core components (target selection, targeted summarization, or failure reflection) resulted in measurable performance degradation, confirming the necessity of each module. Specifically, the absence of targeted summarization or compilation option recommendations caused the most significant drops in coverage.

Significance

The paper posits that GapForge addresses a critical limitation in current compiler testing: the inability of existing techniques to systematically reach "long-tail" coverage gaps. By shifting from a program-driven approach to a coverage-gap-driven approach, GapForge provides a mechanism to explicitly reason about how to reach specific, untested code regions. The results suggest that combining coverage feedback with fine-grained LLM reasoning allows for more efficient and effective testing of large-scale, complex software systems like compilers, ultimately leading to higher reliability and the discovery of subtle defects that remain hidden to traditional fuzzing methods.