💻 computer science

ClozeMaster: Fuzzing Rust Compiler by Harnessing LLMs for Infilling Masked Real Programs

The paper introduces ClozeMaster, a fuzzing tool that leverages large language models to synthesize valid Rust test programs by extracting and infilling masked code snippets from historical bug reports, thereby outperforming existing fuzzers and discovering 27 confirmed compiler bugs.

Original authors: Hongyan Gao, Yibiao Yang, Maolin Sun, Jiangchang Wu, Yuming Zhou, Baowen Xu

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyan Gao, Yibiao Yang, Maolin Sun, Jiangchang Wu, Yuming Zhou, Baowen Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very strict, high-end chef (the Rust compiler) who is famous for making incredibly safe and fast meals. This chef has a very specific set of rules: if you try to put ingredients in the wrong order, use a forbidden spice, or mix textures that don't belong together, the chef doesn't just say "no." Sometimes, the chef gets so confused by your weird request that they drop the whole tray, scream, or freeze up completely. These are the "bugs" the paper talks about.

The problem is that it's very hard to trick this chef. If you just throw random ingredients at them (like a standard "fuzzer" or automated tester), they usually just ignore you or say "invalid order" politely. They rarely get confused enough to crash.

Enter ClozeMaster, the new method proposed by the researchers. Here is how it works, using a simple analogy:

The "Mad Libs" Strategy

Instead of asking the chef to guess what you want from scratch, the researchers decided to look at past mistakes. They went into the chef's complaint log (the "bug reports") and found recipes that had previously made the chef crash or freeze.

They realized these "crash recipes" had a special structure. They were like a game of Mad Libs (a word game where you fill in blanks).

The Masking (The Blank): The researchers took a recipe that once made the chef crash. They looked at the parts inside the parentheses, brackets, and braces (like ( ), { }, [ ]). They covered those specific parts with a "mask" (a blank space), leaving the rest of the recipe intact.
- Analogy: Imagine a recipe that says: "Mix the flour with the ______ and bake." The bold part is the blank. The rest of the sentence (the structure) is safe and grammatically correct.
The AI Filler (The LLM): They used a smart AI (a Large Language Model) to fill in those blanks. But they didn't just ask the AI to write a random sentence. They first "trained" the AI on all those old, broken recipes so the AI learned the specific "flavor" of code that confuses the Rust chef.
- Analogy: The AI is like a student who has studied every time the chef got angry. When asked to fill in the blank, the AI doesn't just write "sugar"; it might write something weird like "a live octopus" because it knows the chef gets confused by strange combinations.
The Result: Because the structure of the recipe was perfect (it was based on a real, working recipe), the chef accepted it. But because the filled-in part was weird and complex, the chef got confused and crashed. This revealed a new bug.

Why This Was Needed

The researchers found that Rust is like a language with very complex grammar rules.

Old methods were like throwing darts blindfolded at a board. They rarely hit the bullseye (the bug) because the rules are too strict.
Direct AI generation was like asking a student to write a recipe from scratch without looking at any examples. The student (the AI) didn't know enough about Rust's strict rules and wrote nonsense that the chef immediately rejected.
ClozeMaster was like giving the student a partially written, complex recipe and asking them to finish just one tricky sentence. The student knew the context, so they wrote something that fit the rules but was still weird enough to break the chef.

What They Found

The team built a prototype tool called ClozeMaster and tested it on two versions of the Rust chef (the official one and a community-made one).

They found 37 new bugs (crashes or freezes) that no one had seen before.
The developers confirmed 27 of them and fixed 10 of them.
Their tool was much better at finding these bugs and exploring the chef's "kitchen" (code coverage) than the previous best tools.

The Takeaway

The paper claims that by taking old, broken code, hiding specific parts of it, and using an AI to fill in those parts, they can create new, tricky tests that expose hidden weaknesses in the Rust compiler. It's a way of using history to predict where the future mistakes might hide.

1. Problem Statement

The Rust programming language is increasingly adopted for critical systems due to its memory and thread safety guarantees. However, the Rust compiler itself is complex, and bugs within it can lead to undefined behavior or crashes in production systems.

Challenge 1: Syntax Complexity. Rust has a steep learning curve and strict syntax rules (ownership, borrowing, lifetimes), making it difficult to generate valid test programs using traditional random generation or mutation-based fuzzing.
Challenge 2: LLM Limitations. While Large Language Models (LLMs) show promise in software testing, directly prompting them to generate Rust test cases often results in invalid code or code that fails to trigger compiler bugs. This is due to the "long-tail" problem: Rust constitutes a small fraction of LLM training data, and existing models struggle with the specific, evolving edge cases of the Rust compiler.
Challenge 3: Lack of Context. Existing LLM-based fuzzers often rely on zero-shot generation or fine-tuning without leveraging the specific structural characteristics of historical bug-triggering code.

2. Methodology: ClozeMaster

The authors propose ClozeMaster, a fuzzing framework that combines historical bug data with LLM capabilities using a novel ClozeMask strategy. The workflow consists of four main steps:

A. Dataset Construction & Augmentation

Source: The system collects code snippets from historical Rust compiler bug reports (specifically rustc issues labeled "C-bug" and "T-compiler"), the official Rust test suite, and the Glacier repository (containing Internal Compiler Error triggers).
Augmentation: To increase the dataset size for fine-tuning, the authors apply data augmentation techniques:
- Random Deletion (RD): Randomly removing code tokens (probability $p=0.2$ ).
- Random Swapping (RS): Swapping the positions of two statements within a program.
- This expanded the dataset from ~25k to 100k snippets.

B. Fine-Tuning the LLM

Model: The open-source Incoder model is selected for its ability to handle code infilling tasks.
Training: The model is fine-tuned on the curated dataset of historical bug-triggering code. This teaches the LLM the specific syntax and semantic patterns associated with Rust compiler vulnerabilities.

C. The ClozeMask Strategy (Core Innovation)

Instead of generating code from scratch, ClozeMaster uses a "fill-in-the-blank" approach:

Masking: A code snippet from the dataset is selected. A Depth-First Search (DFS) identifies bracket structures ((), {}, [], <>). The content inside these brackets is replaced with a [mask] token.
- Rationale: Brackets define Rust's syntax and semantics. Preserving the outer structure ensures the generated code remains syntactically valid, while the masked inner content allows the LLM to explore edge cases.
Infilling: The fine-tuned LLM is prompted to fill in the masked portions.
- Special handling is applied to "feature" blocks (e.g., #![feature(...)]) to encourage the generation of experimental features known to be unstable.
- If the infilled code is identical to the original, it is discarded to ensure novelty.

D. Oracle and Filtering

Oracles: The system compiles the generated code using rustc and mrustc.
- ICE (Internal Compiler Error): Detected via crash or "internal compiler error" messages.
- Hang: Detected if compilation exceeds a 180-second timeout.
Deduplication: Stack traces and time-passes information are stored to filter out duplicate bugs.

3. Key Contributions

ClozeMask Strategy: A novel approach that masks bracket-enclosed code segments in historical bug reports and uses LLMs to infill them. This preserves the structural integrity required for Rust while leveraging LLMs to generate novel, potentially buggy variations.
CLOZEMASTER Framework: The first fuzzing tool specifically designed for Rust compilers that integrates historical bug data, data augmentation, and LLM fine-tuning.
Empirical Validation: Extensive evaluation on two compilers (rustc and mrustc) demonstrating superior performance over existing baselines.

4. Experimental Results

The authors evaluated ClozeMaster against three baselines: RustSmith (generative), Rustlantis (generative), and SPE (mutation-based).

Bug Detection:
- Total Reported: 37 bugs.
- Confirmed: 27 bugs (10 fixed by developers).
- Comparison: ClozeMaster found 11 unique bugs on rustc v1.73, whereas baselines found 0–1.
- Novelty: Many bugs were latent in versions as old as v1.56 (over 2 years old), evading detection by other tools.
Code Coverage:
- ClozeMaster achieved 64.34% code coverage.
- This represents a 95.92% relative increase over RustSmith (32.84%) and 117.73% over Rustlantis (29.55%).
- It also outperformed SPE (62.02%).
Component Coverage: Bugs were found across diverse compiler components, including parsers, macros, traits, lifetimes, and const generics. Notably, the "generic const exprs" feature was a major source of ICEs.

5. Significance and Discussion

Effectiveness of Historical Data: The study confirms that code snippets historically triggering bugs contain valuable domain-specific knowledge. Using them as a base for masking/filling is more effective than random generation or pure LLM generation.
Importance of Context: Ablation studies showed that removing historical bug data or using a non-fine-tuned model significantly degraded performance. Furthermore, using an LLM without the "cloze" context (generating from scratch) resulted in mostly invalid syntax.
Model Selection: The paper highlights that general-purpose LLMs (like GPT-4o) or models trained on general code (StarCoder) perform poorly on Rust infilling due to syntax errors. Incoder, trained with random masking strategies, proved superior for this specific task.
Impact: ClozeMaster demonstrates that combining historical bug patterns with LLM infilling is a viable and powerful strategy for testing complex, safety-critical compilers like Rust's, uncovering deep edge cases that traditional fuzzers miss.

Artifact Availability: The implementation and bug list are publicly available at https://github.com/clozeMasterPro/clozeMaster.