FunFuzz: An LLM-Powered Evolutionary Fuzzing Framework

Technical Summary: FunFuzz

Problem Statement
Modern fuzzing of language-processing systems, particularly compilers, faces significant challenges due to the highly structured and semantically rich nature of the input space. Traditional approaches often struggle to maintain sustained semantic diversity, frequently converging on shallow patterns that fail to exercise deep compilation pipelines or rare feature interactions. While Large Language Models (LLMs) offer a promising alternative for synthesizing syntactically valid and semantically varied programs, LLM-driven fuzzing is prone to specific failure modes: sensitivity to prompt initialization, sampling variance leading to repetitive program styles, and a tendency for the search trajectory to collapse early without explicit mechanisms to preserve diversity. Existing LLM-assisted fuzzers often rely on a single shared prompt or implicit feedback, which can bias the search and limit the exploration of novel compiler behaviors. Furthermore, adapting fuzzers to new compilers or language variants typically requires extensive manual engineering, reducing portability.

Methodology
FunFuzz is a multi-island evolutionary fuzzing framework designed to integrate LLM-based program synthesis with a robust search procedure that mitigates stochasticity and maintains diversity. The framework operates in two primary stages:

1. Prompt Distillation and Initialization:

   • The system begins by processing user-provided artifacts (documentation, specifications, code examples) using a Distillation LLM to generate a base prompt.
   • To prevent all search trajectories from starting in the same region of the input space, FunFuzz derives per-island seed instructions. These are short directives emphasizing different topics (e.g., allocation patterns, control flow) extracted from the documentation.
   • The system evaluates batches of these instructions (sampled at different temperatures) combined with the base prompt, selecting the batch that yields the highest compile-validity. This results in a unique initial prompt for each island, creating multiple semantically distinct starting points.

2. Evolutionary Fuzzing Loop (Multi-Island):

   • Island Operation: Each island maintains an independent population of candidate programs. Programs are generated by an LLM based on the island's current prompt, normalized to remove formatting artifacts, and compiled against an instrumented compiler.
   • Fitness Scoring: Candidates are ranked based on marginal compiler coverage (the number of new compiler source lines exercised) and failure signals (e.g., internal compiler errors). The fitness function is computed locally per island to encourage independent exploration.
   • Selection and Evolution: Parent programs are selected using a temperature-controlled softmax distribution over fitness scores. The selected parent is incorporated into the next prompt to steer generation, with mechanisms to penalize overfitting to specific programs.
   • Cross-Island Migration: To mitigate stagnation without discarding accumulated search context, FunFuzz employs a "soft migration" policy. Every 3 hours, the top 51% of islands (by coverage) are designated as "strong," and the bottom 49% as "weak." Strong islands share 10% of their elite population with weak islands, replacing the lowest 30% of the weak islands' populations. Unlike previous "FunSearch" approaches that reset weak islands, this method preserves the remaining search context.

Key Contributions

• Multi-Island LLM-Assisted Evolutionary Fuzzing: The introduction of a fuzzing loop that maintains independent populations with local feedback and utilizes periodic migration to balance exploration and exploitation, preventing premature convergence.
• Prompt Distillation with Per-Island Semantic Initialization: An extension of autoprompting techniques that generates distinct seed instructions for each island, ensuring semantically diverse starting points while maintaining syntactic validity.
• Comprehensive Evaluation on Modern Compilers: A rigorous 24-hour evaluation on GCC and Clang for both C and C++, comparing against strong baselines (Fuzz4All and Kitten) and including controlled ablation studies to isolate the impact of core design choices.

Experimental Results
FunFuzz was evaluated over repeated 24-hour campaigns on GCC and Clang. The results demonstrate:

• Coverage Gains: FunFuzz consistently achieved higher compiler source-line coverage than baselines. On GCC, it reached up to +27.1% higher coverage than standard Fuzz4All in C and +31.0% in C++. On Clang, improvements were +9.3% in C and +4.4% in C++.
• Bug Discovery: The framework discovered 119 unique compiler bugs, of which 80 were confirmed by developers. FunFuzz found significantly more unique failure-triggering inputs than both Fuzz4All and Kitten, particularly in C++ where the complexity of templates and semantics is higher.
• Ablation Insights:
  • Scoring: Coverage-based scoring was the primary driver of performance.
  • Initialization: Distinct per-island seeds significantly improved early exploration compared to shared seeds.
  • Migration: "Soft migration" (preserving context) outperformed full resets of weak islands.
  • Parallelism: A configuration with 5 islands provided the optimal trade-off between exploration diversity and evolutionary depth; increasing to 10 or 20 islands fragmented the search and reduced performance under fixed time budgets.
  • Warm-Start: When initialized with a curated corpus (warm-start), FunFuzz achieved even higher coverage, suggesting that LLM-guided evolution complements traditional seed-based approaches.

Significance
The paper positions FunFuzz as a step toward more robust and portable compiler fuzzing. By combining LLM-driven synthesis with a multi-island evolutionary strategy, FunFuzz addresses the limitations of prompt sensitivity and search collapse inherent in single-prompt LLM fuzzing. The framework demonstrates that it is possible to sustain long-term exploration pressure toward semantically rich programs without relying on extensive manual engineering or target-specific heuristics. The results suggest that maintaining multiple distinct search trajectories and sharing high-value candidates without resetting context is an effective strategy for uncovering deep defects in complex, evolving compiler systems. The authors note that while the evaluation is currently limited to C/C++ compilers, the design is intended to be agnostic to the system under test, with future work planned for other structured-input domains.