Fuzzing with Agents? Generators Are All You Need

1. Problem Statement

Fuzz testing aims to find bugs by executing programs with automatically generated inputs. A fundamental challenge is generating inputs that are both structurally valid (passing syntax checks) and semantically meaningful (reaching deep program logic).

• The Limitation of Mutation: Traditional coverage-guided fuzzing (CGF) relies on mutating existing inputs. For complex targets (e.g., compilers, parsers), random byte mutations often fail early validation checks, preventing the fuzzer from reaching deep logic where bugs reside.
• The Limitation of Manual Generators: Prior research developed "heavyweight" domain-specific generators that encode structural and semantic constraints directly. While effective at reaching deep states without mutation, these require significant manual effort to write and are tightly coupled to specific targets.
• The Research Question: Can AI coding agents automatically synthesize these high-quality, target-specific input generators such that they are strong enough to obviate the need for coverage guidance and mutation strategies entirely?

2. Methodology: Gentoo

The authors propose Gentoo, a system that uses an LLM-based coding agent to iteratively synthesize and refine input generators for fuzzing.

Core Components

1. Agent Loop: The agent is provided with terminal access, the source code of the fuzz target, and its libraries. It is instructed to write a generator (specifically a JQF-style generator) and iteratively refine it.
2. Feedback Mechanisms (Three Configurations):
   • Gentoo-Base: A "plain" agent with no specialized guidance. It refines the generator based solely on raw execution statistics and inspection of generated inputs.
   • Gentoo-S (Static): Uses interprocedural dominator analysis to rank conditional branches (predicates) by their "branch dominance score." This score quantifies how much unexplored code is controlled by a specific branch. The agent is given a list of high-value predicates to target.
   • Gentoo-L (LLM): Instead of static analysis, the agent itself scans the source code to identify and rank important predicates based on its natural language understanding of the code's intent.
3. Dynamic Predicate Feedback: Both Gentoo-S and Gentoo-L receive fine-grained feedback after each fuzzing iteration. This report shows exactly which branch outcomes of high-value predicates were triggered (or missed) by the current generator, allowing the agent to identify specific semantic gaps.

The Workflow

1. The agent reads the target library and fuzz driver.
2. It synthesizes an initial generator.
3. The generator runs a short fuzzing session (2 mins).
4. The system provides a predicate coverage report (showing which deep branches were missed).
5. The agent analyzes the gap, edits the generator code to encode the missing logic, and repeats.

3. Key Contributions

1. Agentic Synthesis of Generators: Demonstrated that LLM agents can autonomously synthesize complex, domain-specific input generators that encode deep semantic logic (e.g., inheritance hierarchies, type constraints) directly into the generation process.
2. Decoupling from Mutation: Showed that high-quality agent-synthesized generators can reach deep program states without relying on coverage-guided mutation, effectively making the fuzzer a simple execution harness.
3. Predicate Feedback Evaluation: Investigated whether static analysis (Gentoo-S) or LLM reasoning (Gentoo-L) is better for guiding the agent, finding that both are effective, though they reason differently.
4. Comprehensive Benchmarking: Evaluated the approach against human-written baselines across 7 real-world Java libraries (including BCEL, Bouncy Castle, Closure, and ChocoPy).

4. Results

The evaluation involved 7 benchmarks and compared agent-synthesized generators against human-written baselines under both coverage-guided and random (no mutation) modes.

• Coverage Superiority: Agent-synthesized generators achieved statistically significantly higher branch coverage than human-written generators on 4 out of 7 benchmarks (Closure, Commons, Gson, Rhino). On average, they covered 11–21% more branches.
  • Example: On the Rhino benchmark, Gentoo-S covered 31% more branches by synthesizing specific JavaScript constructs (e.g., with statements, destructuring) that the human generator missed.
  • Exception: On BCEL, agents failed to discover the library's high-level API for constructing class files, resulting in 54% lower coverage than the human baseline.
• Redundancy of Coverage Guidance:
  • For agent-synthesized generators, adding coverage guidance (mutation) provided no statistically significant improvement over random sampling (differences were < 3%).
  • For human-written generators, coverage guidance was crucial, providing an average 40% improvement (up to 229% on Commons).
  • Conclusion: Once the generator encodes the necessary structural and semantic logic, mutation strategies become largely unnecessary.
• Impact of Feedback:
  • Predicate feedback improved coverage in 74% of runs during the iterative refinement process.
  • However, the final coverage of Gentoo-Base, Gentoo-S, and Gentoo-L was broadly comparable, suggesting that a well-prompted baseline agent can often synthesize strong generators without explicit predicate guidance.
• Static vs. LLM Analysis:
  • Gentoo-S (Static) ranked predicates by structural dominance (how much code they control).
  • Gentoo-L (LLM) ranked predicates by semantic difficulty (how hard it is to satisfy the condition).
  • Both achieved similar final results, but Gentoo-L was better at filtering out "easy" high-dominance branches (e.g., token streams) that didn't require complex generation logic.

5. Significance and Implications

• Paradigm Shift: The paper argues for a shift from "Fuzzer Engineering" (optimizing mutation strategies and power schedules) to "Generator Synthesis." If a generator can encode the target's constraints, the fuzzing loop becomes trivial.
• Agents as Bespoke Analysts: The work demonstrates a new paradigm where AI agents act as constructors of application-specific analyses. Instead of applying a generic tool to every target, the agent reads the source, understands the specific constraints of that target, and builds a custom generator.
• Future of Automated Testing: As agents become more capable, the barrier to building complex, semantic-aware fuzz generators will drop, potentially making generator-based fuzzing a primary, first-class option for automated software validation, reducing reliance on mutation-based exploration.

In summary, the paper proves that AI agents can synthesize input generators so effective that they render traditional coverage-guided mutation obsolete for the targets they cover, provided the agents are given the right context and feedback to encode the necessary domain logic.