Human in the Loop for Fuzz Testing: Literature Review and the Road Ahead

1. Problem Statement

Fuzz testing (fuzzing) is a dominant technique for discovering software vulnerabilities, yet it faces significant limitations when dealing with complex, semantic-rich, or deep program behaviors.

• Limitations of Automation: Automated fuzzers often struggle with semantic roadblocks (e.g., complex protocols, state machines), constructing robust drivers, and interpreting vast amounts of runtime data. They frequently plateau in code coverage due to a lack of domain-specific reasoning.
• Fragmented Human Integration: While existing research acknowledges the value of human expertise (Human-in-the-Loop or HITL), current approaches are fragmented. They often treat humans merely as initial configuration providers or final report reviewers, rather than active participants during the fuzzing execution.
• The LLM Challenge: The rise of Large Language Models (LLMs) offers new automation capabilities but introduces risks like hallucination and brittleness. There is currently no systematic roadmap for integrating human oversight with LLM-assisted fuzzing to balance efficiency and reliability.

2. Methodology

The authors conducted a systematic literature review to bridge the gap between fuzzing, Human-Computer Interaction (HCI), and LLM-assisted testing.

• Search Strategy:
  • Sources: Google Scholar, IEEE Xplore, ACM Digital Library, and DBLP.
  • Keywords: Combinations of fuzzing terms (e.g., "fuzzing," "fuzz test") with human interaction terms (e.g., "visualization," "dashboard," "human," "interactive," "guided").
  • Snowballing: Both backward (reviewing references) and forward (checking citations) snowballing were employed to ensure comprehensive coverage.
• Selection Criteria:
  • Papers must explicitly integrate human involvement (monitoring or steering) into the fuzzing process.
  • Excluded: Papers where humans only reviewed final results, fully automated tools, non-peer-reviewed works, or papers with fewer than 4 pages.
  • Final Corpus: 44 papers (35 technical research articles, 9 surveys) published between 2007 and 2025.
• Analytical Framework: The review categorized human involvement across the five canonical phases of fuzzing:
  1. Preprocessing
  2. Seed Management
  3. Scheduling
  4. Testcase Synthesis
  5. Execution
  • Interaction Modes: Categorized into Human Monitoring (passive observation/analysis) and Human Steering (active intervention/guidance).

3. Key Contributions

The paper makes three primary contributions:

1. Systematic Literature Review: A structured taxonomy of 44 HITL fuzzing works, analyzing their strengths, limitations, and application scenarios across the fuzzing lifecycle.
2. Interaction Design Space Analysis: A mapping of the design space for human involvement, highlighting the trade-offs between passive monitoring and active steering, and identifying that current efforts are heavily skewed toward monitoring in the execution phase and steering in the seed management phase, with a notable gap in real-time steering during execution.
3. Visionary Research Roadmap: A three-stage roadmap for the future of HITL fuzzing, specifically addressing the integration of LLMs to create a synergistic human-LLM collaboration model.

4. Key Results & Findings

• Current State of HITL:
  • Imbalance: Most existing work focuses on Human Monitoring (visualizing crashes, coverage, and seed distributions) during the Execution phase.
  • Steering Gaps: Human Steering is mostly limited to Preprocessing and Seed Management (e.g., annotating code, curating seeds). There is a critical lack of tools allowing real-time, in-situ steering of scheduling or mutation strategies during execution.
  • Decline in Traditional HITL: The number of traditional HITL studies has declined as the field shifts toward LLM-assisted fuzzing, where LLMs are increasingly substituting for human roles without sufficient human oversight mechanisms.
• Limitations Identified:
  • High cost of human labor and lack of scalability.
  • Poor generalization across different domains and testers.
  • Lack of explainability in why automated heuristics fail.
  • LLMs alone suffer from hallucinations and lack of accountability.

5. The Road Ahead: A Three-Stage Research Agenda

The authors propose a progressive evolution of HITL fuzzing:

Stage I: Towards Fine-Grained and Explainable Fuzzing Monitoring

• Goal: Move beyond coarse metrics (e.g., total coverage) to deep, multi-modal observability.
• Directions:
  1. Standardized Observability Model: Create a unified data schema to integrate signals from all five fuzzing phases (preprocessing to execution), enabling cross-tool comparison and root-cause analysis.
  2. Interactive Visualization: Develop diagnostic tools that allow testers to trace inefficiencies (e.g., mutation operator effectiveness, seed selection bias) and perform "what-if" analyses.

Stage II: Achieving Real-time and Strategic Human Steering

• Goal: Transition from offline, post-execution adjustments to in-situ, real-time intervention.
• Directions:
  1. Runtime Pipelines: Design fuzzing loops that expose actionable runtime data (e.g., incremental coverage per seed) and accept dynamic human guidance without restarting the fuzzer.
  2. Advanced Interaction Techniques: Move away from code-level modifications to parameterized workflows, visual programming interfaces (drag-and-drop modules), and natural language interfaces for steering (e.g., "increase focus on parsing routines").

Stage III: Towards Synergistic Human–LLM Collaboration

• Goal: Establish a symbiotic relationship where LLMs handle low-level tasks and humans provide high-level strategic oversight.
• Directions:
  1. Automating Low-Level Effort: Use LLMs to automate repetitive tasks like seed deduplication, driver generation, and crash triaging.
  2. Orchestrating High-Level Strategies: Deploy multi-agent LLM systems (e.g., Explorer, Analyst, Strategist agents) to manage fuzzing plans. Humans act as "commanders," setting goals and validating strategies, while LLMs execute the adaptive logic.

6. Significance

• Paradigm Shift: The paper advocates for a fundamental shift from viewing humans as mere setup/configurators to viewing them as active, real-time collaborators in the fuzzing loop.
• Bridging AI and Expertise: It provides a critical framework for integrating LLMs into fuzzing without losing the reliability and domain context that human experts provide, addressing the "hallucination" risk of AI.
• Future-Proofing: By defining a clear roadmap, the paper guides the research community toward building the next generation of explainable, adaptive, and scalable fuzzing ecosystems that combine the speed of automation with the intuition of human expertise.