Investigating Metamorphic Fuzz Oracle Enhancement via Large Language Models

Technical Summary: Investigating Metamorphic Fuzz Oracle Enhancement via Large Language Models

Problem Statement
Greybox fuzzing relies heavily on fuzz drivers to define the test space and encapsulate target interfaces. However, a systematic analysis of 1,068 drivers from OSS-Fuzz reveals a critical limitation: 81.1% of existing drivers lack functional oracles. Instead, they rely solely on "implicit oracles" (program crashes) or "pseudo oracles" (input validity checks) to detect bugs. This practice overlooks the functional semantics of the libraries under test, restricting the bug-mining capability of fuzzing to only those defects that cause crashes. While metamorphic testing offers a solution by defining relationships between inputs and outputs (Metamorphic Relations or MRs) to detect functional bugs, the manual construction and integration of these oracles into existing drivers are labor-intensive and require deep domain knowledge, creating a barrier to automation.

Methodology: MetaFOE
The authors propose MetaFOE, a novel framework that leverages Large Language Models (LLMs) to automate Metamorphic-based Fuzz Oracle Enhancement (MFOE). The pipeline consists of two primary phases:

1. Metamorphic Relation Generation (MRG): The LLM analyzes the source code and documentation of the target library to generate candidate MRs. These relations describe expected behaviors (e.g., "appending whitespace should not change parsing results").
2. Fuzz Oracle Implementation (FOI): The LLM takes the original fuzz driver and a valid MR to synthesize a "meta driver." This involves injecting explicit oracles (typically assert statements) into the driver to verify the MR during execution.

To ensure quality, the framework employs a rigorous validation process:

• MR Validation: Generated MRs are deduplicated and manually validated by experts for applicability.
• Driver Validation: Generated meta drivers undergo a three-phase check:
  • Compilation-level: Iterative reprompting to fix compilation errors.
  • Runtime-level: Short-term fuzzing to discard drivers that are unstable (excessive crashes) or ineffective (low coverage).
  • Fuzzing Capability: Long-term (3-hour) campaigns to measure edge coverage and crash detection.

The study evaluates MetaFOE using three state-of-the-art LLMs (GLM-4-Plus, DeepSeek-V3.2, GPT-4.1) and five prompting strategies (Zero-shot, Few-shot, Critique, Expert, Chain-of-Thought).

Key Contributions

• Novel Pipeline: This is the first work to propose and implement an automated pipeline for enhancing existing fuzz drivers with metamorphic oracles, shifting focus from driver generation to oracle enhancement.
• Systematic Empirical Study: The authors provide a comprehensive analysis of the current state of fuzz oracles in OSS-Fuzz and conduct extensive experiments consuming 213 million tokens and over two CPU years of fuzzing time.
• Artifact Release: The study generates and releases 6,228 valid meta drivers, along with the dataset and scripts used for the research.

Experimental Results
The evaluation was conducted on 10 representative projects from OSS-Fuzz:

• MR Generation: MetaFOE generated 3,475 MRs, of which 77.3% (2,685) were deemed applicable. The applicability showed a weak negative correlation with project size (LoC).
• Driver Validity: From the applicable MRs, the framework implemented 12,351 meta driver candidates. After validation, 6,228 (50.4%) were valid.
• Fuzzing Performance: Compared to the original drivers, the valid meta drivers achieved:
  • An average 18.7% improvement in edge coverage.
  • The discovery of 1,528 unique crashes across the 10 projects. Notably, original drivers triggered crashes in only 2 of the 10 projects, whereas meta drivers found crashes in all 10.
  • The crashes included both Memory Corruptions (MC) and Assertion Failures (AF), with AFs being significantly more prevalent due to the nature of the injected oracles.
• LLM and Prompt Sensitivity:
  • LLMs: DeepSeek-V3.2 performed best in generating applicable MRs (82.9% applicability), while GPT-4.1 achieved the highest validity rate for generated drivers (55.4%).
  • Prompts: The system was less sensitive to prompt strategies. Chain-of-Thought (COT) yielded the highest number of valid drivers, while Expert (EXP) prompts produced drivers with the best fuzzing capabilities (coverage and crash detection).

Significance and Claims
The paper claims that metamorphic oracles are essential for uncovering functional bugs that crash-based oracles miss. The study demonstrates that LLMs are feasible and effective for automating the complex tasks of extracting metamorphic relations and integrating them into fuzz drivers.

The authors highlight that while LLMs can significantly enhance fuzzing capabilities, they are not perfect; the study observed false positives (e.g., in the Libyaml case study) where MR assumptions did not hold for all edge cases. Consequently, the paper advocates for a hybrid approach where LLMs generate the initial oracles, followed by human or iterative LLM-assisted refinement to mitigate false positives. The work offers a valuable insight to the fuzzing community: moving beyond crash-only oracles via automated metamorphic enhancement can substantially increase the bug-hunting efficacy of greybox fuzzing.