Technical Summary: Data-driven Test Generation for Fuzzing AI Compiler

Problem Statement

Artificial Intelligence (AI) compilers (e.g., TVM, TensorRT, ONNXRuntime) are essential for deploying AI models across diverse hardware, yet they remain prone to bugs that compromise compiler reliability and model correctness. The compilation process involves three distinct stages: (1) Model Loading, where models from libraries like PyTorch or Keras are converted to high-level Intermediate Representations (IR); (2) High-level Optimization, involving hardware-independent graph transformations; and (3) Low-level Optimization, which applies hardware-specific transformations to generate executable code.

Existing testing techniques (e.g., NNSmith, HirGen, Tzer) primarily focus on generating valid and diverse inputs but fail to provide comprehensive, stage-specific coverage. Specifically, they struggle with:

1. Diverse Frontend Coverage: Ensuring all operator conversion logic from various AI libraries is correctly handled during model loading.
2. Optimization-Stage Awareness: Generating computational graphs that effectively trigger complex, context-sensitive high-level optimizations.
3. Hardware-Specific Validation: Creating tests that incorporate features necessary to trigger low-level, hardware-specific optimizations.

Methodology

The authors propose OPERA, a unified data-driven testing framework that integrates three complementary components, each targeting a specific stage of the AI compilation pipeline:

1. OPERA: Migration-Based Model Loading Testing

OPERA addresses the model loading stage by migrating existing tests from AI libraries.

• Approach: It instruments APIs in AI library source code to execute library tests and extract operator instances. These instances are encapsulated into single-operator models via template-based generation.
• Optimization: To manage the high cost of running migrated tests, OPERA employs a diversity-aware prioritization algorithm. This clusters operators by semantics to maximize coverage while eliminating redundancy.
• Goal: To maximize coverage of diverse operator conversion logic by leveraging the core semantics already verified by library tests.

2. OATest: Synthesis-Based High-Level Optimization Testing

OATest targets high-level optimizations by synthesizing optimization-aware computational graphs.

• Approach: It instruments the compiler source code to extract optimization patterns from tests crafted by compiler developers. The process involves identifying the computational graph corresponding to an optimization and deriving patterns based on the optimization's granularity.
• Synthesis: OATest combines these extracted patterns with diverse contexts (seed graphs). It randomly selects a pattern and a seed, inserting the pattern at a random synthesis point to enhance structural diversity.
• Validity Guarantees: To prevent invalid tests caused by undefined inputs, OATest employs two fixing strategies: (1) reusing compatible inputs/outputs from existing nodes, and (2) creating new nodes to establish correct connections between patterns and contexts.

3. HARMONY: Mutation-Based Low-Level Optimization Testing

HARMONY focuses on hardware-specific optimizations using a mutation-based approach.

• Seed Generation: Instead of generating tests from scratch, HARMONY uses Large Language Models (LLMs) guided by operator constraints collected from multiple knowledge sources to generate valid, diverse low-level IR seeds. This cross-checking mitigates risks associated with outdated documentation.
• Mutation: An analysis LLM extracts optimization patterns from documentation and code examples. A code LLM then performs lightweight mutations on the existing IR seeds according to these patterns.
• Goal: To produce semantically valid tests that are specifically likely to trigger targeted low-level optimizations (e.g., memory latency hiding, intrinsic mapping) while minimizing the risk of invalid outputs caused by LLM hallucinations.

Key Contributions

The paper presents three specific techniques that form a unified framework:

1. OPERA: A migration-based approach that leverages existing AI library tests to uncover bugs in operator conversion logic.
2. OATest: An optimization-aware synthesis technique that extracts semantic patterns from developer-crafted tests to generate context-sensitive computational graphs.
3. HARMONY: A mutation-based technique that utilizes documentation and LLMs to generate and mutate low-level IR seeds for hardware-specific optimization testing.

Results

The framework was evaluated across four widely used AI compilers (TVM, TensorRT, OpenVINO, and ONNXRuntime), detecting 266 previously unknown bugs. The breakdown by component is as follows:

• OPERA: Uncovered 170 bugs (90 confirmed by developers) across eight frontends. The prioritization strategy improved testing efficiency by 11.9%–47.4% compared to general strategies. Bug root causes spanned categories such as Tensor Shape Problems, Type Problems, and Incorrect Code Logic.
• OATest: Discovered 56 bugs (42 confirmed, 24 fixed) in TVM and ONNXRuntime. It achieved 60.2% higher branch coverage and 66.98% higher line coverage compared to NNSmith and WhiteFox.
• HARMONY: Detected 40 bugs (26 confirmed and fixed) in TVM. It demonstrated substantially higher coverage at the low-level optimization stage compared to state-of-the-art fuzzers like NNSmith, WhiteFox, and Tzer.

Significance and Claims

The paper claims that this work provides a comprehensive, stage-aware framework that systematically addresses the specific challenges of each compilation stage, rather than treating the compiler as a monolithic black box. By integrating migration, synthesis, and mutation techniques driven by data (library tests, developer patterns, and documentation), the framework significantly enhances testing coverage and effectiveness.

The authors position this work as a step toward ensuring the reliability of AI compilers across the entire pipeline. They note that while the current results are significant, future work will extend the framework to emerging compilers (e.g., Triton, MLC-LLM) and investigate automatic bug localization and repair to further reduce developer maintenance efforts.