Hawkeye: Reproducing GPU-Level Non-Determinism

1. Problem Statement

Modern machine learning (ML) relies heavily on cloud-based services for training and inference. To ensure these services are trustworthy (i.e., they are not cutting corners, using smaller models, or introducing backdoors), third-party auditors need to verify that computations were executed correctly.

However, Verifiable ML faces a fundamental hurdle: Non-determinism in GPU hardware.

• The Issue: Modern ML workflows use specialized hardware like NVIDIA Tensor Cores. These units perform matrix multiplications using floating-point arithmetic, which is inherently non-associative.
• Sources of Variance: Differences in rounding strategies, handling of subnormal numbers, and the specific order of accumulation can cause the same input on different GPUs (or even the same GPU with different schedules) to produce slightly different numerical results.
• The Consequence: An auditor cannot simply re-run a computation on a CPU and compare the output bit-for-bit with the GPU's output. Even a perfectly correct execution might fail a bitwise comparison due to hardware-induced numerical drift. Existing solutions either introduce massive computational overhead, degrade model quality, or rely on unproven heuristics that ignore these discrepancies.

2. Methodology: The Hawkeye Framework

Hawkeye is a system designed to faithfully reproduce the exact bit-level behavior of NVIDIA Tensor Cores on a standard CPU. Instead of trying to force determinism (which slows down GPUs), Hawkeye reverse-engineers the specific numerical pipeline of the GPU to create a software simulator that mimics it exactly.

Core Approach

The authors developed a systematic methodology involving targeted CUDA kernel tests to characterize the internal behavior of Tensor Cores. By isolating specific variables, they reconstructed the exact accumulation pipeline.

Key Characterization Tests:

1. Summation Dependency & Order: Determined the specific sequence in which partial products are summed. The authors used "computationally neutral subgroups" to map the hierarchical structure of the accumulation (e.g., identifying if the hardware sums 8 products, then adds the next 8, vs. summing all 16 at once).
2. Internal Precision Detection: Identified the effective precision of the internal accumulator. They tested how small values survive addition to determine if the internal register uses FP32 (24-bit significand) or a higher precision (e.g., 25-bit).
3. Rounding Mode Detection: Determined how low-order bits are handled during alignment shifts and final conversion. Tests revealed whether the hardware uses "round to nearest," "round to even," or "truncate" (round towards zero).
4. Normalization Timing: Investigated whether intermediate products are normalized immediately after multiplication or if raw products are passed to the accumulator (deferred normalization).
5. Subnormal Handling: Analyzed how the hardware treats subnormal inputs (numbers close to zero) and whether they are renormalized before accumulation.

Architecture-Specific Findings

The framework successfully characterized three distinct architectures:

• Ampere (e.g., A100): Uses a two-stage accumulation (first 8 products + accumulator, then last 8 products). Internal precision is 24-bit (FP32). Rounding is truncate (towards zero). Normalization is deferred.
• Hopper (e.g., H100): Uses a single-stage accumulation (all 16 products summed together). Internal precision is 25-bit. Rounding is truncate.
• Lovelace (e.g., L40S): Follows the same structural logic as Ampere.
• Precision Types: The methodology was validated for FP16, BF16, and FP8.

3. Key Contributions

1. Systematic Characterization Framework: Hawkeye provides the first systematic suite of tests to reverse-engineer the hidden numerical behaviors of Tensor Cores, moving beyond black-box assumptions.
2. Bit-Exact CPU Reproduction: The authors successfully implemented software simulators that replicate the exact bit-level output of NVIDIA Tensor Cores on a CPU for matrix multiplication.
3. Discovery of Hardware Nuances: The paper reveals specific, undocumented behaviors, such as:
   • The specific two-stage vs. single-stage accumulation trees.
   • The use of truncation (round towards zero) rather than round-to-nearest during internal shifts and final conversion.
   • The handling of subnormal numbers (propagating reduced precision rather than renormalizing).
   • The ability of BF16 Tensor Cores to temporarily hold values exceeding FP32 range during accumulation before casting back.
4. Open Source Implementation: The authors released a public repository containing the simulators for Ampere, Hopper, and Lovelace architectures across FP16, BF16, and FP8.

4. Results

• Accuracy: The Hawkeye simulators achieved a 100% success rate in bit-exact replication across 100,000 randomly generated 16x16 matrix multiplications across different architectures and precision types.
• Performance: While the CPU simulation is slower than native GPU execution, the authors note that for auditing purposes, a single re-execution is sufficient. The overhead is acceptable for the goal of verification.
• Validation: The framework successfully reproduced complex edge cases, including overflow handling in BF16 and the preservation of small terms in FP16 accumulation that would otherwise be lost in standard IEEE 754 implementations.

5. Significance and Impact

• Enabling Verifiable ML: Hawkeye solves the "non-determinism" blocker in verifiable computing. Auditors can now run a reference simulation on a CPU and compare it bit-for-bit with a GPU execution. Any discrepancy now definitively indicates a malicious or incorrect execution by the service provider, rather than hardware noise.
• Trust in Cloud AI: This paves the way for trustworthy "ML-as-a-Service" models where clients can verify that their models are being trained or inferred correctly without needing to own the expensive hardware.
• Hardware Transparency: The paper sheds light on the "black box" nature of modern AI accelerators, providing a blueprint for understanding how specialized hardware handles floating-point arithmetic, which is crucial for debugging and optimizing ML systems.
• Future Directions: While currently focused on Matrix Multiplication (MatMul), the methodology provides a foundation for extending verifiable ML to other operations (like convolutions and attention mechanisms) and other hardware vendors.

In summary, Hawkeye transforms the challenge of GPU non-determinism from an unsolvable verification problem into a manageable engineering task by creating a precise, reproducible software model of the hardware's numerical behavior.