🤖 machine learning

Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL

This paper introduces HEAL, a hybrid error alleviation framework that achieves mission-critical LLM inference reproducibility across heterogeneous GPUs by combining INT16 quantization for KV tensors with algebraic error compensation on 16-bit Tensor Cores, thereby eliminating catastrophic output divergence without the prohibitive performance and memory costs of a global FP32 pipeline.

Original authors: Zhenting Zhu, Lucas Thai, Shan Yu, Yicheng Liu, Yifan Qiao, Chenxi Wang, Harry Xu, Junyi Shu

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenting Zhu, Lucas Thai, Shan Yu, Yicheng Liu, Yifan Qiao, Chenxi Wang, Harry Xu, Junyi Shu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Shaky Hand" of AI

Imagine you are a judge in a high-stakes courtroom (like a medical diagnosis or a legal ruling). You need a witness (the AI) to give you the exact same testimony every time you ask the same question, regardless of which microphone or recording studio you use.

In the world of Large Language Models (LLMs), this is currently a nightmare. Even if you ask the AI the exact same question twice, using the exact same settings, the answer might change slightly.

Scenario: You ask, "What is the code for this disease?"
Run 1: The AI says "Code A."
Run 2: The AI says "Code B."

In a casual chat, this doesn't matter. But in finance, medicine, or law, a single wrong letter can be a disaster. The paper calls this non-reproducibility.

The Investigation: Why is the AI Shaking?

The researchers (from UCLA, Berkeley, etc.) wanted to know: Why does the AI change its mind?

They suspected the AI was doing math in a "fuzzy" way. To find the culprit, they looked under the hood of the computer chips (GPUs) that run these models.

The Misconception:
Most people thought the AI was doing all its math with low precision (like using a ruler with only big, chunky markings) because that's faster. They thought the "fuzziness" came from the math itself.

The Real Culprit (The "Kernel Boundary"):
The researchers discovered the math inside the chip is actually very precise (like using a laser-measured ruler). The problem happens at the boundaries—the doorways between different parts of the calculation.

The Analogy: The Bucket Brigade
Imagine a team of workers passing buckets of water (data) down a line to put out a fire.

Inside the hands: The workers hold the buckets perfectly steady (the math inside the chip is precise).
The Handoff: When a worker passes the bucket to the next person, they have to pour the water into a slightly different container.
The Spill: Every time they pour, a tiny drop of water spills out.
The Result: If the line is short, one drop doesn't matter. But in an LLM, the line is miles long (thousands of layers). Those tiny drops add up. By the time the water reaches the end, the bucket is empty or has the wrong amount, causing the AI to pick the wrong answer.

This "spilling" happens because the computer saves the water in a smaller container (lower precision) to save space, losing a tiny bit of detail every time it moves data.

The Old Solutions: The "Brute Force" Approach

Before this paper, there were two ways to fix this, and both were terrible:

The "Strict Order" Method: Force the workers to pass buckets in a specific, rigid order so no one gets confused.
- Downside: It's incredibly slow and only works on specific types of hardware. If you switch to a different brand of GPU, it breaks.
The "Big Bucket" Method: Stop using small containers entirely. Use giant, heavy, ultra-precise buckets (FP32) for the whole line.
- Downside: It's so heavy that the workers move in slow motion. The AI becomes 13 times slower, making it useless for real-time applications.

The New Solution: HEAL (Hybrid Error ALleviation)

The authors propose a clever middle ground called HEAL. Instead of making the whole line heavy or rigid, they fix the specific spots where the water spills.

1. The "Smart Packing" Trick (For Attention)

The Problem: The "Key" and "Value" buckets (data used for attention) are often mostly empty or have simple numbers. Using a giant bucket for them is wasteful.
The Fix: HEAL uses a special "packing tape" (INT16 quantization). It compresses the data into a smaller, tighter package that fits perfectly.
The Magic: Even though the package is small, the workers unpack it into two smaller buckets (Dual-FP16) to do the math. This keeps the water level high (precise) without needing the giant, heavy buckets that slow everything down.

2. The "Error Compensation" Trick (For Math)

The Problem: When doing heavy math (GEMM), the standard buckets lose a tiny bit of precision.
The Fix: HEAL splits the math into two steps.
- Step A: Do the main math with the standard bucket.
- Step B: Do a quick "cleanup" math with a tiny bucket to catch the tiny drops that spilled in Step A.
- Result: You get the precision of the giant bucket, but you only use the fast, light buckets for the heavy lifting.

The Results: Fast, Accurate, and Reliable

The researchers tested this new method on a new benchmark they created called MCR-Bench (Mission-Critical Reproducibility Benchmark), which includes tough questions from medicine, law, and finance.

Reproducibility: HEAL achieved the same level of "perfect consistency" as the super-slow, heavy "Big Bucket" method.
Speed: It was 7.1 times faster than the heavy method.
Memory: It didn't require extra memory, unlike the heavy method which doubled the memory needs.

The Takeaway

The paper proves that we don't need to slow down the entire AI to make it reliable. We just need to patch the specific "leaks" where data gets lost during handoffs.

In a nutshell:
Instead of replacing the entire race car engine with a slow, heavy tractor engine to ensure it doesn't stall, the authors found a way to tighten the bolts on the existing engine. The car (the AI) now runs just as reliably as the tractor, but it's still fast enough to win the race.

Technical Summary: Demystifying Numerical Instability in LLM Inference with HEAL

Problem Statement

As Large Language Models (LLMs) are deployed into mission-critical domains such as finance, medicine, and law, output reproducibility has become a strict system requirement. While practitioners typically employ greedy decoding to eliminate algorithmic stochasticity, empirical deployments using 16-bit precisions (e.g., BF16) still exhibit catastrophic output divergence across heterogeneous hardware (e.g., different generations of NVIDIA GPUs or AMD accelerators).

Existing mitigation strategies face an unacceptable dilemma:

Strict Bit-Wise Determinism: Enforcing deterministic floating-point reduction orders or using specialized kernels restricts deployment to narrow execution paths, disables modern hardware features, and sacrifices throughput.
Global High-Precision Pipelines: Performing all calculations in FP32 eliminates downcasting noise but incurs prohibitive system penalties. Serving entirely in FP32 can increase the Time Per Output Token (TPOT) by up to 13.1× and doubles the memory footprint of the Key-Value (KV) cache, rendering it economically impractical for large-scale serving.

The core challenge is achieving robust, deployment-ready reproducibility without sacrificing the efficiency of 16-bit hardware accelerators.

Methodology: HEAL (Hybrid Error ALleviation)

The authors propose HEAL, a targeted intervention that approximates FP32-level reproducibility while adhering to hardware constraints. The methodology is grounded in a systematic investigation of the root causes of non-reproducibility, moving from macroscopic failure to microscopic origins.

1. Root Cause Analysis

Through SASS-level profiling and error decomposition, the paper identifies that non-reproducibility is not driven by internal kernel arithmetic (which predominantly operates in FP32 via Tensor Cores) but by truncation errors introduced during downcasting at kernel boundaries.

Microscopic Origin: The instability stems from the mandatory truncation of high-precision registers to lower-precision storage formats at the boundaries of kernels (e.g., writing results from FP32 accumulators back to BF16 memory).
Accumulation: These microscopic losses of fractional precision accumulate exponentially across network layers. Once the accumulated error breaches a specific threshold, it shifts the final log-probabilities enough to alter the top-1 token selection, causing "result flipping."
Key Insight: Robust reproducibility does not require strict mathematical parity with an FP32 ground truth; it only requires bounding numerical noise below the threshold that triggers result flipping.

2. The HEAL Solution

HEAL addresses the two primary bottlenecks of a global FP32 pipeline (Memory for Attention, Compute for GEMM) through two targeted mechanisms:

A. Mitigating Attention Truncation via INT16 Quantization

Problem: Upcasting Key-Value (KV) caches to FP32 doubles memory usage, creating a memory bottleneck.
Solution: HEAL recognizes that standard floating-point formats underutilize their bit-width for Q, K, and V tensors. It applies INT16 quantization to store these tensors.
Mechanism: During execution, the INT16 tensor is algebraically unpacked into two FP16 tensors ( $X_{hi}$ and $X_{lo}$ ). The lower tensor is shifted by a factor of $2^{10}$ , allowing the pair $(X_{hi} + X_{lo}/2^{10})$ to emulate 20 bits of continuous fractional precision.
Benefit: This preserves the strict 16-bit memory footprint of the KV cache while significantly reducing truncation error, effectively preventing divergence without expanding memory usage.

B. Mitigating GEMM Truncation via Algebraic Error Compensation

Problem: Upcasting GEMM inputs to FP32 forces hardware to bypass high-throughput 16-bit Tensor Cores, causing severe performance regression.
Solution: HEAL synthesizes high-precision matrix multiplications using an algebraic error compensation strategy executed entirely on 16-bit Tensor Cores.
Mechanism: Input activations are decomposed into a primary 16-bit value and a residual error term. The system performs two parallel matrix multiplications using standard 16-bit inputs and sums the results. This captures the truncation error while maintaining the throughput of native 16-bit hardware.
Overflow Handling: To address the narrow dynamic range of FP16, HEAL applies targeted rescaling to rare outlier channels (which constitute <1% of fixed channels) to prevent catastrophic overflow without adding significant overhead.

Key Contributions

Diagnosis of Instability: The paper demonstrates that intrinsic non-reproducibility is fundamentally driven by boundary truncation errors rather than internal arithmetic non-associativity. It establishes that robust reproducibility requires bounding this noise below a specific result-flipping threshold.
HEAL Framework: Introduces a two-pronged solution that bypasses hardware penalties by safeguarding Attention via INT16 quantization and synthesizing high-precision GEMM via algebraic error compensation.
MCR-Bench: Introduces the Mission-Critical Reproducibility Benchmark, a new evaluation suite targeting reproducibility in high-stakes domains (medical, legal, financial) where outcome consistency is paramount.
Efficiency-Reproducibility Trade-off: Demonstrates that HEAL achieves FP32-level reproducibility for downstream tasks across varied hardware types while reducing performance overhead by up to 7.1× compared to full FP32 pipelines.

Results

Evaluated across five models (Qwen3-8B/14B/32B, Llama-3.1-8B, DeepSeek-R1-Distill-Qwen-14B) and three hardware types (NVIDIA A100, H100, AMD MI300X):

Reproducibility: HEAL-Full (the complete hybrid pipeline) achieves an answer flip rate and Final Hidden MSE nearly identical to the FP32 baseline (e.g., 0.0% flip rate for Llama-8B and DeepSeek-14B, compared to 11.7% and 12.8% for native BF16).
Performance Overhead:
- FP32 Baseline: Increases TPOT by 4.2× to 13.1×.
- HEAL-Full: Incurs only a 40%–109% TPOT overhead relative to the native BF16 baseline.
- Memory: HEAL maintains the 16-bit memory footprint for the KV cache, avoiding the 2× memory expansion required by full FP32 upcasting.
Hardware Agnosticism: The accumulated numerical error remains consistent across different hardware types under fixed precision configurations, confirming that HEAL effectively neutralizes hardware-specific divergence.

Significance

The paper claims that the assumption that an "all-or-nothing" cost of full high-precision pipelines is required for robust reproducibility is incorrect. By identifying that non-reproducibility is a localized sensitivity to specific outlier features (boundary truncation) rather than a pervasive noise problem, HEAL restores deterministic behavior across heterogeneous hardware. Crucially, the method achieves this stability by upgrading less than 1% of the total compute (via algebraic compensation) and optimizing storage formats, offering a practical path to deploying LLMs in mission-critical environments where exact numerical outputs are required without the economic burden of full FP32 inference.