🔢 mathematics

Analysis of Floating-Point Matrix Multiplication Computed via Integer Arithmetic

This paper proposes an inexpensive method to estimate the minimum number of integer slices required for accurate floating-point matrix multiplication via integer arithmetic, while analyzing how row and column scaling impacts the algorithm's efficiency and accuracy on modern mixed-precision hardware.

Original authors: Ahmad Abdelfattah, Jack Dongarra, Massimiliano Fasi, Mantas Mikaitis, Françoise Tisseur

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmad Abdelfattah, Jack Dongarra, Massimiliano Fasi, Mantas Mikaitis, Françoise Tisseur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Fast but Blurry" Camera Problem

Imagine you are trying to take a high-resolution photograph of a complex landscape (a scientific calculation). You have two types of cameras:

The Master Lens (Binary64): This camera takes incredibly sharp, perfect pictures. It captures every tiny detail, from the largest mountain to the smallest pebble. However, it is slow. It takes a long time to process each photo.
The Speed Lens (Integer/INT8): This camera is a racing car. It can snap thousands of photos per second. But, it only sees in "blocky" pixels. If you try to take a picture of something very small or very large, the image gets blurry or distorted.

The Problem: Modern supercomputers (like the ones on the Top 500 list) are built with these "Speed Lenses" because they are essential for Artificial Intelligence. But scientists need the "Master Lens" for weather forecasting, physics simulations, and engineering. If we only use the Speed Lens, our scientific results might be wrong.

The Goal: The authors of this paper want to trick the "Speed Lens" into taking a "Master Lens" quality photo, without actually using the slow Master Lens.

The Solution: The "Ozaki Scheme" (The Lego Analogy)

The paper discusses a method called the Ozaki Scheme. Here is how it works, using a Lego analogy:

Imagine you have two giant, complex Lego structures (Matrix A and Matrix B) that you need to combine to build a new, even bigger structure (Matrix C).

The Problem: The "Speed Lens" (the computer's fast integer math unit) can only handle small, simple Lego bricks. It can't handle the giant, complex structures directly.
The Strategy (Slicing): Instead of trying to combine the whole structures at once, you break them down into smaller, manageable slices (layers of Lego bricks).
- You take the top layer of Structure A and the top layer of Structure B.
- You combine them using the fast "Speed Lens."
- Then you take the next layer, combine them, and so on.
The Accumulation: Finally, you stack all these combined layers back together to rebuild the final result.

The Catch: If you only use a few slices (a few layers), the final result is a bit "blocky" or inaccurate. If you use many slices, the result is sharp and accurate, but it takes much longer because you have to do more steps.

The New Discovery: "Badly Scaled" Matrices

The authors discovered a hidden trap in this Lego strategy.

Imagine one of your Lego structures is a skyscraper (very tall) and the other is a mud pie (very flat).

When you try to slice them up, the skyscraper needs tiny, precise slices to capture the top floors.
The mud pie is easy to slice.

If the "skyscraper" (a matrix with numbers that vary wildly in size) is not handled carefully, the "Speed Lens" gets confused. It tries to fit the tiny top floors of the skyscraper into the same bucket as the giant base, and the tiny details get lost in the noise.

The Paper's Finding:

The Trap: If your data is "badly scaled" (some numbers are huge, some are tiny), the standard method fails. You might need hundreds of slices to get it right, which defeats the purpose of using the fast computer.
The Fix: The authors found a way to estimate how many slices you actually need before you start.
- If Matrix A is a "skyscraper" and Matrix B is a "mud pie," you might need 20 slices for A but only 2 for B.
- By using different numbers of slices for each matrix, you can save time without losing accuracy.

The Real-World Test: The Race Track

The authors tested this on the newest, fastest supercomputer chips (NVIDIA's Blackwell and Hopper GPUs).

The Result: When the data was "nice and normal" (like random Lego blocks), their method was 7 times faster than the standard slow method, while still being accurate enough for most science.
The Warning: When they tested it on "badly scaled" data (the skyscraper vs. mud pie), the method struggled. It either became inaccurate or required so many slices that it became slower than the original method.

The "So What?" for You

Speed vs. Accuracy: We are entering an era where computers are getting faster at "rough" math (for AI) but slower at "precise" math (for science).
The Hybrid Approach: This paper gives us a recipe to use the fast "rough" math to do "precise" science work.
The Caveat: You can't just use this recipe for everything. If your data is messy (badly scaled), you have to be very careful. The authors provide a "calculator" to tell you exactly how much effort (slices) you need to put in to get a good result.

Summary Metaphor

Think of the Ozaki Scheme as making a smoothie.

Old Way: You blend the whole fruit (slow, but perfect).
New Way (Ozaki): You chop the fruit into tiny pieces (slices), blend the pieces quickly, and then mix them all together.
The Discovery: If you have a huge watermelon and a tiny blueberry, and you chop them into the same size pieces, the blueberry gets lost. The authors figured out how to chop the watermelon into big chunks and the blueberry into tiny crumbs, so you can blend them all quickly without losing the blueberry flavor.

Conclusion: This paper is a vital guide for scientists who want to use the world's fastest supercomputers for high-precision work, telling them exactly how to slice their data to get the best speed without breaking the math.

1. Problem Statement

Modern exascale supercomputers increasingly rely on hardware accelerators (e.g., NVIDIA Tensor Cores, AMD Matrix Cores) optimized for low-precision arithmetic (INT8, FP16, BF16) and machine learning workloads. While these units offer throughput orders of magnitude higher than standard binary64 (double precision), they lack the precision required for traditional scientific simulations.

The core challenge is how to perform high-precision (binary64) matrix multiplication ($AB$) efficiently on hardware that primarily supports low-precision integer or floating-point operations. Existing approaches, such as the Ozaki scheme, attempt to solve this by decomposing floating-point matrices into integer slices, computing exact integer products, and accumulating the results in floating-point arithmetic. However, previous analyses lacked precise error bounds regarding matrix scaling, and it was unclear how to optimize the trade-off between performance (number of slices) and accuracy, particularly for badly scaled matrices.

2. Methodology

The paper analyzes and extends the Integer Ozaki Scheme, originally proposed by Ootomo, Ozaki, and Yokota, and improved by Uchino, Ozaki, and Imamura.

The Algorithm

Block Fixed-Point Conversion: Rows of matrix $A$ and columns of matrix $B$ are implicitly converted to a block fixed-point representation. Each row/column shares a common scaling factor ( $\alpha_i$ or $\beta_j$ ), chosen as the smallest power of 2 greater than the maximum absolute value in that block.
Slicing: The fixed-point representations are split into $s$ slices (integer matrices) based on bit positions. If $t$ is the number of bits per slice, the $k$ -th slice contains bits from position $(k-1)t+1$ to $kt$.
Integer Multiplication: The slices $A^{(\ell)}$ and $B^{(h)}$ are multiplied using hardware integer Matrix Multiply-Accumulate (MMA) units. These products are exact (assuming no overflow in the accumulator).
Accumulation: The partial products are scaled by $2^{-(\ell+h)t}$ $2^{- (ℓ + h) t}$ and accumulated in floating-point arithmetic to produce the final result.
- Optimization: To reduce cost, products where $\ell + h > s+1$ are often skipped (truncation).
- Enhancement: Uchino et al. proposed accumulating along diagonals (constant $\ell+h$ ) using integer arithmetic before converting to floating-point to reduce rounding errors.

Theoretical Framework

The authors develop a rigorous error analysis separating two error sources:

Truncation Error: Caused by discarding lower-order bits during slicing. This depends on the number of slices ( $s$ ) and the conditioning of the rows/columns (specifically, the ratio of max to min absolute values within a row/column, denoted as $\kappa_A$ and $\kappa_B$ ).
Rounding Error: Caused by floating-point accumulation of the integer products.

The analysis introduces a novel approach allowing different numbers of slices ( $s_A$ and $s_B$ ) for matrices $A$ and $B$ , rather than forcing a uniform $s$ .

3. Key Contributions

A. New Error Analysis

The paper provides a refined error bound showing that the algorithm's accuracy is heavily dependent on the scaling of the input matrices.

Bad Scaling Failure: If rows of $A$ or columns of $B$ are "badly scaled" (i.e., contain elements with vastly different magnitudes), the relative truncation error can become large, regardless of the number of slices, unless $s$ is increased significantly.
Asymmetric Slicing: The analysis proves that if one matrix is well-scaled and the other is not, one can reduce the number of slices for the well-scaled matrix without degrading the final accuracy. This offers a strategy to minimize computational cost ( $s_A \times s_B$ ) while maintaining precision.
Accumulation Strategy: The authors derive conditions under which partial sums can be accumulated exactly in floating-point arithmetic by aligning diagonals of the product matrix, minimizing rounding errors.

B. Estimation Strategy

They propose an inexpensive method to estimate the minimum number of slices required for a prescribed accuracy. By computing the scaling factors ( $\kappa_A, \kappa_B$ ) during the initial scan of the matrix, one can determine the necessary $s_A$ and $s_B$ to meet error bounds without trial-and-error.

C. Comprehensive Numerical Experiments

The authors validate their theory through extensive simulations and benchmarks on:

Hardware: NVIDIA Grace-Hopper (H100) and the newly released NVIDIA Blackwell (B200) GPUs.
Libraries: Comparison against cublas (native binary64) and cuIMMA (NVIDIA's prototype integer emulation).
Scenarios:
- Random matrices.
- Badly scaled matrices (constructed to test the theoretical limits).
- High-level algorithms (Block LU factorization, Symmetric Eigensolver).

4. Results

Accuracy vs. Performance Trade-off

Random Matrices: On the B200 GPU, using 7 slices with cuIMMA achieves accuracy indistinguishable from native cublas (binary64) while providing a 7.6 $\times$ speedup. On the H100 (GH200), the speedup is lower (2.4 $\times$ ) because the ratio between INT8 and binary64 peak performance is smaller.
Badly Scaled Matrices: The algorithm fails to achieve binary64 accuracy for badly scaled matrices if the number of slices is fixed at a low value (e.g., $s=8$ ). Even increasing slices to 18 was insufficient for extreme scaling ( $\kappa_D = 10^{40}$ ). This confirms the theoretical prediction that bad scaling requires a prohibitive number of slices, potentially making the method inefficient.
Asymmetric Slicing: In Block LU factorization tests, matrices with specific scaling properties (e.g., $s^*_A=9, s^*_B=1$ ) achieved full accuracy with asymmetric slicing ( $s_A=8, s_B=1$ ), avoiding the performance penalty of using 8 slices for both.

Impact on Higher-Level Algorithms

Integrating the emulated DGEMM into LAPACK routines (QR, Eigensolver) revealed that the optimal number of slices depends on the specific algorithm.
For the Symmetric Eigensolver, using 7 slices (sufficient for raw GEMM) resulted in a loss of orthogonality. 8 slices were required to match the stability of native binary64. This suggests that "one-size-fits-all" slice counts are insufficient for scientific computing.

5. Significance and Conclusion

Viability for Scientific Computing: The Integer Ozaki scheme is a viable path to utilizing next-generation AI hardware for scientific computing, but it is not a drop-in replacement for binary64 arithmetic.
Scaling Sensitivity: The method is highly sensitive to input scaling. Without a pre-check or dynamic adjustment of slices, it risks producing inaccurate results for ill-conditioned or badly scaled problems.
Hardware Trends: The results highlight a shift in hardware design: vendors are prioritizing low-precision performance (INT8) over binary64. The B200's massive INT8-to-binary64 performance ratio (112.5 $\times$ ) makes the emulation approach highly attractive, whereas older architectures (H100) offer less benefit.
Future Work: The authors note that current implementations do not handle special IEEE 754 values (NaNs, infinities, negative zeros) correctly, which remains a barrier to full adoption in general-purpose linear algebra libraries.

In summary, the paper provides the theoretical foundation and practical guidelines to safely deploy integer-based matrix multiplication on modern GPUs, emphasizing that adaptive slicing based on matrix properties is essential for balancing the massive performance gains of integer hardware with the strict accuracy requirements of scientific simulation.