💻 computer science

BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX

This paper presents a CPU-oriented algorithm that leverages Intel AMX BF16 matrix products to emulate high-precision FP32 and FP64 GEMM operations, achieving competitive throughput and tunable accuracy by decomposing operands into multiple low-precision components and accumulating them in higher precision.

Original authors: Bing Cui, Yu Liu

Published 2026-09-07✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bing Cui, Yu Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern computers are built with a growing split in their internal machinery. On one side, there are powerful engines designed specifically for artificial intelligence, which excel at performing billions of simple calculations very quickly. These engines work best with numbers that are short and simple, sacrificing a tiny bit of detail for massive speed. On the other side, the world of scientific discovery—simulating weather patterns, modeling how atoms bond, or predicting fluid flow—still relies on numbers that are long and precise. These scientific calculations need every bit of detail to remain stable and accurate, but the standard computer parts that handle them are often slower and less efficient than the new AI engines. This creates a dilemma: scientists need the speed of the new hardware, but they cannot afford to lose the precision their work demands.

Researchers at Maginfra Co., Ltd. in China have explored a way to bridge this gap using a specific type of computer chip called Intel AMX. Their goal was to see if the fast, low-precision AI engines could be tricked into performing the slow, high-precision math required for science. Instead of asking the chip to do the hard math directly, they broke the problem down into smaller, simpler pieces. Imagine trying to measure a very long distance with a ruler that only has markings for whole inches. You could measure the whole inches, then measure the remaining fraction, then measure the tiny sliver left over, and add them all together to get a precise total. The researchers applied this same logic to numbers. They took a single complex number and split it into several simpler parts that the fast AI engine could handle easily. They then ran many quick calculations on these parts and carefully added the results back together to reconstruct the final, highly accurate answer.

The team tested this approach on two different levels of precision. First, they tackled single-precision math, which is the standard for many scientific applications. They found that by splitting each number into three parts and running six specific calculations, they could achieve results that were just as accurate as the best existing software, but significantly faster. On the computer chips they tested, this method ran between 1.14 and 2.56 times faster than the standard way of doing the math. The speedup was most noticeable with larger sets of data, where the overhead of splitting and reassembling the numbers became less important compared to the sheer speed of the calculations.

When they moved to double-precision math, which is even more exact and used for the most demanding scientific simulations, the challenge increased. Here, the researchers had to split each number into six parts. Because the calculations needed to be reassembled with extreme care, the process became more complicated. They tested different versions of this method, keeping anywhere from six to twenty-one of the small calculation pieces. They discovered a clear trade-off: keeping more pieces made the answer more accurate, but it also slowed the process down. With just six pieces, the method was fast enough to beat the standard software for very large problems, running up to 1.7 times faster. However, as they added more pieces to improve accuracy, the extra work required to manage them ate up the speed advantage. Eventually, trying to keep twenty-one pieces made the method slower than the standard approach, even though it was more accurate.

The study also highlighted that this technique is not a universal fix for every situation. It works best when the numbers being calculated stay within a specific range, similar to how a ruler with a limited length cannot measure a distance that is too vast or too tiny without special adjustments. The researchers noted that their method does not work for every possible type of number, particularly those that are extremely large or extremely small, and it does not guarantee a perfect, bit-for-bit match with existing software. Instead, it offers a new tool for scientists who need high speed and high accuracy, provided their data fits within the method's limits. By showing that low-precision hardware can be used to solve high-precision problems, the work suggests a future where the specialized engines built for artificial intelligence can also accelerate the heavy lifting of scientific discovery.

Technical Summary: BF16 Component-Product Emulation of FP32 and FP64 GEMM on Intel AMX

1. Problem Statement

Modern CPUs increasingly integrate high-throughput matrix engines (such as Intel Advanced Matrix Extensions, or AMX) optimized for low-precision AI workloads (e.g., BF16, FP8). However, many scientific computing applications in fields like computational fluid dynamics and quantum chemistry still rely on FP32 and FP64 General Matrix Multiplication (GEMM) for numerical stability and reproducibility. This creates a hardware mismatch: the high-performance low-precision engines are not natively usable for high-precision scientific tasks.

The central question addressed by this paper is whether the low-precision matrix throughput designed for AI can be algorithmically repurposed to accelerate high-precision scientific GEMM on CPUs. The challenge lies in emulating FP32 and FP64 arithmetic using BF16 components without incurring prohibitive overheads from decomposition, data movement, and reconstruction, while maintaining numerical accuracy comparable to standard libraries like Intel oneMKL.

2. Methodology

The paper proposes a component-product emulation framework that decomposes high-precision operands into multiple BF16 matrices, computes selected low-precision products using AMX, and reconstructs the result in the target format.

2.1 Floating-Point Formats and Domains

BF16: Retains the 8-bit exponent range of FP32 but reduces significand precision to 8 bits.
FP32 Emulation: Operates on standard FP32 inputs.
FP64 Emulation: Operates on a restricted domain ( $FB_{64}$ ) where FP64 values lie within the normal BF16 exponent range. This restriction avoids the need for per-panel scaling factors, preserving performance headroom.

2.2 Decomposition Strategies

AMX-FP32 Path: Uses a three-component BF16 residual decomposition. Each FP32 operand is split into three BF16 slices ( $A_0, A_1, A_2$ ) using direct residual rounding. This follows the "Henry-style" approach, which exploits the shared exponent range to ensure exact subtraction of residuals in FP32.
AMX-FP64 Path: Uses a simplified, fixed six-slice Ozaki decomposition. Each FP64 operand is split into six BF16 slices. Unlike full Ozaki schemes that use dynamic scaling, this method stores extracted chunks directly as BF16 matrices (assuming they remain normal), eliminating the need for explicit scale metadata and reconstruction weights.

2.3 Product Scheduling and Reconstruction

FP32 Reconstruction: The method evaluates six selected component products (the first three diagonals of the component interaction matrix). Crucially, these products are accumulated directly in FP32 AMX tiles. The implementation employs an operand-reuse schedule (e.g., $A_1B_1 \to A_1B_0 \to \dots$ ) that keeps accumulator tiles resident, minimizing tile load/store traffic.
FP64 Reconstruction: The method evaluates a variable number of products (6, 10, 15, or 21). Unlike the FP32 path, each selected BF16 product is computed in FP32, stored to memory, widened to FP64, and accumulated in an FP64 block accumulator. This prevents FP32 accumulation errors from contaminating the high-precision result but incurs significant overhead due to repeated stores and format conversions.

2.4 Implementation Details

The implementation uses a blocked GEMM structure with $256 \times 256$ output blocks and $32 \times 32$ micro-tiles. It relies on precomputed, block-contiguous component buffers to optimize memory access patterns. The A operands are stored in standard row-major layout, while B operands are pre-packed into the VNNI layout required by AMX BF16 dot-product instructions.

3. Key Contributions

Algorithmic Bridge: Demonstrates a practical method to emulate FP32 and FP64 GEMM using Intel AMX BF16 instructions, bridging the gap between AI-optimized hardware and scientific computing requirements.
FP32 Emulation with FP32-Level Accuracy: Shows that a six-product BF16 schedule with FP32 accumulation can achieve accuracy comparable to oneMKL SGEMM (measured by normwise relative error and GEMM-scaled componentwise error), without claiming bitwise identity.
Controlled FP64 Emulation: Introduces a tunable accuracy-performance tradeoff for FP64 emulation by varying the number of retained component products (6 to 21) within a fixed six-slice decomposition.
Optimized Data Movement: Develops specific scheduling strategies (operand reuse for FP32) and data layouts (VNNI packing, precomputed buffers) to mitigate the overheads of decomposition and reconstruction on CPU matrix engines.

4. Experimental Results

Experiments were conducted on a dual-socket Intel Xeon Platinum 8462Y+ server, comparing the proposed kernels against Intel oneMKL SGEMM and DGEMM.

FP32 Performance: The AMX-FP32 implementation outperforms oneMKL SGEMM across all tested square matrix sizes ( $N=256$ to $N=32768$ ). Speedups range from 1.14 $\times$ to 2.56 $\times$ , with the highest gains observed at larger matrix sizes where the BF16 throughput advantage amortizes the decomposition overhead.
FP64 Performance: The results are more constrained.
- AMX-FP64-6 (6 products) achieves speedups up to 1.72 $\times$ over oneMKL DGEMM for large matrices ( $N \ge 16384$ ).
- AMX-FP64-10 provides marginal gains only for the largest matrices.
- AMX-FP64-15 and AMX-FP64-21 do not outperform DGEMM in the tested configurations, as the costs of storing, converting, and accumulating additional products consume the available performance headroom.
Accuracy:
- AMX-FP32 achieves errors in the $10^{-7}$ range, consistent with FP32-level accuracy.
- AMX-FP64 variants show decreasing error as the product count increases, with the 21-product variant approaching DGEMM-level accuracy but at a performance cost that negates the speedup.

5. Significance and Claims

The paper positions this work as an early algorithmic study rather than a complete replacement for native high-precision libraries. Its significance lies in demonstrating that:

Low-precision CPU matrix engines can be effectively utilized for high-precision scientific computing through careful algorithmic decomposition and data movement optimization.
There is a distinct performance ceiling for emulation: while FP32 emulation is highly effective due to efficient FP32 accumulation, FP64 emulation is limited by the overhead of widening and accumulating results outside the AMX tile registers.
The method offers a tunable tradeoff for FP64, allowing applications to select a product count that balances their specific accuracy requirements against performance needs.

The authors explicitly note limitations: the current prototype supports only square matrices, requires inputs within the BF16 exponent range (for FP64), and does not handle overflow/underflow or exceptional values. Future work is identified as extending the framework to rectangular matrices, broader input domains via scaled decomposition, and adaptation to other CPU matrix engines (e.g., Arm SME).