💻 computer science

85.30 GFLOPS Single-Core FP32 Matrix Multiplication on AMD Zen 3: A Systematic Study of Cache Blocking, Register Blocking, FMA Chaining, and On-the-Fly Packing

This paper presents a systematic optimization study on the AMD Zen 3 microarchitecture that achieves 85.30 GFLOPS in single-core FP32 matrix multiplication by evaluating 28 distinct configurations of cache/register blocking, FMA chaining, and on-the-fly packing, ultimately identifying a champion design that reaches 63.5% of the theoretical peak performance while introducing a predictive model for future optimizations.

Original authors: Lucas Lima Freitag

Published 2026-07-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lucas Lima Freitag

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to move a massive pile of sand (data) from one side of a giant warehouse to another, but you have to do it using a tiny, super-fast robot arm (the processor). The goal is to mix the sand with a special formula (matrix multiplication) as quickly as possible. This paper is a detailed logbook of one researcher, Lucas, trying to make that robot arm move as fast as humanly possible on a specific type of computer chip called the AMD Zen 3.

The Big Goal: How Fast is Fast?

The robot arm has a theoretical top speed of 134.4 GFLOPS (that's 134.4 billion math operations per second). Think of this as the speed limit on a highway. Lucas wanted to see how close he could get to that speed limit without building a brand-new car, just by tuning the engine.

After testing 28 different driving strategies, he found a "champion" setup (called MX24) that hit 85.30 GFLOPS. That's about 63.5% of the maximum speed. It's not a perfect 100%, but it's a huge leap from the starting line, which was a slow, clumsy method running at only 1.51 GFLOPS. In fact, his best method was 57 times faster than the basic, unoptimized version.

The Winning Strategy: The "4-Row, Chain-4" Dance

To get this speed, Lucas had to figure out how to organize the sand and the robot's movements. Here are the key moves he discovered:

1. The "On-the-Fly" Packing Trick
Imagine the sand is stored in a grid where you have to walk diagonally to grab the next grain. That's slow and tiring. Lucas found that copying a small chunk of the sand into a neat, straight line just before the robot needs it (called on-the-fly packing) was the magic move. It's like having a helper run ahead and stack the bricks in a perfect row so the robot can just grab them one after another without stumbling. This beat the old method of just grabbing them as they were, and it was even better than rearranging the entire warehouse beforehand.

2. The "4-Row" Stacking
The robot has a limited number of hands (registers) to hold sand while it works. Lucas tried holding 2 rows of sand, then 4, then 8.

2 Rows: Too few. The robot had to stop and fetch new sand too often.
8 Rows: Too many! The robot's hands were so full it had to drop sand on the floor (memory) and pick it up again constantly. This was a disaster.
4 Rows: The sweet spot. Even though the robot had to drop a few grains on the floor and pick them back up (a process called "spilling"), the extra work was worth it because it could process more sand at once. This single change made the robot 59% faster than the 2-row method.

3. The "Chain-4" Rhythm
The robot's arm takes 4 seconds (cycles) to finish a single math move before it can start the next one on the same piece of sand. If the robot just did one move and waited, it would be idle for 3 seconds.
Lucas found that if the robot held 4 different piles of sand in its hands and worked on them in a loop (Chain-4), it could keep the arm moving constantly. While one pile was "cooking," it worked on the others. This perfectly matched the robot's 4-second cooking time, keeping the engine humming at full speed.

What Didn't Work (The "Don't Do It" List)

Sometimes, what you think should work actually makes things worse. Lucas tested some popular ideas and found they were terrible for this specific robot:

The "Pre-fetch" Mistake: People often tell robots to "look ahead" and grab the next sand grain before they need it. Lucas tried this, but the robot's built-in eyes were already so good at seeing the pattern that the extra "look ahead" commands just got in the way. It slowed the robot down by about 8%.
The "Non-Temporal" Dump: There's a trick where you tell the robot to dump sand directly onto the floor without putting it in a bin first. This works great if you are just dumping trash. But here, the robot has to mix the sand, which means it has to pick it up again. Dumping it directly caused the robot to trip over its own feet, crashing its speed down to a pathetic 1.24 GFLOPS.
The "8-Row" Overload: As mentioned, trying to hold 8 rows of sand caused the robot to drop so much sand that it spent more time picking it up than moving it.

How Sure Are We?

The paper is very confident in these numbers because they were measured, not just guessed. Lucas ran the code 15 times for each strategy, threw out the fastest and slowest runs (to avoid weird computer glitches), and averaged the rest. He also checked the math against a simple, slow version to make sure the fast version didn't cheat.

He even built a mathematical "pre-filter" model—a sort of crystal ball—to predict how fast a strategy would be before actually running it. This crystal ball was pretty good, getting within 11.3% of the real speed for most strategies. It tended to be a bit conservative, predicting the best strategy would be 78.1 GFLOPS, but when they actually ran it, it hit 85.30 GFLOPS.

The Takeaway

This paper proves that you don't need to be a wizard writing code in "assembly language" (the robot's native tongue) to get amazing speed. By using standard tools (C++ intrinsics) and carefully tuning the "dance steps" (blocking, chaining, and packing), you can get your computer to run at 63.5% of its theoretical maximum speed.

The main lesson? Don't guess. What works for one type of robot (or computer chip) might break another. Lucas tested 28 different combinations to find the one that worked, showing that sometimes the "obvious" tricks (like looking ahead or dumping directly) are actually the wrong moves.

Title: 85.30 GFLOPS Single-Core FP32 Matrix Multiplication on AMD Zen 3: A Systematic Study of Cache Blocking, Register Blocking, FMA Chaining, and On-the-Fly Packing

Problem Statement
Matrix multiplication (GEMM) is a fundamental operation in scientific computing and machine learning, yet achieving peak performance on modern processors requires navigating a complex landscape of memory hierarchies, register files, and pipeline constraints. While libraries like BLIS and OpenBLAS provide highly optimized kernels, there is a need to systematically understand how specific optimization techniques interact on contemporary microarchitectures, particularly AMD Zen 3. This paper addresses the challenge of maximizing single-core floating-point throughput for the GEMM kernel ( $C = \alpha AB + \beta C$ ) on the AMD Ryzen 5 5500 processor, aiming to approach the theoretical peak performance without relying on hand-coded assembly or external proprietary libraries.

Methodology
The study employs a systematic exploration of a design space comprising 28 distinct optimization models (MX00–MX28) targeting the AMD Zen 3 microarchitecture. The methodology involves:

Target Architecture Analysis: Detailed examination of the Zen 3 core, specifically its 4-cycle FMA latency, dual 256-bit FMA ports, 16 architectural YMM registers, and three-level cache hierarchy (32 KB L1, 512 KB L2, 16 MB L3).
Optimization Dimensions: The authors combinatorially vary key techniques:
- Cache Blocking: Three-level tiling (L1, L2, L3) with varying block sizes ( $B_I, B_J, B_K$ ).
- Register Blocking: Partitioning the accumulator matrix $C$ into 2-row, 4-row, and 8-row schemes to utilize the 16 available YMM registers.
- FMA Chaining: Implementing chain lengths from 1 to 6 to hide the 4-cycle FMA latency.
- Packing Strategies: Comparing no packing, explicit matrix transposition ( $B^T$ ), and on-the-fly $B$ -packing.
- Auxiliary Techniques: Memory alignment, software prefetching, and non-temporal stores.
Experimental Setup: Experiments were conducted on a single core of an AMD Ryzen 5 5500 running at 4.2 GHz. The code was written in C++ using intrinsics and compiled with GCC 12.2 at -O3. Performance was measured for square matrices of dimension $N=2048$ using FP32 data types.
Predictive Modeling: A mathematical pre-filter model was developed to estimate performance a priori based on pipeline penalties, register pressure, and cache miss rates, calibrated against the 28 experimental models.

Key Contributions and Results
The paper presents a champion configuration, MX24, which achieves 85.30 GFLOPS, representing 63.5% of the theoretical single-core peak of 134.4 GFLOPS. This result is approximately 57× faster than a naive triple-loop implementation.

Optimal Configuration: The winning model utilizes a 4-row register block, chain4 FMA chaining (matching the 4-cycle latency), on-the-fly B-packing, and a deep tile size of $B_K = 256$ (fitting within the L2 cache).
Counter-Intuitive Findings:
- Register Spilling: Despite requiring 16 register spills (32 logical registers needed vs. 16 physical), the 4-row scheme outperformed the spill-free 2-row scheme by 59%. The compiler efficiently scheduled spill operations during pipeline bubbles, and the increased reuse of $A$ elements outweighed the memory overhead.
- Software Prefetching: Contrary to common intuition, software prefetching degraded performance by approximately 8%, as the Zen 3 hardware prefetcher was already highly effective for the streaming access patterns of GEMM.
- Non-Temporal Stores: Using non-temporal stores for the $C$ matrix caused a catastrophic performance collapse to 1.24 GFLOPS because GEMM is a read-modify-write operation, and bypassing the cache invalidated data needed for subsequent loads.
- Packing Efficiency: On-the-fly packing outperformed full matrix transposition by avoiding cache pollution during the setup phase, while providing contiguous memory access similar to transposition.
Predictive Model: The pre-filter mathematical model achieved a Mean Absolute Percentage Error (MAPE) of 11.3% across the design space, successfully identifying high-performing configurations with a conservative bias.

Significance and Claims
The authors claim that this work provides a reproducible reference for single-core GEMM optimization on contemporary x86-64 microarchitectures. The primary significance lies in demonstrating that carefully hand-tuned vectorized code using C++ intrinsics can approach within 2× of the roofline ceiling without external library dependencies or hand-written assembly.

The paper emphasizes that systematic, combinatorial optimization yields better results than relying on general heuristics, as evidenced by the failure of techniques like software prefetching and non-temporal stores in this specific context. The 85.30 GFLOPS result is competitive with mature, vendor-optimized libraries (such as AMD AOCL and Intel MKL) on comparable hardware, validating the effectiveness of the described optimization template (4-row, chain4, on-the-fly packing) for x86-64 architectures with AVX2/FMA support. The work also highlights the critical role of compiler quality, noting that GCC 12.2 produced significantly better code than Microsoft Visual C++ for these specific intrinsics.

The Big Goal: How Fast is Fast?

The Winning Strategy: The "4-Row, Chain-4" Dance

What Didn't Work (The "Don't Do It" List)

How Sure Are We?

The Takeaway

More like this