Rigel: Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max GPU

This paper presents Rigel, an empirical study that reverse-engineers Apple's M4 Max Metal 4.1 tensor compute path to reveal that its fp8 matmul2d operation is memory-bound and emulated rather than hardware-accelerated, while also uncovering hidden execution details and enabling a hand-fused kernel that outperforms the standard decomposed path by up to 12.9%.

The Gist

Imagine Apple's new M4 Max computer chip as a massive, high-tech kitchen. For years, developers (the chefs) have been told that this kitchen has a special, super-fast "Tensor Oven" designed specifically to bake complex AI recipes (like Large Language Models) at lightning speed. The official instruction manual (the Metal 4.1 specification) says, "Yes, we have this oven, and it supports these specific ingredients." But the manual is frustratingly vague: it doesn't say how the oven works, where it is located, or if it's actually faster than the standard stove.

The paper RIGEL is like a team of food scientists who decided to stop reading the manual and start tasting the food to figure out what's really happening in the kitchen. They built a set of ultra-precise measuring tools to reverse-engineer the M4 Max's behavior.

Here is what they discovered, translated into everyday analogies:

1. The "Special Oven" is Actually Just a Regular Stove

The Claim: The paper proves that the M4 Max does not have a dedicated "Tensor Core" (a special hardware unit just for AI math).
The Analogy: Think of the "Tensor" operation as a specific type of cake. The manual implies there's a special, high-speed conveyor belt just for these cakes. RIGEL found that, on the M4 Max, there is no conveyor belt. Instead, the kitchen staff (the GPU shader cores) are just making these cakes by hand on the regular stove, using the same tools they use for everything else. They are doing it efficiently, but they aren't using a secret, dedicated machine.

2. The "Magic Ingredient" (FP8) is a Trick, Not a Superpower

The Claim: The paper tests a low-precision data format called FP8 (which uses half the memory of the standard FP16 format). The spec suggests this might be faster because it's smaller. RIGEL found it is not faster; it's actually slightly slower.
The Analogy: Imagine you are carrying water buckets. The FP16 format is a large bucket; the FP8 format is a small bucket. You might think, "If I use small buckets, I can carry more trips in the same time!" But the scientists found that on the M4 Max, the workers have to stop and pour the small water into a big bucket before they can use it. This "pouring" (unpacking) takes time.

• The Result: Using the small buckets (FP8) doesn't make the job go faster. It only saves you from having to carry as many heavy buckets at once (saving memory space). It's a space-saver, not a speed-saver. The paper calls this a "memory-footprint feature, not a performance feature."

3. The "Secret Recipe" Layout

The Claim: The manual says the way data is arranged in the "Tensor" memory is "opaque" (hidden) and "device specific." RIGEL figured out exactly how it's arranged.
The Analogy: Imagine the kitchen staff is told to arrange ingredients in a grid, but the manual just says, "Do it in a secret pattern." RIGEL watched the staff and realized they are arranging the ingredients in a very specific 8x8 square pattern. Once the scientists figured out this secret grid, they could rearrange the ingredients to make the cooking process smoother.

4. The "Version Gate" (The Bouncer)

The Claim: The manual says you can use these features with a certain software version (Xcode 26.1+), but RIGEL found that's a lie.
The Analogy: The manual says, "Anyone with a red hat can enter the VIP room." But when RIGEL tried to enter with a red hat from last year (Xcode 26.5), the bouncer kicked them out. You actually need a brand new red hat (Xcode 27) and a specific ID card (macOS 27.0) to even get the door to open. The manual was simply wrong about who could get in.

5. The "Super-Chef" Optimization

The Claim: Because the scientists now know exactly how the kitchen works (no secret oven, 8x8 grid, FP8 is slow), they wrote a new, custom recipe.
The Analogy: Instead of following the standard, step-by-step instructions (Bake Cake -> Add Icing -> Add Fruit), which involves walking back and forth to the pantry three times, the scientists wrote a "fusion" recipe. They combined the baking, icing, and fruit steps into one smooth motion right at the stove.

• The Result: This custom recipe was 6.5% to 12.9% faster for tasks that fit in the kitchen's immediate workspace (cache-resident). However, for huge tasks that require running back and forth to the warehouse (main memory), the speedup disappears because the walking time dominates the cooking time.

Summary of the "Big Reveal"

The paper concludes that on the Apple M4 Max:

• No Magic Hardware: There is no special AI engine; it's all running on the standard graphics cores.
• FP8 is for Storage, Not Speed: Using smaller numbers saves space but doesn't make calculations faster.
• The Manual is Incomplete: The official documentation hides the hardware reality, the exact memory layout, and the true software requirements.
• Custom Code Wins: If you know the secret layout, you can write a custom program that beats the standard Apple tools by a small but measurable margin.

The authors emphasize that these findings are based on hard data from a single M4 Max chip running a beta version of the operating system, and they have released all their code so anyone can verify the results themselves.

Technical Summary

Technical Summary: RIGEL – Reverse-Engineering the Metal 4.1 Tensor Compute Path on the Apple M4 Max

Problem Statement

Apple's Metal 4.1 specification (shipped with Xcode 27/macOS 27.0) introduces a tensor compute path via the matmul2d operation in Metal Performance Primitives (MPP), supporting low-precision formats like fp8 (E4M3), fp4, and block-scaled MXFP4. However, the specification documents the software interface while deliberately obscuring hardware behavior. It declares data types as "supported" without clarifying if they are hardware-accelerated or emulated, leaves the cooperative tensor fragment layout "opaque," and provides no details on dispatch targets, accumulator widths, or microarchitectural constraints. Consequently, developers cannot determine if low-precision operations offer performance gains or merely memory footprint benefits, nor can they optimize kernels without knowing the underlying execution model.

Methodology: The RIGEL Harness

The authors present RIGEL, an empirical characterization framework designed to recover hidden hardware facts through microbenchmarking. The methodology relies on a "discovery bar": a finding is only considered a contribution if it is either hidden by the specification or contradicts it.

Key components of the RIGEL harness include:

• Checksum-Gated Verification: To distinguish between input quantization error and hardware accumulation error, the harness computes a float64 reference using the exact same quantized bytes consumed by the GPU. Results are discarded if they exceed per-data-type error tolerances.
• Provenance Tracking: Every measurement cell is tied to a specific code commit, input generation recipe, random seed, and environment hash (OS build, toolchain, device). All figures are regenerated from committed CSVs via a make reproduce target.
• Baseline Comparison: Performance claims are validated against four baselines: scalar-ALU roofline, naive threadgroup-tiled GEMM, legacy simdgroup matrix instructions, and stock fp16 MPP paths. A kernel must beat the strongest baseline by at least 1.10× to be considered a win; low-precision formats must reach 1.5× fp16 throughput to be considered "accelerated" rather than "emulated."
• Triangulation: The dispatch target is determined via three independent signals: throughput ceilings, comparison against simdgroup matrix, and per-rail power attribution.

The study was conducted on a single Apple M4 Max (40 GPU cores, 64 GB unified memory) running macOS 27.0 beta. The M4 generation is noted as pre-dating the "Neural Accelerators" introduced in the A19/M5 family.

Key Findings and Results

1. Execution Target: No Dedicated Matrix Unit

Contrary to speculation that Apple may have introduced a dedicated tensor core analogous to NVIDIA's, RIGEL confirms that matmul2d executes entirely on the GPU shader cores via the legacy simdgroup matrix path.

• Throughput Ceiling: Tiled fp16 matmul2d peaks at 14.8 TFLOP/s, which is approximately 46% of the scalar-ALU fp16 roofline (32 TFLOP/s). No hidden datapath exceeds the ALU ceiling.
• Performance Proximity: MPP matmul2d outperforms a naive tiled GEMM by 2.9–5.5× but only beats the legacy simdgroup matrix by 1.05–1.21×, indicating it lowers onto the same pipeline.
• Power Attribution: Under sustained load, GPU power draw rises significantly while hardware-active residency hits 100%. While direct ANE (Apple Neural Engine) power measurement was blocked by the beta OS, the lack of throughput beyond the GPU-only path suggests no ANE offloading.

2. The "Headline": fp8 is Emulated, Not Accelerated

The most significant finding is that the Metal 4.1 fp8 (E4M3) matmul2d is emulated, not hardware-accelerated.

• Throughput: fp8 sustains only 0.94× the throughput of fp16 (peaking at 0.941× at 2048³).
• Implication: Since fp8 operands are half the size of fp16, a memory-bound kernel should be faster. The fact that it is slower indicates the operation is compute-bound and incurs pure unpack overhead. On the M4 Max, fp8 is a memory-footprint and dynamic-range feature, not a performance feature. True acceleration is predicted to arrive with the M5 generation.

3. Numeric Semantics and Hidden Constraints

• Accumulator Width: The internal accumulator is at least fp32. A hostile test case (summing one large term and 127 small terms) showed that fp8 matmul2d produced the exact result (455.9375), whereas an fp16 accumulator would have lost precision (stalling at 448).
• Fragment Layout: The opaque 8×8 cooperative tensor fragment layout was reconstructed. It is a unified 8×8 base fragment where each SIMD lane owns two vertically adjacent rows of one column within its quadrant. This layout is consistent across inputs (A, B) and the output.
• Alignment Constraints: Host-bound tensor inlines require the inner stride to be 128-byte aligned. This constraint is enforced via a SFINAE note in headers but is absent from the specification, causing naive ports to fail with misleading compiler errors.
• Saturation: E4M3 formats clamp to ±448 on overflow (no infinity encoding), whereas E5M2 yields ±∞.

4. Optimization Opportunities

Using the recovered fragment layout, the authors implemented a hand-fused GEMM + bias + GELU kernel.

• Performance Gain: In the cache-resident regime, this fused kernel outperforms the decomposed path by +6.5% to +12.9%.
• Mechanism: The win comes from avoiding O(n²) memory traffic by fusing the epilogue in-register. The benefit decays as O(n³) compute dominates.
• Attention Limitation: Attempts to fuse FlashAttention (replacing matrix units with scalar code) resulted in kernels 3.6–5× slower than the decomposed path, as the M4's high memory bandwidth makes the S×S round-trip cheap, and the fusion overheads outweighed the benefits.

Contradictions and Version Gates

The paper identifies several discrepancies between the specification and reality:

• Version Gate: Metal 4.1 requires Xcode 27 and macOS 27.0. Binaries compiled with Xcode 27 (Metal 4.1) refuse to load on macOS 26.5, contradicting the spec's claim of availability from "Xcode 26.1+."
• Feature Absence: The 4.1 low-precision formats (fp8, fp4) and block-scaled tensor blocks (MXFP4) are absent under the 4.0 toolchain, despite the spec's feature tables suggesting otherwise.

Significance and Conclusion

RIGEL provides the first empirical characterization of the Metal 4.1 tensor path on Apple Silicon. Its primary significance lies in demystifying the hardware reality behind a deliberately opaque interface.

• It clarifies that the M4 Max lacks a dedicated matrix unit, placing fp8 performance expectations in the correct context (footprint vs. speed).
• It recovers the "device-specific" 8×8 fragment layout, enabling layout-aware optimizations.
• It establishes a reproducible, checksum-verified methodology for reverse-engineering high-level primitives.

The paper concludes that while the vendor documents the interface, the hardware behavior must be discovered empirically. The findings are fully reproducible via MIT-licensed code, CSVs, and regeneration scripts, offering a concrete foundation for developers deploying quantized models on Apple Silicon before the arrival of dedicated tensor accelerators in future generations.