🤖 AI

When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

This paper demonstrates that on Apple Silicon, a specialized fused Metal kernel for int4 KV cache quantization not only preserves model quality but also outperforms fp16 in latency by leveraging unified memory bandwidth and advanced rotation techniques to eliminate per-token degradation.

Original authors: Mohamed Amine Bergach

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed Amine Bergach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Breaking the "Speed vs. Quality" Rule

Usually, when you want to make a computer program run faster, you have to sacrifice some quality. It's like driving a car: if you want to go super fast, you might have to take a shortcut that isn't as smooth, or you might have to remove the air conditioning to save weight.

In the world of AI (specifically Large Language Models), there is a "memory traffic jam." As the AI reads a long story or conversation, it has to remember everything it just read. This memory (called the KV Cache) gets huge.

The Old Way: Keep the memory in high-quality, heavy format (like a full HD video file). It's accurate, but it takes up a lot of space and moves slowly over the computer's "highway" (memory bandwidth).
The Standard Fix: Compress the memory (like turning the video into a smaller MP4). This saves space, but usually, the computer has to work harder to unpack it, making the whole process slower.

This paper claims that on Apple Silicon (M1/M2/M3 chips), this rule is broken. They found a way to compress the memory so effectively that the AI actually runs faster than the uncompressed version, without losing any quality.

The Analogy: The Library and the Librarian

Imagine the AI is a Librarian trying to answer questions based on a massive library of books (the context).

The Problem (The Heavy Books):
The librarian has to keep the most recent pages of the books open on the desk to reference them. If the books are heavy, hardcover encyclopedias (the standard fp16 format), the librarian spends most of their time just carrying them back and forth from the shelves to the desk. The desk is small, so they can only keep a few books open at once.
The Standard Solution (The Photocopy):
Usually, to save space, you photocopy the pages onto thin paper (compression). But the librarian has to stop, unfold the photocopies, read them, and then fold them back up every time they need to look something up. This "folding/unfolding" takes so much time that the librarian is actually slower than if they just carried the heavy books.
The Apple Silicon Solution (The Magic Folder):
The authors built a special Magic Folder (the Fused Metal Kernel).
- The Trick: They don't just shrink the paper; they rearrange the words on the page using a special mathematical shuffle (called SRFT, or Sign-Randomized FFT) so the text looks like random noise. This makes the text easy to compress into tiny, 4-bit "nibbles" (like turning a paragraph into a single line of code).
- The Speed: Because the Apple computer's memory is "unified" (the librarian and the shelves are right next to each other), moving these tiny, compressed pages is incredibly fast. The time it takes to "shuffle and compress" the text is so short that it's actually faster than moving the heavy, uncompressed books.
- The Result: The librarian can now keep 3 times more books open on the desk, and they are still reading them faster than before.

Key Findings in Plain English

It's Not a Trade-Off: On Apple chips, you get the best of both worlds: 3x more memory capacity AND faster speed. The paper calls this "Quantization is Free."
The "Magic Shuffle" (SRFT): They use a mathematical trick called a "Sign-Randomized Fourier Transform." Think of it like shuffling a deck of cards so perfectly that the high-value cards (outliers) are spread out evenly. This prevents the "photocopy" from getting blurry. They found this works just as well as other shuffling methods (Hadamard) but is better suited for Apple's hardware.
Fixing the "Catastrophe": On one specific model (Qwen), simply compressing the text caused the AI to make terrible mistakes (like a librarian reading gibberish). The authors fixed this by adding a tiny "volume knob" (per-channel scaling) for each specific word before compressing it. This saved the quality without slowing things down.
Learning vs. Random: They tested if they could "teach" the AI the perfect shuffle. Surprisingly, a random shuffle worked better than a "learned" one for the final result, even though the learned one looked more accurate on a math test. It turns out the random shuffle acts like a helpful "regularizer" that keeps the AI stable.

The Bottom Line

The paper demonstrates that on Apple computers, you can shrink the AI's memory footprint by 3 times (saving massive space) and, because of how the computer's memory works, the AI actually runs 3% to 8% faster than before.

It's like finding a way to pack a suitcase so tightly that it becomes lighter, yet you can still grab your clothes out of it faster than if the suitcase was half-empty and bulky.

Who is this for?
This is specifically for running AI models on Apple Silicon devices (laptops, phones, tablets). The authors note that on other computers (like standard NVIDIA GPUs), this speed boost might not happen because their memory architecture is different.

What does it enable?
It allows these devices to handle much longer conversations or read much longer documents without running out of memory or getting slow, all while keeping the AI's answers just as smart as before.

Technical Summary: When Quantization Is Free

Problem Statement

Long-context transformer inference is typically constrained by the memory footprint and bandwidth requirements of the Key-Value (KV) cache. As context lengths increase (e.g., 128K tokens), the accumulated KV tensors in fp16 can exceed the size of the model weights themselves, creating a bottleneck where memory bandwidth, rather than compute, limits decode speed.

Standard industry practice addresses this via "rotate-then-quantize" strategies (e.g., QuaRot, SpinQuant), which apply random orthogonal rotations to Gaussianize activations before quantizing them to low bits (int4/int8). However, this approach is traditionally framed as a quality–latency trade-off: the computational overhead of rotation and de-rotation, combined with quantization errors, typically results in a 5–20% increase in decode latency, even if memory is saved.

The authors posit that this trade-off is inverted on Apple Silicon (unified memory architecture). They hypothesize that the bandwidth savings from a 3× compressed cache can outweigh the computational cost of the quantization kernel, provided the kernel is highly optimized and fused.

Methodology

1. Sign-Randomized Fourier Transform (SRFT)

The paper adopts SRFT (Sign-Randomized FFT) as the rotation mechanism, defined as $SRFT(x) = \text{pack}(F \cdot \text{diag}(s) \cdot x)$ , where $F$ is the unitary DFT and $s$ is a fixed random sign vector.

Rationale: While Hadamard transforms (SRHT) are common, SRFT offers flexibility for non-power-of-two dimensions and aligns with 16×16 matrix-multiply operations (AMX, Tensor Cores).
Finding: The authors demonstrate that SRFT and SRHT are statistically indistinguishable in terms of perplexity (PPL) quality across five models (SmolLM2, Qwen2.5, Gemma-3) and various bit budgets.

2. Fused Metal Kernel

To eliminate dispatch overhead, the authors implement a single fused Metal compute kernel that performs the entire pipeline:

Load & Sign-Multiply: Apply the random sign diagonal in-register.
FFT: Execute log2(d/2) radix-2 Stockham butterfly passes.
Packing: Unpack Hermitian symmetry to a real representation.
Scaling & Quantization: Perform per-channel $\lambda$ rescaling (learned or calibrated), per-group absolute-max reduction, and round to signed int4/int8.
Nibble Packing: Pack two int4 values into a single byte using SIMD shuffles.

This kernel is exposed via torch.mps.compile and integrated as a HuggingFace Cache subclass (SRFTInt4Cache).

3. Advanced Scaling Strategies

Per-Channel + Per-Group: For models with small head dimensions (e.g., Qwen2.5-1.5B, $d=128$ ), per-token scaling fails catastrophically due to outlier coordinates. The authors combine per-channel scaling (one scale per coordinate) with per-group scaling (groups of 16) to recover quality.
Learned Rotations: The paper investigates post-training calibration where a learnable rotation $R$ and scale $\lambda$ are added. A key finding is that learning $R+\lambda$ without the SRFT base minimizes calibration MSE significantly more (84.9% vs 50.3%) but yields worse downstream PPL, suggesting the fixed random SRFT acts as a regularizer.
Householder Rotations: At $d=256$ , Householder rotations with $k=d/2$ reflectors achieve effectively lossless quantization ( $\Delta PPL \approx 0.000$ ) with half the parameters of full Cayley rotations.

4. CPU Pathway (AMX)

A complementary CPU pathway utilizes Apple's AMX matrix coprocessor via cblas sgemm. By batching length-8 DFTs as 16×16 real matrix multiplies, the authors achieve up to 91% of the single-core AMX peak, offering a low-latency alternative for small-batch regimes where GPU dispatch overhead is dominant.

Key Results

Performance (Latency)

On Apple M1 (16GB unified memory), the fused int4 KV cache outperforms fp16 in end-to-end decode latency:

Gemma-3 1B: Runs 3–8% faster (negative latency overhead) across 256–4096 token prefixes.
Qwen2.5-1.5B: Runs 0.7–2.6% faster at short contexts (≤1K tokens).
Mechanism: The kernel overhead (~25 ns/vec) is lower than the memory bandwidth savings achieved by transferring 3× less data per decode step.

Quality (Perplexity)

SRFT vs. SRHT: No statistically significant difference in $\Delta PPL$ across all tested models.
Qwen2.5-1.5B: The fused kernel with per-channel + per-group scaling reduces the 4-bit $\Delta PPL$ from a catastrophic +7975 (per-token baseline) to +638.6 (a 12.5× reduction), matching the Python reference implementation.
Gemma-3 1B: 4-bit $\Delta PPL$ improved from +18.7 (per-token) to +3.6 with the fused scaled kernel.
Lossless Regime: At $d=256$ , learned Householder rotations achieve $\Delta PPL = 0.000$ .

Memory Efficiency

Compression: Achieves 3.0× to 3.56× persistent memory compression on full-attention layers.
Real-world Impact: For a 128K context on Llama-3-70B, the int4 cache reduces memory from ~41 GB (fp16) to ~10.9 GB.

Significance and Claims

The paper claims that quantization is "free" (throughput-positive) on unified-memory hardware, inverting the standard industry assumption that low-bit quantization always incurs a latency penalty.

Inversion of the Trade-off: The primary contribution is demonstrating that on Apple Silicon, the bandwidth savings from a 3× compressed cache exceed the computational cost of the quantization kernel. This makes int4 KV cache strictly superior to fp16 for latency-sensitive, long-context inference on laptops and edge devices.
Hardware-Aware Design: The work highlights that the choice of rotation (FFT vs. Hadamard) is less about mathematical superiority and more about hardware alignment (e.g., FFT's radix-8 structure matching AMX 16×16 tiles).
Calibration vs. PPL Separation: The paper isolates a novel phenomenon where minimizing calibration MSE does not guarantee better downstream PPL, suggesting the fixed random rotation provides a necessary regularization prior.
Deployment Viability: By fusing the entire pipeline into a single Metal kernel and integrating it with HuggingFace, the authors prove that aggressive 4-bit quantization can be deployed without sacrificing model quality or speed, provided the hardware architecture (unified memory) and kernel optimization (fusion) are aligned.

The authors conclude that while the standard quantization trade-off holds for GPUs with separate HBM and high dispatch overheads, it is inverted for the unified-memory regime where laptop and edge inference operate.