SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

1. Problem Statement

Large Language Models (LLMs) are increasingly supporting massive context windows (millions of tokens), causing the Key-Value (KV) cache memory footprint to grow linearly with sequence length. In production serving systems (e.g., vLLM, SGLang), KV cache memory is a primary bottleneck limiting throughput and concurrency.

While many KV-cache compression methods exist, they often fail in real-world deployment due to a "System Gap":

• Incompatibility with Paged Memory: Modern systems use fixed-size, non-contiguous blocks (PagedAttention). Methods requiring mixed-precision buffers (e.g., KIVI, Kitty) or token eviction cause memory fragmentation or break page-table assumptions.
• Violation of Fused Kernels: High-throughput decoding relies on FlashAttention-style kernels that require regular, coalesced memory access. Methods using vector quantization (codebook lookups) or complex channel-wise scaling introduce irregular memory access patterns and extra latency, destroying the efficiency of fused attention kernels.
• Naive INT4 Failure: Simple token-wise 4-bit integer quantization (INT4) reduces memory by 4x but causes catastrophic accuracy degradation (often dropping scores to near zero) due to channel-wise activation outliers.

Core Question: What is the minimal, system-compatible 4-bit KV-cache quantization method that recovers accuracy without sacrificing serving throughput?

2. Methodology: SAW-INT4

The authors propose SAW-INT4, a system-aware framework that identifies Token-wise INT4 quantization combined with Block-Diagonal Hadamard Rotation (BDR) as the optimal solution.

Key Components:

1. Token-Wise INT4 Quantization:

   • Quantizes each token's key/value vectors independently using per-token/per-head scales and zero-points.
   • System Benefit: Preserves the tensor layout required for PagedAttention and allows for 4x memory reduction without fragmented memory management.

2. Block-Diagonal Rotation (BDR):

   • Motivation: Naive INT4 fails because specific channels have extreme outliers (high magnitude) that expand the quantization grid, causing massive errors in other channels.
   • Mechanism: Applies an orthogonal transform (Hadamard matrix) to the KV vectors before quantization. This redistributes energy across dimensions, smoothing outliers and making the distribution more uniform.
   • Block-Diagonal Optimization: Instead of a full dense rotation (which is computationally expensive), the hidden dimension is partitioned into smaller blocks (e.g., size 128). Independent Hadamard transforms are applied to each block.
   • Result: This approximates a full rotation but is significantly more efficient for GPU kernels, balancing quantization quality with computational cost.

3. Fused Implementation:

   • The authors implement a custom CUDA/Triton kernel that fuses the rotation and quantization steps directly into the decode attention kernel.
   • Zero Overhead: The rotation is applied "on-the-fly" as the query is streamed, avoiding separate kernel launches or extra global memory passes over the KV cache. This ensures the method integrates seamlessly into existing PagedAttention pipelines.

3. Key Contributions

• System-Aware Empirical Study: The paper conducts a systematic comparison of various quantization strategies (Naive INT4, Vector Quantization/K-Means, Hessian-aware quantization, and Rotation) under strict serving constraints (PagedAttention, fused kernels).
• Identification of BDR as the Key Enabler: The study reveals that Block-Diagonal Rotation is the single most critical factor. It recovers nearly all accuracy lost by naive INT4 (bringing performance within 1–3 points of BF16) while complex methods like Hessian-aware calibration or large K-Means codebooks offer only marginal gains with significant deployment overhead.
• Zero-Overhead Kernel: A fused rotation-quantization kernel is developed that matches the throughput of plain INT4, proving that the rotation cost is negligible when integrated correctly.
• Comprehensive Evaluation: Validated across multiple models (Qwen3-4B, 8B, 32B, GLM-4.7) and diverse benchmarks (GPQA, HumanEval, LiveCodeBench, AIME, MATH).

4. Experimental Results

• Accuracy Recovery:
  • Naive INT4: Collapses to ~0% accuracy on sensitive models (e.g., Qwen3-4B).
  • SAW-INT4 (BDR-128): Recovers accuracy to 73.78 (vs. 75.64 for BF16) on Qwen3-4B and 69.86 (vs. 70.84 for BF16) on Qwen3-8B.
  • Complexity vs. Gain: Adding Hessian-aware calibration or increasing K-Means centroids (up to 2048) provided negligible improvements over BDR alone but added calibration complexity.
• Throughput & Efficiency:
  • End-to-End Throughput: SAW-INT4 matches the throughput of plain INT4 and significantly outperforms BF16 (up to 41% higher system throughput in high-concurrency, long-context scenarios) because the 4x memory reduction allows for larger batch sizes.
  • Kernel Overhead: Profiling shows that fusing rotation into the decode kernel hides the rotation cost. The total kernel time for SAW-INT4 is nearly identical to plain INT4.
  • Comparison with Alternatives: Complex methods like K-Means or Hessian-based approaches suffer from significant throughput degradation (up to 2.5x slower in attention kernels) due to irregular memory access and extra computation, making them unsuitable for real-world serving.

5. Significance and Conclusion

• Systems Co-Design: The paper demonstrates that effective KV-cache compression is fundamentally a systems co-design problem. A method that is mathematically optimal offline may fail in production if it violates memory layout or kernel fusion constraints.
• Simplicity Wins: The most effective solution is surprisingly simple: Token-wise INT4 + Block-Diagonal Rotation. It avoids the pitfalls of complex vector quantization or heavy calibration while delivering near-lossless accuracy.
• Practical Impact: SAW-INT4 enables the deployment of LLMs with massive context windows on existing hardware without sacrificing latency or throughput, making it a viable candidate for production LLM serving engines.

In summary, SAW-INT4 bridges the gap between theoretical compression and practical deployment by showing that a lightweight, fused rotation strategy is sufficient to make 4-bit KV caching viable for real-world, high-throughput LLM serving.