Technical Summary: OSCAR - Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

1. Problem Statement

Long-context inference in Large Language Models (LLMs) is increasingly bottlenecked by the memory footprint of the Key-Value (KV) cache. As context length, batch size, and model depth grow, the cache consumes significant GPU memory, limiting throughput and scalability. While compressing the KV cache to ultra-low precision (e.g., INT2) offers an 8× memory reduction, it faces a critical accuracy challenge: KV activations contain severe channel-wise outliers.

Under low-bit quantization, these outliers dominate the quantization scale, forcing the majority of "normal" values into a few effective quantization levels and severely degrading attention quality. Existing rotation-based methods (e.g., Hadamard transforms) attempt to smooth these outliers by redistributing energy across channels. However, these methods are data-oblivious; they smooth the raw cache distribution but fail to align with the specific directions that downstream attention mechanisms actually consume. At INT2, where only four quantization levels exist, this misalignment causes naive rotation methods to collapse, resulting in near-zero accuracy on reasoning tasks.

2. Methodology: OSCAR

The authors propose OSCAR (Offline Spectral Covariance-Aware Rotation), a method that estimates attention-aware covariance structures offline to derive fixed rotations and clipping thresholds. Unlike prior work that minimizes raw Euclidean reconstruction error ($\|K - \hat{K}\|_F$), OSCAR minimizes the distortion in downstream attention scores and outputs.

2.1 Theoretical Motivation

The paper argues that the optimal rotation target should be derived from the statistics of the attention computation itself:

• For Keys ($K$): The distortion in attention logits is controlled by the query covariance $Q^\top Q$, not just the key covariance $K^\top K$.
• For Values ($V$): The distortion in the attention output is weighted by the attention scores $S$, leading to a target covariance of $V^\top S^\top S V$.

2.2 Algorithm Design

OSCAR operates in two phases:

A. Offline Calibration Phase:

1. Covariance Estimation: Using a small calibration dataset, the method computes empirical target covariances:
   • $C_Q = \frac{1}{N} \sum q_n^\top q_n$ (Query-aware key target).
   • $C_S = \frac{1}{N} V^\top S^\top S V$ (Score-aware value target).
2. Rotation Derivation: The method performs eigen-decomposition on these matrices to find the optimal eigenbases ($U_Q$ and $U_S$).
3. Composite Rotation Construction: The final rotation matrices ($R_K, R_V$) are constructed as a product of three components:
   • $U_Q/U_S$: The attention-aware eigenbasis that aligns channels with importance directions.
   • $H_{Had}$: A Hadamard transform that equalizes the importance metric across channels and spreads outlier energy.
   • $P_{br}$: A bit-reversal permutation that interleaves high- and low-variance channels to ensure balanced dynamic ranges within quantization groups.
   • Formula: $R_K = U_Q H_{Had} P_{br}$ and $R_V = U_S H_{Had} P_{br}$.
4. Clipping Thresholds: Per-layer clipping thresholds are estimated from the rotated calibration activations to control outliers.

B. Online Serving Phase:

• Mixed-Precision Layout: The system maintains a paged KV cache where "sink" tokens (initial) and "recent" tokens are kept in full BF16 precision. The historical middle context is stored in INT2 after applying the fixed OSCAR rotation and clipping.
• Kernel Integration: The authors developed a custom fused Triton kernel for the SGLang serving framework. This kernel handles the unpacking, dequantization, inverse rotation, and attention computation for the INT2 history, ensuring compatibility with paged attention and prefix caching without introducing extra memory traffic.

3. Key Contributions

1. Identification of the Missing Target: The paper identifies that generic rotations scatter outliers but fail at INT2 because they do not align with downstream attention. The rotation target must be induced by attention statistics ($Q^\top Q$ and $V^\top S^\top S V$), not raw cache reconstruction.
2. OSCAR Framework: A fully deployable INT2 KV-cache quantization system that uses offline calibration to derive attention-aware rotations. Theoretical analysis shows these targets are optimal under a frozen-error surrogate assumption.
3. Production-Ready System: A custom INT2 attention kernel integrated into SGLang that supports paged KV-cache layouts and prefix caching. It achieves seamless integration with modern serving frameworks (SGLang, vLLM) while maintaining high throughput.

4. Experimental Results

The authors evaluated OSCAR on reasoning models (Qwen3-4B, Qwen3-8B, Qwen3-32B) and a frontier model (GLM-4.7, 358B parameters) across five benchmarks (AIME25, GPQA, HumanEval, LiveCodeBench, MATH500) and long-context retrieval (RULER-NIAH up to 128k).

• Accuracy Preservation:

  • On Qwen3-4B-Thinking and Qwen3-8B, OSCAR reduces the accuracy gap to BF16 to 3.78 and 1.42 points, respectively. In contrast, naive rotation INT2 collapses to nearly zero accuracy.
  • On Qwen3-32B and GLM-4.7, OSCAR remains effectively on par with BF16 (within 0.02–0.27 points), while other INT2 baselines suffer significant degradation.
  • In long-context retrieval (RULER-NIAH), OSCAR maintains robust performance up to 128k tokens, whereas naive INT2 and Hadamard-only methods collapse at much shorter lengths (e.g., 16k).

• System Performance:

  • Memory: Reduces KV-cache memory footprint by approximately 8× compared to BF16.
  • Throughput: Improves throughput by up to 7× at large batch sizes under the same memory budget.
  • Latency: Accelerates batch-size-1 decoding by up to 3× over BF16 due to reduced memory bandwidth overhead.
  • Scalability: Demonstrates the ability to handle up to 28 concurrent requests with 100k-token inputs on a single H100, a regime where BF16 and INT4 baselines run out of memory.

5. Significance and Claims

The paper claims that OSCAR successfully bridges the gap between the theoretical promise of ultra-low-bit (INT2) KV caching and practical deployability. By shifting the rotation target from raw data reconstruction to attention-aware covariance, OSCAR recovers the accuracy lost by naive quantization while retaining the massive memory and throughput benefits of 2-bit storage.

The authors emphasize that their approach is not merely an algorithmic improvement but a system-level solution. By providing a custom kernel that integrates with paged attention and prefix caching, they demonstrate that INT2 quantization can be used in production serving frameworks (like SGLang) without sacrificing the efficiency features (like prefix reuse) that modern LLM serving relies on. The work suggests that for long-context reasoning, the key to successful quantization lies in aligning the compression strategy with the specific statistical demands of the attention mechanism.