Technical Summary: Not All Thoughts Need HBM

Problem Statement

Reasoning-capable Large Language Models (LLMs), such as OpenAI o1 and DeepSeek-R1, generate extensive chains of thought (CoT), often comprising thousands of tokens. This deliberative process imposes a severe memory burden: every generated token adds a Key-Value (KV) pair to a cache that must reside in scarce, expensive GPU High-Bandwidth Memory (HBM) during inference. For instance, a 2,000-token reasoning chain on a 7B model consumes approximately 800 MB of HBM per request.

The dominant industry response to HBM scarcity has been eviction: permanently discarding tokens deemed "low-importance" to fit the cache within GPU memory. However, the authors demonstrate that this approach is catastrophic for reasoning tasks. Unlike standard language modeling where distant context is often redundant, reasoning chains function like programs where intermediate results computed hundreds of tokens ago may be referenced unpredictably later. The paper shows that evicting just 50% of the KV cache causes accuracy to collapse from ~22% to 0–2.5% (a "cliff effect"), regardless of the sophistication of the importance scoring mechanism.

Methodology: Semantics-Aware Memory Hierarchy

The authors propose a fundamental shift in the question from "which tokens can we discard?" to "where should each token live?" They introduce a four-tier, semantics-aware memory hierarchy that classifies tokens based on cumulative attention scores rather than permanently discarding them.

1. Four-Tier Classification

The system sorts reasoning tokens into four tiers based on real-time importance scoring:

• T0 (HBM): Contains protected tokens (prompt, attention sinks, recent window) and the highest-importance reasoning tokens. These remain in GPU HBM for immediate access.
• T1 (DDR): Medium-importance tokens are offloaded to CPU DDR memory at full precision. Crucially, these tokens are prefetched back to the GPU before every attention step.
• T2 (Compressed): Low-importance tokens stored in reduced precision (e.g., 8-bit quantization) on CPU. The authors note this tier is currently less effective for reasoning due to high precision sensitivity.
• T3 (Evicted): Only the absolute lowest-importance tail of tokens is permanently removed.

2. Importance Scoring

The system uses cumulative attention scoring. At each decoding step $t$, the importance score $s_i$ for position $i$ is updated by summing attention weights across all layers and heads:
$$s_i^{(t)} = s_i^{(t-1)} + \frac{1}{|L|} \sum_{\ell \in L} \frac{1}{H} \sum_{h=1}^{H} \alpha_{\ell,h}^{(t)}[i]$$
This approach captures the full history of a token's utility, accommodating the non-stationary nature of reasoning where a token might be critical hundreds of steps later.

3. Zero-Approximation-Error Offloading

The core theoretical contribution is the formalization of zero-approximation-error offloading. Because T1 tokens are prefetched to the GPU at full precision before attention computation, they contribute exactly the same terms to the attention output as if they had never left the HBM.

• Proposition 1: The approximation error depends solely on the set of permanently evicted tokens ($T3$), not on the partition between HBM ($T0$) and DDR ($T1$).
• Implication: Accuracy is determined by the eviction ratio (how many tokens are destroyed), not the HBM ratio (how many tokens remain in GPU memory).

4. System Implementation

The system employs dynamic tier assignment every $\Delta=64$ decoding steps to amortize sorting costs. It utilizes asynchronous differential prefetching, transferring only newly promoted tokens from DDR to GPU, minimizing PCIe traffic. A system prototype was built using real GPU–CPU data movement (PCIe Gen5) to validate latency and throughput.

Key Results

1. Accuracy vs. Eviction Ratio

Experiments across three model scales (7B, 14B, 32B) and four benchmarks (GSM8K, MATH-500, MATH Level-5, ARC-Challenge) confirm that:

• Eviction is destructive: Pure eviction (removing tokens without offloading) causes accuracy to collapse to near-zero at 50% budget.
• Offloading is effective: With only 3% permanent eviction, the hierarchy retains 91% of full-cache accuracy on GSM8K (n=200) and 71% on MATH-500.
• HBM Independence: Varying the HBM ratio (30%, 50%, 70%) while holding the eviction ratio constant results in statistically insignificant accuracy differences (p > 0.3). Conversely, varying the eviction ratio causes massive accuracy swings (up to 18%).

2. Comparison with State-of-the-Art

• vs. H2O/StreamingLLM: The hierarchy vastly outperforms heavy-hitter eviction baselines. At comparable memory budgets, H2O achieves ~2–30% accuracy, while the hierarchy achieves 58–65%.
• vs. R-KV (Reasoning-Aware Eviction): The authors reproduced R-KV (a concurrent SOTA method using joint importance-redundancy scoring) on their setup. R-KV achieved 0–32% accuracy depending on the budget, whereas the hierarchy achieved 56–62% with only 3% eviction. The authors argue that no scoring function can recover information that has been permanently destroyed.

3. System Overhead

The system prototype (RTX 5080, PCIe Gen5) demonstrates that the cost of preserving tokens in DDR is modest:

• Transfer Overhead: PCIe transfers account for only 5–7% of total inference time.
• Throughput: Throughput decreases by ~14% (from 21.8 to ~18.7 tokens/sec) compared to full-cache inference, but accuracy remains matched to the baseline.
• Memory Savings: Scaling projections indicate that for production batch sizes (e.g., batch=8, 70B models), this approach can save 2–48 GB of HBM, potentially enabling single-GPU serving for models that otherwise require multi-GPU setups.

4. Precision Sensitivity

The paper notes that reasoning tokens are highly precision-sensitive. Offloading to T2 (8-bit quantization) caused accuracy to drop from 58% to 20% at 5% eviction, suggesting that reasoning-aware compression is a necessary future direction but not yet a viable replacement for full-precision offloading.

Significance and Claims

The paper claims to fundamentally alter the paradigm of KV-cache management for reasoning LLMs:

1. Decoupling Capacity from Quality: It proves that HBM capacity can be aggressively reduced without sacrificing reasoning quality, provided that tokens are offloaded to cheaper memory (DDR) rather than discarded.
2. The "Cliff" vs. The "Knob": It identifies the "cliff effect" of eviction as a fundamental failure mode for reasoning, whereas the eviction ratio acts as a tunable knob for the hierarchy.
3. Architectural Parallel: The work borrows a principle from six decades of computer architecture—memory hierarchies—to solve a modern AI systems problem, arguing that "not all data needs to be equally close to the processor."
4. Zero-Error Guarantee: It provides a formal guarantee that offloading introduces no approximation error, distinguishing it from compression or quantization methods that inherently lose information.

The authors conclude that as reasoning chains grow longer with test-time compute scaling, the gap between "what must be remembered" and "what must be fast" will widen, making memory hierarchies a necessary evolution beyond the "eviction-or-nothing" paradigm.