💻 computer science

Tamarin (HSQ): Reversible Hierarchical KV-Cache Compression for LLM Inference

Tamarin (HSQ) introduces a reversible, three-tier hierarchical KV-cache compression scheme that routes tokens into a focal tier, learned summary vectors, and a CPU-based archive to achieve 12–28× GPU memory reduction for long-context LLM inference while maintaining near-baseline perplexity and retrieval accuracy.

Original authors: Alikhan Bazakov, Kirill Kiselev

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alikhan Bazakov, Kirill Kiselev

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to read a massive, 32,000-page book on a tiny tablet. The problem isn't the book's text; it's that your tablet's memory fills up instantly just by holding the pages you've already read. In the world of AI, this "memory" is called the KV cache, and for big models, it eats up all your computer's power, forcing you to throw away old pages to make room for new ones.

Most current solutions are like a librarian who, when the shelf gets full, decides to toss out the books they think you won't need. They might keep the first few pages and the last few, but if you suddenly ask a question about a detail in the middle of the book, that page is gone forever. The paper calls this "eviction," and it argues that this is a fatal flaw because you can't predict what a user will ask next.

The Big Idea: The Magic Index, Not the Trash Can

The authors of this paper, from Siamang Labs, propose a different approach called Tamarin (technically known as HSQ). Instead of throwing pages away, they treat the memory like a giant, searchable index.

Here is how their three-tier system works, using a library analogy:

The "VIP" Shelf (L1): A small, high-speed section on your tablet (GPU) holds the most important pages and the very latest ones. These are kept in full detail, just shrunk down a bit (4-bit precision).
The "Clue Cards" (L2): For the rest of the book, the system creates tiny, 3-bit "clue cards" for every group of 8 to 16 pages. These cards don't contain the story; they just contain a summary of what the pages are about. They live on your tablet too.
The "Deep Archive" (L3): The full, original text of every single page is stored in a massive warehouse across the street (CPU RAM), also shrunk down to 4-bit precision.

How It Works in Real Time

When the AI is reading and suddenly needs to answer a question, it doesn't guess. It looks at the Clue Cards (L2) first. A smart, tiny "reader" AI scores these cards to see which groups of pages are relevant to the current question.

If a card looks promising, the system instantly runs to the Deep Archive (L3), grabs the original pages, and brings them back to the tablet just before the AI makes its decision.

The paper proves that this "indexing" approach is the key. In a test where they disabled the "fetch from archive" step, the system's ability to find hidden information (called "needle-in-a-haystack" retrieval) dropped to 0%. This confirms that the clue cards are just for routing; the real content lives in the archive.

The Results: Massive Savings with Almost No Cost

The paper measured this on several models (specifically the Qwen3 family, ranging from 0.6 billion to 14 billion parameters). Here is what they found:

Memory Savings: At a context length of 32,000 tokens, Tamarin shrinks the GPU memory usage by 12 to 28 times. For example, a model that usually needs 4.5 GiB of memory for the cache at 32K tokens only needs about 172 MiB on the GPU. The rest lives in the CPU RAM.
Quality: For larger models (4B and 8B parameters), the quality is almost identical to the uncompressed version. The "perplexity" (a measure of how confused the AI gets) only goes up by 0.1% to 0.4%. The AI agrees with the original version on 96% to 97% of the words it chooses.
Retrieval: In tests with 500 attempts to find a hidden code at different depths in the text, the 8B model found the code 500 out of 500 times, statistically matching the uncompressed baseline.

What the Paper Rules Out and Where It Fails

The authors are very clear about what this is not and where it struggles:

It is not a speed boost: The paper explicitly states that Tamarin does not make the AI faster. In fact, because it has to fetch data from the CPU archive, the speed at 32K tokens drops to about 17–18% of the normal speed. The trade-off is capacity (fitting more text) for speed.
It doesn't work perfectly on tiny models: For the smallest model tested (0.6B parameters), the quality cost is higher. The paper notes a 6–10% increase in perplexity when using 4-bit weights, which is considered a failure for that specific size.
It's not a magic fix for all AI families: The method works great on Qwen3 models, but when they tried it on a different family (Mistral-7B), the results were messy, with a 20 percentage point drop in accuracy at certain depths. The authors suspect this is because Mistral lacks a specific "normalization" step that Qwen has, making the "clue cards" harder to read.
One specific "fragile" checkpoint: Even on the 14B model, there were two specific spots in the text where the system struggled to find the hidden needle, dropping to 80.4% accuracy. The paper attributes this to a specific flaw in that model's training, not the method itself.

The Bottom Line

The paper suggests that for large models (4B and up), Tamarin is a proven way to fit massive amounts of text into limited memory without permanently deleting information. It turns the problem of "running out of space" into a problem of "managing an index." While it slows things down and requires careful tuning for different types of AI, it successfully solves the memory bottleneck for long-context tasks where keeping every single piece of information is critical.

Technical Summary: Tamarin (HSQ) – Reversible Hierarchical KV-Cache Compression

1. The Problem

In autoregressive large language model (LLM) inference, the Key-Value (KV) cache is the dominant consumer of GPU memory, particularly in long-context scenarios. Unlike model weights, which can be quantized once, the KV cache grows linearly with context length. For example, a Qwen3-8B model in fp16 requires approximately 4.5 GiB of VRAM for a 32K token context, often exceeding the capacity of standard GPUs.

Existing solutions face a dichotomy with a shared blind spot:

KV Quantization: Compresses all entries to low precision (2–4 bits). While safe, the compression ratio is bounded (typically ~4× over fp16) and can degrade quality at lower bit-widths.
Token Eviction/Merging: Methods like StreamingLLM, H2O, and SnapKV permanently drop or fuse tokens deemed unimportant during the prefill phase. These methods achieve high compression ratios but are irreversible. If a token evicted early becomes critical for a query generated at a later decoding step, the information is lost.

The core limitation of prior work is that token importance is a function of the future query, which eviction-based methods must predict at compression time without the benefit of that future context.

2. Methodology: Tamarin (HSQ)

Tamarin (Hierarchical Semantic Quantization, or HSQ) reframes KV-cache compression as an indexing problem rather than a deletion problem. It maintains three synchronized representations of the context, ensuring every compressed token remains addressable and recoverable.

The Three-Tier Architecture

L1 (Focal Tier, GPU): A small, non-summarized set of tokens stored as 4-bit quantized vectors. This includes attention sinks, an optional recent window, and tokens explicitly marked as important by a learned reader.
L2 (Summary Tier, GPU): The remaining tokens are partitioned into groups (size $N \in \{8, 16\}$ ). Each group is compressed into two learned representative vectors ( $\hat{k}, \hat{v}$ ) stored at 3-bit precision. Crucially, these vectors act as routing keys, not content replacements.
L3 (Archive Tier, CPU RAM): The original (quantized) keys and values for every compressed token are stored in CPU RAM at 4-bit precision. This serves as the source of truth for content.

The Decode Process

During decoding, the process operates as a reversible index lookup:

Routing: A per-head router scores the L2 summary keys against the current query.
Fetching: Based on a routing budget ( $\beta$ ), the router selects the top-scoring groups. The system fetches the corresponding original tokens from the L3 CPU archive, dequantizes them on the GPU, and rotates them to their semantic positions.
Attention: The attention mechanism runs over the union of L1 tokens, L2 summaries, and the restored L3 originals. L2 summaries are masked for groups where originals were fetched to prevent double-counting.

Key Technical Components

Reader: A small MLP that scores token importance based on early decoder hidden states. It directs tokens to L1, L2 (medium groups), or L2 (low groups).
Compressor: A Perceiver-style cross-attention pooler that generates the 3-bit routing keys. It is trained to rank groups effectively under the model's query projections, not to reconstruct content.
Quantization (TurboQuant): Uses randomized scalar quantization with a fixed random orthonormal rotation to decorrelate coordinates. L1 and L3 use 4-bit precision; L2 uses 3-bit precision.
Implementation: Implemented as a ~9K line Python library on top of HuggingFace Transformers, requiring no custom CUDA kernels or model weight modifications.

3. Key Contributions

Architecture: A three-tier reversible hierarchy that separates routing (L2) from content (L3), allowing for high compression ratios without permanent information loss.
Validation Methodology: A rigorous statistical framework using paired trials ( $n=500$ per cell), exact McNemar tests, TOST (Two One-Sided Tests) for equivalence in binary retrieval, and one-sided non-inferiority margins for perplexity.
Findings on Scaling: The method demonstrates that quality costs decrease as model size increases, and it composes freely with 4-bit weight quantization for models $\ge$ 4B parameters.
Open Source: Release of the beta HSQ package under Apache 2.0, including model adapters and evaluation harnesses.

4. Results

Evaluated on the Qwen3 family (0.6B–14B) across fp16, int8, and int4 weights, and contexts up to 32K:

Compression Ratio: Tamarin reduces GPU-resident KV cache by 12–28× (e.g., Qwen3-8B at 32K drops from 4.5 GiB to 172 MiB on GPU, with 2.29 GiB on CPU).
Retrieval Performance:
- On 4B and 8B models, retrieval is statistically equivalent to the uncompressed baseline (TOST equivalence) across all depths.
- At 32K context, Qwen3-8B-int4 achieves 500/500 needle retrieval parity with zero discordant pairs.
Perplexity (PPL) Overhead:
- For 4B–14B models, PPL overhead is minimal (0.1–0.4%).
- On 4B models, multilingual perplexity was measured below the baseline (ratio 0.9972), suggesting the compression cost is below the measurement floor.
- Small models (0.6B) incur higher costs (4–6% in fp16, 6–10% in int4).
Token Agreement: Greedy token-level agreement is 96–97% with the baseline for 4B–14B models.
Speed:
- Prefill overhead amortizes at longer contexts (1.2–1.3× at 32K).
- Decode throughput is currently a bottleneck, running at 40–73% of baseline for contexts up to 8K, and dropping to 17–18% at 32K due to the cost of fetching and dequantizing from CPU.

5. Claims and Significance

The paper claims that Tamarin successfully shifts the paradigm of KV-cache management from deletion to indexing. By deferring the decision of which tokens are "important" until the query time (via the router), the system avoids the irreversibility of eviction methods.

Specific Claims:

Capacity over Latency: Tamarin is presented as a solution for fitting larger contexts and batch sizes into limited GPU memory, not for reducing latency.
Reversibility: Every compressed token is addressable; the system does not permanently discard information.
Model Scaling: The method becomes more effective as model capacity increases, with 4B and 8B models showing near-perfect equivalence to baselines.
Composition: It works seamlessly with 4-bit weight quantization for models of at least 4B parameters without measurable additional cost.

Explicit Limitations and Boundaries:

Small Models: Models below 4B parameters (specifically 0.6B) suffer from noise compounding, leading to higher PPL overhead.
Fragile Checkpoints: The Qwen3-14B-int4 checkpoint showed narrow "holes" in single-needle retrieval at specific depths, attributed to quantization fragility rather than the method itself.
Model Families: The method is statistically validated on Qwen3. An initial test on Mistral-7B showed a "family effect" with degraded performance, hypothesized to be caused by the absence of per-head key normalization.
Throughput: The current implementation trades decode speed for memory capacity, with significant throughput penalties at very long contexts (32K).

The paper concludes that Tamarin offers a practical, reversible path to long-context inference on consumer and datacenter GPUs, provided the user accepts the current trade-off of reduced decode throughput for massive memory savings.