💬 NLP

Knowledge Packs: Zero-Token Knowledge Delivery via KV Cache Injection

This paper introduces Knowledge Packs, a method that achieves zero-token knowledge delivery and behavioral steering in causal transformers by injecting pre-computed KV caches, provided strict chat template formatting is maintained to ensure exact equivalence with standard RAG while enabling up to 95% token savings.

Original authors: Andrey Pustovit

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrey Pustovit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly forgetful assistant (the AI model). Usually, if you want them to know a specific fact—like "The capital of France is Paris"—you have to write it down in a note and hand it to them every single time you ask a question.

The Problem: The "Note-Taking" Tax
In the current world of AI (called RAG), every time your assistant needs to look up a fact, you have to type that fact out again.

The Analogy: Imagine you are playing a game of "20 Questions" with a friend. Every time they need to remember a rule, you have to read the entire rulebook page out loud to them. If you ask 5 questions, you've read the book 5 times. It's slow, it wastes your time (tokens), and it fills up the whiteboard (context window) so fast you run out of space.

The Solution: "Knowledge Packs" (The Magic Clipboard)
This paper proposes a new trick called Knowledge Packs. Instead of reading the rulebook out loud every time, you pre-write the rules onto a special "Magic Clipboard" (the KV Cache) once.

How it works: You give the AI the fact once in a special format. The AI processes it and writes the "essence" of that fact onto its internal clipboard.
The Magic: Next time you ask a question, you don't need to read the fact out loud. You just hand them the clipboard. The AI looks at the clipboard, and poof, it knows the fact.
The Result: You get the exact same answer, but you didn't have to spend any "words" (tokens) to say the fact again. It's like getting a free lunch.

The Catch: The "Secret Handshake"

The paper discovered a crucial detail: The Magic Clipboard only works if you use the exact right secret handshake (the Chat Template).

The Analogy: Imagine the AI is a bouncer at a club. If you walk up and say "I know the capital of France," they let you in. But if you walk up and say "I know the capital of France" without the secret password (like "System: Fact"), the bouncer thinks you're an imposter and kicks you out (or gives you a wrong answer).
The Mistake: Previous researchers tried to use the clipboard but forgot the password. They thought the clipboard was worse than reading the note out loud. The paper proves: "No, the clipboard is perfect! You just forgot the password." Once you use the password, the results are 100% identical to reading the note, but with zero cost.

The Superpower: "Steering" the Personality

Here is where it gets really cool. The paper found a way to not just give the AI facts, but also change its personality without saying a word.

The Analogy: Imagine the AI's brain has two types of notes:
1. The "What" notes (Keys): These are the facts.
2. The "How" notes (Values): These are the style, tone, and attitude.
The Trick: The paper found that you can tweak the "How" notes on the clipboard. You can take a "Rough and Ready" version of a fact and a "Polite and Professional" version, mix them together on the clipboard, and the AI will suddenly start answering in a polite, professional tone—even though you never typed the word "polite."
The Limit: You can't tweak the "What" notes (the facts) because the AI's internal math (RoPE) would break if you tried to add numbers to them. But you can tweak the "How" notes.

The "Dual-Channel" Mode

The best part? You can do both at the same time.

Scenario: You want the AI to know a complex medical fact (Knowledge Pack) and you want it to answer like a calm, reassuring doctor (Steering).
Result: You load the fact onto the clipboard, and you apply a tiny "calmness" tweak to the middle layers of the clipboard. The AI answers the medical question correctly and sounds like a gentle doctor, all without you typing a single extra word.

Summary in Plain English

Stop wasting words: Instead of typing facts over and over, save them as a pre-computed "memory file" (Knowledge Pack). It saves up to 95% of the space and time.
Don't forget the format: You must wrap these facts in the AI's specific "chat template" (like a specific envelope), or the AI won't understand them.
Change the vibe: You can tweak these memory files to change how the AI sounds (e.g., more defensive, more formal) without retraining the AI.
Do it all at once: You can give the AI facts and change its personality simultaneously, and it works perfectly.

In short: This paper is like discovering that instead of reading a library book to your friend every time they ask a question, you can just hand them a pre-loaded brain chip. It's faster, cheaper, and you can even program the chip to make your friend sound nicer.

1. Problem Statement

Retrieval-Augmented Generation (RAG) solves the knowledge problem by inserting retrieved text directly into the model's prompt. While effective, this approach incurs a linear token tax: every retrieved fact consumes context window space and inference tokens.

The Bottleneck: In agentic workflows requiring multiple sequential searches (e.g., 5 steps), the token cost for facts alone can exceed 700 tokens, rapidly depleting the context budget.
The Goal: Deliver the same knowledge with zero token cost during inference while maintaining exact output equivalence to standard RAG.

2. Methodology: Knowledge Packs

The authors propose Knowledge Packs, which utilize pre-computed Key-Value (KV) caches to inject knowledge into the model without re-tokenizing the text at inference time.

A. KV-Prefix Equivalence

The core theoretical insight is KV–Prefix Equivalence. In a decoder-only transformer with causal masking:

The KV cache generated by a standalone forward pass on a fact sequence $F$ is bit-identical to the KV entries for $F$ generated during a joint forward pass on $F \circ q$ (facts followed by a query).
Mechanism: Because causal masking restricts token $x_i$ to attend only to $j \leq i$ , the internal state (KV) of the prefix $F$ remains unchanged regardless of whether the query $q$ is appended later.
Implementation:
1. Write Phase (Offline): Facts are formatted using the model's specific chat template (including system messages and special tokens), tokenized, and a forward pass is run to extract the KV cache ( $K^{(l)}, V^{(l)}$ ).
2. Read Phase (Inference): The query is formatted as a user message. The pre-computed KV cache is injected as the past_key_values argument, and generation proceeds from the query tokens.

B. Critical Constraint: Chat Template Formatting

The paper identifies a critical pitfall: Chat Template Compliance.

Instruction-tuned models expect specific special tokens (e.g., system, <|start_header_id|>) at the start of input.
Injecting raw text KV caches (without these tokens) causes an Out-of-Distribution (OOD) shift, degrading accuracy by 6–7 percentage points (pp).
Resolution: The KV cache must be computed on text wrapped in the exact chat template used for RAG. When formatted correctly, the output is byte-identical to standard RAG.

C. Scaling: Banked Routing

To handle large knowledge bases (>1 context window):

Facts are clustered into "banks" using k-means on BGE embeddings.
At query time, the system selects the nearest bank, ranks facts within it, and recomputes the KV cache for the top fact on-the-fly (latency ~5ms).
Storage cost is minimal (<1 KB/fact), allowing 5,000+ facts to fit in ~4 MB.

D. KV Composition

To combine multiple knowledge packs, caches cannot be simply concatenated (which breaks Rotary Positional Embeddings/RoPE). Instead, sequential composition is used: Process Pack A, then process Pack B using Pack A's KV cache as the prefix, ensuring position indices continue correctly ( $T_A + 1$ ).

3. Key Contributions

Proof of KV–Prefix Equivalence: Theoretically and empirically verified that pre-computed KV caches produce zero divergences compared to text-in-prompt RAG across 700 questions on Qwen3-8B and Llama-3.1-8B, provided chat templates are correct.
Identification of Formatting Pitfalls: Demonstrated that prior claims of KV outperforming RAG were likely artifacts of formatting differences (raw text vs. templated text). Correct formatting eliminates the accuracy gap.
Token Efficiency: Achieved up to 95% token savings in multi-step retrieval scenarios (e.g., 5 steps) by replacing linear text growth with constant KV injection.
Value-Space Steering: Discovered that KV caches allow for behavioral steering via contrastive deltas on cached values ( $V$ $V$ ), while leaving keys ( $K$ $K$ ) untouched.
- Why Values? RoPE rotates keys based on position; arithmetic on keys destroys coherence. Values are not rotated and tolerate arithmetic.
- Localization: Steering effects localize to mid-layers (33–66%).
- Composability: Independent steering directions (e.g., "defensive coding" vs. "documentation") are nearly orthogonal ( $\cos \approx 0$ ) and can be composed without interference.

4. Experimental Results

Accuracy Equivalence:
- Qwen3-8B & Llama-3.1-8B: KV-chat (templated) and RAG achieved identical Exact Match (EM) scores (e.g., 65.2% on HotpotQA).
- Divergence: 0/700 questions showed different outputs between the two methods when templates matched.
- Raw Text Penalty: KV-raw (no template) dropped 5.5–6.0 pp in accuracy.
Token Savings (Accumulation):
- RAG: Token cost grows linearly (~140–150 tokens per search step).
- KV: Token cost remains constant (~31–35 tokens, just the query).
- Result: At 5 search steps, KV saves ~700 tokens per query (95% reduction).
Value Steering Performance:
- Effect: Applying $\alpha \cdot (V_{good} - V_{bad})$ to mid-layer values successfully steered coding style (defensive patterns).
- Optimal $\alpha$ : 2.0 for Qwen, 1.5 for Llama.
- Coherence: High coherence (100%) maintained at optimal $\alpha$ , though Qwen collapsed at $\alpha=3.0$ .
- Dual-Channel: Knowledge delivery and steering can run simultaneously at $\alpha \leq 0.7$ without degrading factual accuracy, while increasing formality and response length.

5. Significance and Implications

Lossless Optimization: Knowledge Packs offer a drop-in replacement for RAG that is zero-token and requires no weight modification or training.
Cost Reduction: Drastically reduces inference costs for pay-per-token APIs and enables agents to perform deep multi-hop reasoning without hitting context limits.
New Control Mechanism: The ability to steer model behavior via mid-layer value deltas opens a new avenue for "activation steering" that is cheaper than re-running inference and more persistent than prompt engineering.
Clarification of Prior Work: The paper resolves confusion in the literature regarding KV vs. RAG performance, attributing previous discrepancies to chat template formatting errors rather than inherent model superiority.

Limitations:

KV caches are model-specific (cannot transfer between architectures).
Steering effects are currently about half as strong as text-in-prompt steering.
Dual-channel mode requires lowering the steering coefficient ( $\alpha$ ) to prevent interference, suggesting the knowledge and steering subspaces are not perfectly orthogonal.