💻 computer science

QKVShare: Quantized KV-Cache Handoff for Multi-Agent On-Device LLMs

QKVShare is a framework for efficient multi-agent on-device LLMs that utilizes quantized KV-cache handoff with mixed-precision allocation and a self-contained representation to significantly reduce time-to-first-token latency compared to full re-prefilling, particularly in deeper-hop scenarios.

Original authors: Pratik Honavar, Tejpratap GVSL

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pratik Honavar, Tejpratap GVSL

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of AI assistants working on a complex puzzle on a small, battery-powered laptop (an "edge device"). They need to pass the work back and forth.

In the current setup, when Assistant A finishes a step and hands the work to Assistant B, Assistant B usually has to re-read the entire history of the conversation from scratch just to remember what happened. This is slow and wastes a lot of memory. Alternatively, they could just pass a giant, high-definition photo of the memory, but that photo is so heavy it fills up the laptop's RAM instantly.

QKVShare is a new method proposed in this paper to solve this problem. Here is how it works, using simple analogies:

1. The Problem: The "Heavy Briefcase" vs. The "Re-Reading"

The Old Way (Re-prefill): Imagine Assistant A writes a 100-page story. When they pass it to Assistant B, Assistant B throws the story away and starts reading the first page again, all the way to the end, just to get up to speed. This takes a long time.
The "Full Precision" Way: Assistant A hands Assistant B a giant, high-resolution hard drive containing the exact memory of the story. It's fast to read, but the hard drive is so heavy it breaks the laptop's memory limit.

2. The Solution: The "Smart Compressed Card"

QKVShare introduces a new way to pass the memory. Instead of a heavy hard drive or re-reading the story, Assistant A creates a "CacheCard."

Think of the memory as a long line of words. Some words are critical (like the main plot points), and some are less important (like "um" or "the").

Quantization (Compression): QKVShare shrinks the memory. It turns the heavy "Full Precision" data into a lighter, compressed format (like turning a high-res photo into a smaller, efficient JPEG).
The "Smart" Part (Adaptive Allocation): This is the paper's main innovation. Instead of compressing everything equally, the system acts like a smart editor.
- It looks at the story and asks: "Which words does the next assistant really need to understand?"
- It keeps the most important words in high quality (high precision).
- It compresses the less important words heavily (low precision).
- The Goal: To make the "briefcase" as light as possible without losing the plot.

3. What the Paper Actually Found

The authors tested this on a specific math reasoning task (GSM8K) using a popular AI model (Llama-3.1-8B) on a laptop. Here is what they discovered:

Speed Wins: Passing the compressed "CacheCard" is significantly faster than making the next assistant re-read the whole history.
- Analogy: If re-reading takes 10 seconds, passing the card takes 3 seconds. As the story gets longer (more context), the time saved gets huge.
The "Smart Editor" is Promising but Not Perfect:
- When they used the "Smart Editor" (adaptive quantization) to decide which words to keep clear, it worked well, especially when the team had to pass the work back and forth many times (deep "hops").
- However, the paper admits that the "Smart Editor" didn't always beat a simpler method that just compresses everything the same way. The "Smart Editor" is a good idea, but the proof that it is consistently better than the simple method is still a work in progress.
Where the Time Goes: The authors broke down exactly where the time is spent. They found that the time it takes to create the card and hand it over is very fast. The slow part is actually the next assistant reading the card and starting to think.
- Analogy: The bottleneck isn't the delivery truck; it's the person unpacking the box.

4. What This Paper Does Not Claim

To be clear about the limits of this research:

It does not claim this works on every type of phone or tablet yet; it was tested on a specific laptop GPU.
It does not claim the "Smart Editor" is perfect; the authors admit they need more tests to prove it beats the simple methods in every scenario.
It does not claim this is ready for commercial apps today; it is a "prototype" (a working model) to prove the concept.

Summary

QKVShare is a new technique for AI assistants on small devices. Instead of making them re-read old notes or carry heavy files, they pass a "smartly compressed" version of the memory. This makes the team work much faster. The paper shows this is a very promising direction, though the "smart" part of the compression needs a bit more tuning to be perfect.

Technical Summary: QKVShare

Problem Statement
Multi-agent Large Language Model (LLM) systems deployed on edge devices face a critical bottleneck: when an agent passes context to a downstream agent, the receiving agent typically must reconstruct the latent state from scratch via a full re-prefill. This process is latency-intensive. Conversely, sharing the full-precision (FP16) Key-Value (KV) cache is memory-prohibitive on devices with limited RAM, as cache size scales linearly with context length. Existing solutions either waste latency (re-prefill) or memory (FP16 sharing), while current quantization methods focus on single-agent inference and do not address the interaction of quantized caches across agent boundaries.

Methodology
QKVShare proposes a framework for quantized KV-cache handoff between agents, centered on three components:

Topology-Aware Bit Allocation: The system extends the "Don't Waste Bits" controller to include cross-agent features. It assigns per-token bit-widths (from {2, 4, 8, FP16}) based on a combined score:
- Local Importance ( $I(t)$ ): Derived from frequency, quality score, attention variance, and entropy.
- Transferability ( $T(t)$ ): A new signal estimating downstream demand, calculated via segment-level priors (e.g., shared system prompts) and attention anchor overlap with previously shared tokens.
CacheCard Representation: A compact artifact packaging quantized KV groups, token-level bit-width assignments, and metadata (sequence length, model ID, position offsets) for inter-agent transfer.
Injection Path: For compatibility with HuggingFace Transformers, the current prototype reconstructs FP16 past_key_values from the quantized CacheCard before the receiving agent's forward pass. This allows for latency measurement while deferring the development of a fully fused quantized attention kernel to future work.

Key Contributions

Cross-Agent Token Scoring: An extended controller that allocates bits based on both local fidelity and the specific needs of the next agent in the topology.
Quantized Handoff Protocol: Definition of the CacheCard format and a receiver-side injection mechanism that enables quantized state transfer.
Measured Prototype Analysis: An end-to-end evaluation on Llama-3.1-8B-Instruct covering multi-hop reasoning accuracy and handoff Time-To-First-Token (TTFT), including a stage-level breakdown of latency.

Experimental Results
The evaluation utilizes 150 GSM8K problems with sequential agent chains (2–5 hops) and synthetic context benchmarks (1K–8K tokens) on a 12 GB NVIDIA RTX 5070 Ti Laptop GPU.

Accuracy (Multi-Hop):
- Uniform Quantization: Shows instability as hop depth increases, particularly at 4-bit precision.
- Adaptive Quantization: Demonstrates clear gains over uniform quantization in deeper-hop, higher-budget settings (e.g., at ~8 bits, adaptive topology-aware allocation reached 82.67% accuracy at 5 hops vs. 71.33% for uniform Q8).
- Topology vs. Local: At ~4 bits, the topology-aware controller showed a marginal net advantage (+2 wins) over a local-only adaptive controller but did not consistently outperform it. The paper treats the topology-aware advantage as an open hypothesis rather than a settled win at aggressive bit budgets.
Latency (TTFT):
- QKVShare significantly reduces TTFT compared to full re-prefill across all tested contexts. For an 8K context, QKVShare achieved 397.1 ms versus 1029.7 ms for re-prefill.
- Bottleneck Analysis: Stage timing reveals that the current latency is dominated by the post-injection generation (the receiving forward pass), not the CacheCard creation or injection itself.

Significance and Claims
The paper positions QKVShare as a promising systems direction for on-device multi-agent inference, arguing that quantized KV handoff avoids the high latency of re-prefilling while managing memory constraints.

However, the authors maintain a modest stance regarding the current evidence:

The prototype clarifies that avoiding re-prefill materially improves TTFT.
Adaptive mixed-precision allocation is promising under repeated handoff, but the specific value of topology-aware features over local-only adaptation is not yet conclusively isolated, particularly at 4-bit budgets.
The current results are directional due to mixed runtime stacks (PyTorch/HuggingFace vs. llama.cpp) and the reliance on a dequantization step for injection.
The paper explicitly identifies the need for stronger controller ablations and "apples-to-apples" runtime comparisons (e.g., a native fused Q4->Q4 kernel) to fully validate the topology-aware claims.