PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference

1. Problem Statement

Large Language Model (LLM) inference is dominated by memory bottlenecks caused by the Key-Value (KV) cache, which stores attention states for every token in the context. This issue is exacerbated in multi-agent inference systems, where $N$ agents processing the same shared document context traditionally require $N$ independent, full-precision KV caches. This results in linear memory scaling ($O(N)$), making it inefficient to run multiple agents on long contexts.

Existing solutions address this in isolation:

• KV Compression: Reduces memory per cache via quantization but operates on isolated, per-request caches.
• Multi-Agent Sharing: Reuses common prefixes among agents but typically uses full-precision caches, failing to leverage compression.

PolyKV aims to solve the intersection of these problems: creating a single, shared, lossy-compressed KV cache pool that can be simultaneously accessed by multiple concurrent agents.

2. Methodology

PolyKV introduces a novel architecture combining asymmetric quantization, rotation-based compression, and a shared memory abstraction.

A. Core Architecture

• SharedKVPool: A write-once, read-many abstraction. It computes the full KV state for a shared document context once, compresses it, and stores it as a single object ($O(1)$ memory complexity relative to agent count).
• PooledAgent: Individual inference agents that receive a reference to the SharedKVPool. At inference time, each agent injects the decompressed KV tensors into its own fresh DynamicCache instance layer-by-layer, bypassing standard per-forward-pass accumulation. This eliminates per-agent copy overhead.

B. Asymmetric Compression Scheme (TurboQuant MSE)

PolyKV employs an asymmetric quantization strategy tailored to the sensitivity of Keys (K) and Values (V):

1. Keys (K): Quantized to int8 (q8_0) using per-tensor linear quantization. This preserves the stability of the softmax attention scores, which are highly sensitive to low-bit quantization.
2. Values (V): Compressed using TurboQuant MSE to 3-bit precision. This involves:
   • Rotation: Applying a Fast Walsh-Hadamard Transform (FWHT) to rotate the input vector, distributing outliers and making the distribution converge toward a Gaussian $N(0, 1)$.
   • Quantization: Mapping rotated coordinates to precomputed Lloyd-Max centroids (optimized for $N(0, 1)$).
   • Dequantization: Retrieving centroid values and applying the inverse FWHT.

Compression Ratio:
With Keys at 8 bits and Values at 3 bits (compared to a 16-bit bfloat16 baseline), the theoretical compression ratio is:
$$\text{Ratio} = \frac{16}{(8 + 3)/2} = \frac{16}{5.5} \approx 2.91\times$$

3. Key Contributions

• Shared-Pool Architecture: The first implementation of a write-once, read-many compressed KV memory model for concurrent agents, eliminating the $O(N)$ memory scaling of prior multi-agent systems.
• Asymmetric TurboQuant: A specific combination of int8 Keys and 3-bit FWHT-rotated Values that achieves a stable 2.91× compression ratio across different model architectures.
• Cross-Model Validation: Demonstrated stability on both SmolLM2-1.7B and Llama-3-8B (including GQA architecture), proving the method is architecture-agnostic.
• Scalability: Successfully scaled to 15 concurrent agents sharing a single context, with memory savings growing linearly with agent count (up to 97.7% reduction).
• PPL Inversion Finding: Discovered that for coherent, long contexts, the compressed cache can actually yield lower perplexity than the full-precision baseline, suggesting the quantization noise acts as implicit regularization.

4. Experimental Results

The system was evaluated across two models, three context lengths (600 to 7,194 tokens), and up to 15 agents.

Memory Efficiency

• Llama-3-8B (15 agents, 4K context): Reduced KV cache memory from 19.8 GB (full precision) to 0.45 GB.
• Reduction: 97.7% memory reduction while maintaining high output quality.
• Scaling: Memory usage for the pool remains constant ($O(1)$) regardless of the number of agents (e.g., 0.116 GB for 3 agents vs. 10 agents at 2K context).

Quality Metrics

• Perplexity (PPL):
  • Stability: PPL delta is invariant to the number of agents.
  • Context Length Trend: PPL delta improves as context length increases.
    • At 2K tokens: +1.59% degradation.
    • At 4K tokens: +0.57% degradation.
    • Inversion: At 1,851 coherent tokens on SmolLM2-1.7B, the compressed cache achieved -0.26% PPL delta (better than baseline).
• Semantic Similarity (BERTScore):
  • Mean F1 scores ranged from 0.928 to 0.970 across all configurations.
  • Token overlap was perfect (1.000) in SmolLM2 tests.
  • The slight PPL degradation did not translate to semantic loss; phrasing variations preserved factual equivalence.

5. Significance and Hypothesis

Significance:
PolyKV is the first system to combine a single shared, lossy-compressed KV pool with concurrent multi-reader access. It fundamentally changes the economics of multi-agent LLM inference, allowing massive concurrency on shared contexts with minimal memory footprint, making it feasible to run complex agent swarms on consumer-grade hardware (e.g., Kaggle T4 GPUs).

The Regularization Hypothesis:
The authors hypothesize that the TurboQuant FWHT noise acts as implicit regularization. In long, coherent documents, attention heads often attend to redundant, spurious correlations (attention sinks). The quantization noise introduced by FWHT rotation disrupts these spurious patterns, effectively "denoising" the attention mechanism. This explains why PPL improves (or degrades less) as context length and lexical coherence increase.

6. Limitations and Future Work

• Scale: Experiments were limited to 1.7B and 8B models; behavior at 70B+ is unknown.
• Metrics: Time-to-first-token (TTFT) and throughput were not reported.
• Validation: The PPL inversion mechanism requires controlled ablation studies to confirm causality.
• Future: Plans include testing on larger models (Qwen2.5-7B, 70B), standardizing WikiText-2 evaluation protocols, and measuring end-to-end latency.