🤖 AI

KVComm: Enabling Efficient LLM Communication through Selective KV Sharing

KVComm is a novel framework that enables efficient communication between Large Language Models in multi-agent systems by selectively sharing the most informative KV pairs based on attention importance scores, achieving performance comparable to direct input merging while transmitting only 30% of the layers' data.

Original authors: Xiangyu Shi, Marco Chiesa, Gerald Q. Maguire, Dejan Kostic

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiangyu Shi, Marco Chiesa, Gerald Q. Maguire, Dejan Kostic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two brilliant detectives, Detective Sender (Ms) and Detective Receiver (Mr), working on a complex case. Detective Sender has read a massive file of clues (the "Context"), but Detective Receiver only has the specific question to solve (the "Query"). To solve the case, Sender needs to tell Receiver what they found.

This paper, KVComm, is a new, super-efficient way for these two detectives to share information without wasting time or losing details.

The Problem: The Old Ways Were Clunky

Before this new method, there were two main ways the detectives tried to talk:

The "Long Letter" Method (Natural Language):
Sender tries to write a summary of the clues in plain English.
- The Flaw: It takes a long time to write and read the letter. Worse, human language is imprecise. If the clue is a subtle pattern, summarizing it in words might lose the nuance. It's like trying to describe a complex painting using only words; you lose the colors and textures.
The "Brain Dump" Method (Hidden States):
Instead of words, Sender tries to send a snapshot of their brain at the very end of reading the file.
- The Flaw: By the time the brain finishes reading, it's focused so hard on the last thing it read that it forgets the earlier details. It's like asking someone to summarize a 500-page book by only telling you what they thought about the very last sentence. They might miss the plot entirely.

The Solution: KVComm (The "Selective Highlighter")

The authors realized that inside these AI models, there is a hidden layer of data called KV Pairs (Key-Value pairs). Think of these as the raw, unfiltered notes the model takes while reading. They contain the "who, what, where, and why" of every single piece of information.

However, sending all the notes from every page of the file is too heavy and slow. It's like mailing a library to a friend.

KVComm introduces a smart "Selective Highlighter" strategy:

Don't Send Everything: Instead of mailing the whole library, the system picks only the most important pages.
The Selection Strategy: How do they know which pages are important?
- They look at where the model was paying the most attention (like a highlighter pen).
- They use a mathematical "Gaussian Prior" (think of it as a bell curve). This assumes the middle of the book usually contains the most useful story, while the beginning is just setup and the end is just the conclusion.
- They combine these two ideas to pick the "Golden Layers" of information.

The Magic Analogy: The "Smart Flash Drive"

Imagine you are sending a 100-page document to a colleague.

Old Way: You email the whole PDF (slow, heavy) or you read the whole thing out loud over the phone (takes forever, you might stutter and miss details).
KVComm Way: You open the document, and a smart AI instantly highlights the top 30% of the most critical paragraphs. You send only those highlighted paragraphs on a tiny USB drive.

Your colleague (the Receiver) plugs in the USB. Their computer doesn't just read the text; it merges these specific notes directly into their own thinking process. They can now solve the problem almost as if they had read the whole document themselves, but they did it in a fraction of the time.

Why This Matters

Speed: It's much faster because you aren't generating long sentences or processing useless data.
Accuracy: Because you are sending the raw "notes" (KV pairs) instead of a summarized "letter," you lose almost no information.
Efficiency: The paper shows that by sending just 30% of the data (the most important layers), the system performs almost as well as if they had sent everything.

The Bottom Line

KVComm is like giving two AI agents a telepathic shortcut. Instead of struggling to explain complex ideas in slow, clumsy words, they can instantly share the exact "mental notes" that matter most. This allows teams of AI agents to work together on huge, complex problems much faster and smarter than ever before.

1. Problem Statement

The paper addresses the critical challenge of inter-model communication in multi-agent Large Language Model (LLM) systems. While multi-agent systems (e.g., CAMEL, AutoGen) show promise, current communication protocols suffer from significant trade-offs between efficiency and effectiveness:

Natural Language (NL) Communication: Methods like Natural Language Debate (NLD) or CIPHER (embedding exchange) require multiple decoding steps, leading to high inference costs. Furthermore, the sampling process inherent in NL generation causes information loss, failing to capture the rich, nuanced context needed for complex reasoning.
Hidden State Communication: Methods like AC (Activation Communication) transmit the last token's hidden state. The authors identify a "information concentration bias" where the last token's hidden state contains most of the output-relevant information, making it difficult to preserve the receiver's state without corruption. Additionally, transmitting all hidden states from the sender creates a dilemma: early layers offer little computation benefit (similar to concatenating inputs), while late layers cause significant performance drops when prepended to the receiver.

Goal: Design a communication protocol that is effective (preserves rich information), efficient (minimizes computation and transmission), and general (applicable across tasks and model pairs).

2. Methodology: KVComm

The authors propose KVComm, a framework that enables communication by selectively sharing Key-Value (KV) pairs from the sender model ( $M_s$ ) to the receiver model ( $M_r$ ).

Core Mechanism

Medium: Instead of NL or hidden states, KVComm uses the KV pairs generated during the attention mechanism of each decoder layer. These pairs encode rich contextual information without requiring decoding steps for the sender.
Integration: The receiver ( $M_r$ $M_{r}$ ) processes its query ( $Q$ $Q$ ). At specific layers, it concatenates the received KV pairs from $M_s$ $M_{s}$ with its own KV pairs before the attention calculation:
- $k_r^l \leftarrow [k_{s}^{l_i}; k_r^l]$
- $v_r^l \leftarrow [v_{s}^{l_i}; v_r^l]$
  This allows $M_r$ to attend to both its own context and the sender's context simultaneously.

Selective KV Sharing Strategy

To avoid the overhead of transmitting all KV pairs (which is expensive), KVComm employs a layer-wise selection strategy to choose a subset of layers ( $M$ out of $L$ total layers) to share. This selection is based on two hypotheses:

H1 (Semantic Transferability): Intermediate layers encode the most transferable semantic knowledge (early layers capture surface patterns; late layers specialize in task prediction).
H2 (Attention Importance): Layers with stronger attention distributions (higher attention weights on context tokens) are more valuable for communication.

The Selection Algorithm:

Attention Importance Score ( $S_a^l$ ): Calculated as the average attention weights assigned to context tokens by all heads in layer $l$ during the prefill stage.
Gaussian Prior ( $P^l$ ): A prior distribution centered at a specific layer $\mu$ (typically the middle layer) with standard deviation $\sigma$ . This enforces the selection of intermediate layers, aligning with H1.
Final Score ( $S^l$ ): A weighted combination: $S^l = \alpha S_a^l + (1-\alpha)P^l$ .
Selection: The top $M$ layers with the highest scores are selected.
Calibration: The selection strategy is calibrated using a very small set (even a single sample) and remains fixed for the test set, ensuring low overhead.

3. Key Contributions

Analysis of Activation Types: The paper rigorously demonstrates that hidden states are suboptimal for communication due to information concentration bias and the "dilemma" of layer selection. It establishes KV pairs as a superior medium that preserves information without interfering with the receiver's hidden state directly.
KVComm Framework: A novel protocol enabling efficient communication via selective KV sharing. It is the first approach to enable non-contiguous layer selection, allowing the system to pick the most informative layers scattered throughout the network rather than a single contiguous block.
Selection Strategy: A heuristic based on attention importance scores and a Gaussian prior. The authors show this strategy can be robustly calibrated with minimal data (1 sample) and generalizes well across diverse tasks.
Empirical Validation: Extensive experiments across 9 model pairs and 8 diverse datasets (including HotpotQA, QASPER, and synthetic tasks) demonstrate the method's superiority.

4. Experimental Results

Performance: KVComm consistently outperforms baseline methods (NLD, CIPHER, AC) and approaches the Skyline upper bound (where inputs are concatenated directly without communication overhead).
- With 70% of layers shared, KVComm matches Skyline performance.
- With only 30% of layers shared, KVComm still significantly outperforms all other communication baselines.
- In some cases, KVComm exceeds the Skyline performance, attributed to the regularization effect of selective sharing, which helps the receiver focus on the most relevant signals.
Efficiency:
- Computation: KVComm reduces computation costs by 2.5x to 6x compared to the Skyline method (which processes the full concatenated context) and significantly more compared to NLD (which requires multiple decoding rounds).
- Memory: KVComm uses 23% to 73% less memory than Skyline by reducing the number of active layers and KV cache size.
- Communication: Transmits as few as 30% of the KV pairs compared to methods transmitting the full set.
Ablation Studies:
- Non-contiguous vs. Contiguous: Selecting non-contiguous layers (KVComm) consistently outperforms selecting a single contiguous chunk of layers (as done in DroidSpeak), proving the importance of targeting specific informative layers regardless of depth.
- Random vs. Heuristic: The attention-based selection strategy significantly outperforms random selection, especially at low transmission ratios (30%).

5. Significance and Future Work

Scalability: KVComm provides a scalable path for multi-agent systems by drastically reducing the communication and computation bottlenecks associated with current LLM collaboration methods.
Generalization: The method works effectively across different model families (Llama, Qwen, Falcon) and fine-tuned variants, provided they share the same base architecture.
Future Directions: The authors suggest extending KVComm to heterogeneous architectures (via learned projections), multi-sender/receiver scenarios (demonstrated to improve performance via information diversity), and online adaptive calibration where layer selection is dynamically adjusted per query.

In conclusion, KVComm represents a paradigm shift in LLM communication, moving from text-based or hidden-state-based exchanges to structured, selective activation sharing, offering a highly efficient and effective solution for next-generation multi-agent AI systems.