🤖 AI

Dual-Cache Latent Space Communication between Heterogeneous Language Models

The paper introduces XKV, a novel communication protocol that enables heterogeneous, frozen language models to exchange information via a learned translator of their KV caches, overcoming prior limitations in geometry and layer alignment to achieve superior accuracy and significantly faster inference compared to both text-based and previous latent-space communication methods.

Original authors: Jiyao Liu, Qi Zhang, Yaoyi Jia, Ziwen Kan, Song Wang

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiyao Liu, Qi Zhang, Yaoyi Jia, Ziwen Kan, Song Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern landscape of artificial intelligence, large language models are increasingly acting not as solitary thinkers, but as members of a team. Imagine a group of researchers working on a complex mystery, where each person has read only a different set of documents. To solve the case, they must share what they know. In the digital world, these "researchers" are software agents, and the documents they hold are pieces of evidence hidden inside their internal memory. For these agents to work together, they need a way to pass information from one to another. Until recently, the standard method was to have one agent write a summary in plain English and send it to the next. While this is easy for humans to read, it is inefficient for machines. The sending agent must slowly construct the message word by word, and the receiving agent must then read those words and rebuild its own internal understanding from scratch. This process is like a relay race where the runners must stop, write a note, hand it over, and then spend time reading it before they can run again.

Researchers have been exploring a faster way to communicate, one that skips the writing and reading of words entirely. Instead of exchanging text, they tried to pass the raw, internal "thoughts" of one machine directly to another. This is akin to handing a runner a pre-written note that they can instantly absorb without reading. However, early attempts at this direct transfer hit a wall. They worked well only when the two machines were identical twins, or they forced the sender to compress all its knowledge into a single, generic summary that ignored what the receiver already knew. This meant the receiver was often handed a message that was either too vague or irrelevant to its specific needs. The challenge was to create a communication channel that was fast, worked between different types of machines, and allowed the receiver to ask for exactly the information it needed, rather than accepting a one-size-fits-all summary.

A team of researchers has now introduced a new system called XKV that solves these problems. Their approach treats the communication between two different language models as a joint effort rather than a simple handoff. In their system, the two models—one holding a piece of the puzzle and the other holding the rest—remain frozen in their original state, meaning their core intelligence is not altered. Instead, a lightweight translator sits between them. This translator looks at the internal memory of both models at the same time. It does not just summarize what the first model knows; it figures out what the second model is missing by comparing the two sets of memories. It then creates a compact, shared memory bank that blends the sender's evidence with the receiver's current state.

The innovation lies in how this shared memory is used. In previous systems, every part of the receiving model was forced to listen to the same single summary, like a classroom where every student hears the exact same lecture regardless of what they already understand. The new XKV system allows every single position in the receiver's memory to ask a specific question to this shared bank. It is as if each student in the class could raise their hand to ask for the specific fact they are missing, and the teacher would provide just that. The system then delivers a precise, tailored update to each part of the receiver's memory, filling in the gaps without disturbing the rest of its knowledge. This process happens in a fraction of the time it takes to write and read a text message, and it works even when the two models are built on completely different architectures, use different vocabularies, or have different numbers of internal layers.

The researchers tested this system across a wide variety of scenarios, pitting different families of language models against each other on five different reasoning tasks. These tasks involved answering questions that required combining evidence split between the two agents, such as connecting facts from different paragraphs to solve a multi-step problem. The results showed that the new system consistently outperformed both the traditional text-based method and the previous best attempt at direct memory transfer. On average, the new system improved the accuracy of the answers significantly, with some specific tests showing a jump of over four percentage points in exact match scores compared to the previous best method. Beyond just being smarter, it was dramatically faster. The translator component of the new system ran more than ten times faster than the previous leading method and was nearly seven times faster than the standard text-based approach.

The study also highlighted that this speed did not come at the cost of complexity. The new translator required 76 percent fewer trainable parameters than the previous leading method, making it much more efficient to build and run. Crucially, the system maintained its high performance even when the two models communicating were completely different from one another, such as a model from one company talking to a model from another, or a smaller model talking to a larger one. This flexibility suggests that the system can be deployed in real-world scenarios where diverse AI tools need to collaborate without needing to be retrained or forced into a uniform shape. The researchers found that the system worked best when it could build a joint memory from both sides, rather than relying on a summary from just one. This confirmed that the most effective communication happens when the sender and receiver are considered together, allowing the receiver to retrieve exactly what it needs from a shared pool of information.

The implications of this work extend beyond just making AI faster. By removing the need for models to speak in human language to understand each other, the system opens the door for more fluid and efficient multi-agent systems. These systems could coordinate complex tasks, such as analyzing vast amounts of data or solving intricate scientific problems, without the bottleneck of generating and re-reading text. The researchers demonstrated that it is possible to create a communication channel that is aware of the receiver's state, retrieves specific updates from a shared memory, and remains strictly cheaper and faster than the sender-centric summaries it replaces. This represents a shift from treating AI agents as isolated entities that must translate their thoughts into words, to treating them as a cohesive network that can share raw understanding directly. The findings suggest that the future of collaborative artificial intelligence may not lie in better language, but in better ways to share the silent, internal states that drive intelligence.

Technical Summary: Dual-Cache Latent Space Communication between Heterogeneous Language Models

Problem Statement

In multi-agent Large Language Model (LLM) systems, agents often possess complementary pieces of evidence required to solve a task. A "Sharer" model encodes part of the evidence, while a "Receiver" model holds the remainder and must generate the final answer. The core challenge is enabling efficient communication between these agents without the overhead of natural language exchange.

Current approaches face significant limitations:

Text-to-Text (T2T): Agents exchange natural language. This places autoregressive decoding on the critical path, requires the receiver to re-tokenize and re-encode the message, and operates without visibility into the receiver's internal state.
Cache-to-Cache (C2C): Translates Key-Value (KV) caches directly. While efficient, it requires both models to process the same underlying input (shared context) and align token positions, limiting it to scenarios with overlapping content.
Latent Cache Flow - Cross-Context (LCF-X): Removes the shared-context requirement by pooling the Sharer's cache into a position-free summary. However, it retains three critical bottlenecks:
- Receiver-Independent Compression: It compresses only the Sharer's cache without consulting the Receiver's state, forcing a single vector to carry all necessary facts.
- Layer-Local Single-Summary Bottleneck: Every position in a Receiver layer receives the exact same pooled summary, preventing position-specific retrieval or cross-layer information access.
- Geometric Rigidity: It assumes matched layer counts and KV-head geometries between models, failing in truly heterogeneous settings.

Methodology: XKV

The paper introduces XKV, a dual-cache translator designed to lift these restrictions while maintaining efficiency. XKV operates on two frozen models (Sharer and Receiver) that may differ in architecture, depth, KV-head count, head dimension, and tokenizer. Only the translator is trained.

The architecture consists of two primary stages:

Stage 1: What to Communicate (Joint Memory Construction)

Instead of compressing only the Sharer, XKV symmetrically pools both models' KV caches to create a joint, cross-layer memory.

Symmetric Direct Cache Pooling: For each layer and KV head, learned queries pool raw cache positions ( $N_a$ ) down to a small number of candidate slots ( $k$ ) for both Sharer and Receiver. This is a single vectorized step, avoiding hierarchical pooling.
Layer Alignment: Since semantic correspondence across layers varies by architecture, a learned affine map ( $W_L$ ) aligns the Sharer's layer axis to the Receiver's depth.
Cross-Layer Mixing: The pooled summaries from both models are concatenated, down-projected, and processed through a self-attention mechanism over the Receiver-aligned layer tokens. This produces a compact joint memory ( $Z$ ) of length $L_R$ (Receiver layers), representing the Sharer's evidence relative to what the Receiver has already encoded.

Stage 2: Where to Write (Receiver-Position Retrieval)

To address the "single summary" bottleneck, XKV allows every raw Receiver cache position to retrieve a distinct update.

Position Queries: Each raw Receiver key and value (flattened across heads) serves as an explicit query into the joint memory $Z$ via a shared cross-attention decoder.
Gated Residual Updates: The decoder outputs are passed through shared, zero-initialized output heads ( $W_K, W_V$ ) and a sigmoid gate ( $\sigma$ ). This generates per-head-gated residuals ( $\Delta K, \Delta V$ ) in the Receiver's native KV geometry.
Update: These residuals are added to the Receiver's cache ( $C'_R = C_R + \Delta C_R$ ) once, after which the Receiver generates the answer using standard causal decoding.

Efficiency Mechanisms:

The joint memory length is fixed to the Receiver's layer count ( $L_R$ ), independent of prompt length.
Parameters (bottleneck, translation blocks, decoder, gates) are shared across Receiver layers.
The zero-initialized output heads ensure that an untrained translator reproduces the Receiver's original behavior exactly, stabilizing early training.

Key Contributions

Identification of Bottlenecks: The authors identify receiver-independent compression, layer-local single-summary conditioning, and matched-geometry assumptions as the primary limitations of existing cross-context cache communication.
XKV Architecture: A novel translator that combines symmetric direct cache pooling, cross-layer translation via learned layer maps, and receiver-position queries. This enables the construction of a joint memory for receiver-native reconstruction, supporting models with differing depths, attention geometries, and tokenizers.
Comprehensive Evaluation: The paper extends LCF-X to handle mismatched layer counts and dimensions, creating a stronger heterogeneous baseline. It conducts the first full cross-family, split-evidence evaluation across five datasets and nine ordered model pairings (Qwen, Gemma, Llama).

Experimental Results

The evaluation covers 45 dataset–model-pair settings (5 datasets $\times$ 9 Sharer-Receiver pairs).

Performance: XKV achieves the highest macro score (52.4) compared to LCF-X (49.8) and T2T (48.9). It improves upon LCF-X on every dataset. Notably, on the ROPES dataset, XKV improves exact match by 4.6 points and F1 by 4.2 points over LCF-X.
Efficiency:
- Parameter Count: XKV trains 76% fewer parameters than LCF-X.
- Translator Latency: XKV translates a cache pair 10.3 $\times$ faster than LCF-X (5.8 ms vs. 59.9 ms).
- End-to-End Latency: XKV is 26% faster than LCF-X and 6.8 $\times$ faster than T2T.
Robustness: XKV outperforms text communication on four of the five datasets and dominates both baselines across all heterogeneous pairings (e.g., Qwen $\to$ Llama, Gemma $\to$ Qwen).

Significance and Claims

The paper claims to resolve the trade-off between communication quality and efficiency. XKV demonstrates that a communication channel can be receiver-aware (deciding what to communicate based on both models' states) and capable of retrieving position-specific updates from a joint cross-layer memory, while remaining strictly cheaper than the sender-centric summary it replaces.

The authors emphasize that this approach allows frozen models of different families and architectures to collaborate effectively without the cost of autoregressive decoding or the constraints of shared tokenization. The work establishes that efficient, heterogeneous multi-agent collaboration is achievable through learned latent space translation rather than natural language exchange.