Technical Summary: See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

1. Problem Statement

Current Large Language Model (LLM) based Multi-Agent Systems (MAS) predominantly rely on text-based communication. While flexible, this approach incurs significant overhead due to the decode-and-re-encode cycle at every agent handoff, leading to information loss and computational inefficiency. Latent communication, specifically via Key-Value (KV) cache transmission, offers a promising alternative by allowing agents to share internal representations directly.

However, existing latent communication research faces two critical limitations:

1. Homogeneity Bias: Most prior work assumes homogeneous agents (identical model architectures), where latent spaces are naturally aligned. Transferring KV caches across heterogeneous agents (different model sizes, depths, or architectures) is non-trivial due to misaligned coordinate spaces, layer depths, and head structures.
2. Context-Aware Restriction: Existing heterogeneous methods typically operate in a context-aware regime, where the receiver already possesses the original input context. In this setting, the transmitted latent signal only needs to "steer" reasoning, allowing for sparse or lossy information transfer. It remains unclear if heterogeneous agents can perform context-unaware communication, where the receiver must solve a task solely based on the sender's transmitted latents without seeing the original input.

The core question addressed is: Can heterogeneous agents be aligned well enough to perform real "mind reading," transferring both what one agent sees (dense context) and how it thinks (reasoning signals)?

2. Methodology

2.1 Information-Structure Analysis (The Duality)

Before proposing a solution, the authors conduct a compressed-sensing analysis on KV caches in a homogeneous self-communication setting to understand the information bottleneck. They identify a fundamental duality between two communication regimes:

• Context-Aware Regime: The receiver has the input. The channel primarily requires sparse reasoning signals. A small subset of KV groups is sufficient to steer the receiver, as the receiver's local context provides the bulk of the knowledge.
• Context-Unaware Regime: The receiver lacks the input. The channel must carry dense contextual knowledge. Performance remains near chance until a large fraction of the KV cache is transmitted, indicating that task-critical information is distributed broadly across the cache.

This analysis reveals that prior sparse-steering methods are insufficient for context-unaware settings, necessitating a dense alignment framework.

2.2 Dense Latent Communication Framework

The proposed method, Dense Alignment, enables KV-cache transfer between heterogeneous models via a lightweight cross-model adapter and a two-phase training strategy.

Architecture Design

The transformation module $\mathcal{T}_\theta$ maps the sender's KV cache ($\mathcal{C}_S$) to a receiver-compatible cache ($\tilde{\mathcal{C}}_R$) through three key mechanisms:

1. Positional Disentanglement: Rotary positional embeddings (RoPE), which entangle content with model-specific rotations, are stripped from the sender's cache before transformation and re-applied using the receiver's positional convention afterward. This ensures content alignment independent of positional encoding differences.
2. Layer Alignment: A monotonic mapping aligns sender layers to receiver layers based on depth, preserving the progression from local features to task-level abstractions.
3. Per-Head Transformation with Gating: Each KV group is transformed via a lightweight Multi-Layer Perceptron (MLP). Crucially, a learnable gate ($\gamma$) is applied per KV group. This allows the model to dynamically prioritize sparse reasoning signals (identified by the compressed-sensing analysis) while maintaining the dense information required for context-unaware transfer.

Two-Phase Training Strategy

To ensure the adapter learns both dense reconstruction and actionable generation, the training is split into two phases:

• Phase I: Receiver-Cache Reconstruction: The adapter is trained to reconstruct the receiver's own KV cache from the sender's cache on paired inputs. The loss function minimizes the $L_2$ distance between the transformed sender cache and the receiver's native cache. This forces the adapter to learn a dense, reusable representation of the sender's state in the receiver's latent space.
• Phase II: Generation-Oriented Communication: The adapter is fine-tuned for downstream generation tasks. The training data includes a mixture of context-aware (receiver sees input) and context-unaware (receiver sees only the cache) prompts. This joint optimization ensures the transformed cache acts as a structured reasoning signal in the former and a dense surrogate for the missing context in the latter.

3. Key Contributions

1. Compressed-Sensing Analysis & Protocol: The authors provide a rigorous analysis revealing the sparse-vs-dense nature of latent communication. They introduce a stricter context-unaware protocol to evaluate whether latent channels can faithfully carry task-critical knowledge without external context.
2. Dense Alignment Framework: They propose a novel framework for heterogeneous KV-cache transfer that preserves both reasoning and contextual information. Key innovations include positional disentanglement, per-head gating, and a two-stage training strategy (reconstruction followed by generation).
3. Efficiency and Performance: The method demonstrates that dense alignment can outperform sparse-steering baselines and match or exceed text-based communication in accuracy while significantly reducing computational costs.

4. Experimental Results

The method was evaluated across six heterogeneous pairs of Qwen3 models (4B, 8B, 14B) on six benchmarks (GSM8K, MATH-500, ARC-C, MMLU-Redux, MedQA, OpenBookQA).

Context-Aware Results

• Accuracy: The method matches or exceeds Text-to-Text (T2T) communication on all in-domain tasks and performs competitively on out-of-domain (OOD) benchmarks. It consistently outperforms the sparse-steering baseline (C2C).
• Efficiency: The method achieves 2–3× lower compute (measured in TFLOPs per example) compared to T2T. In many cases, it is even more efficient than the "Receiver-only" baseline because the sender avoids autoregressive decoding and the receiver avoids re-encoding long text prompts.

Context-Unaware Results

• Robustness: This regime serves as a strict test of information density. Prior baselines (T2T and C2C) collapse in this setting, with C2C accuracy dropping to near 0% on reasoning tasks.
• Performance: The proposed dense alignment method remains highly effective, maintaining accuracy within 0–10 percentage points of its own context-aware performance across all pairs and benchmarks.
• Efficiency: In the 8B $\to$ 4B direction, the method achieves high accuracy at 6.6 TFLOPs, which is cheaper than the bare 4B receiver (9.2 TFLOPs) and 5× cheaper than T2T context-aware communication.

Visualization

PCA visualizations of the latent space confirm that the transformed sender caches overlap with the receiver's native manifolds, indicating genuine geometric alignment rather than brittle shortcuts.

5. Significance and Claims

The paper claims to solve the fundamental challenge of cross-model latent alignment in heterogeneous multi-agent systems. By distinguishing between sparse reasoning and dense knowledge requirements, the authors argue that prior evaluations in context-aware settings were insufficient to test the true capacity of latent communication.

The significance of this work lies in:

• Feasibility of "Mind Reading": Demonstrating that heterogeneous agents can transfer both "what they see" (context) and "how they think" (reasoning) without shared architectures.
• Computational Efficiency: Offering a communication channel that is not only more accurate than text in dense settings but also significantly cheaper, enabling efficient model switching and collaboration in resource-constrained environments.
• Scalability: Providing a pathway for dense information transfer that does not rely on the restrictive assumption of shared inputs or identical model structures.

The authors note limitations, specifically that the current method requires a training pass for each specific sender-receiver pair and that context-unaware transfer raises new privacy considerations regarding what can be inferred from the channel without the original prompt.