🤖 AI

Emergent Communication between Heterogeneous Visual Agents through Decentralized Learning

This paper demonstrates that heterogeneous visual agents can develop shared, informative communication symbols through decentralized learning and local perceptual evaluation alone, with the resulting language's specificity and symmetry directly shaped by the similarity of their underlying visual representations.

Original authors: Mikako Ochiai, Masatoshi Nagano, Tadahiro Taniguchi

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mikako Ochiai, Masatoshi Nagano, Tadahiro Taniguchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine two people trying to describe a picture to each other, but they are wearing different kinds of "magic glasses."

Person A wears glasses that see the world in sharp, high-definition detail.
Person B wears glasses that see the world in a slightly blurrier, lower-resolution style.

Neither person can see what the other sees through their glasses. They only see the picture through their own lenses. Their goal is to agree on a secret code (a list of numbers) that describes the picture, so that if Person A says the code, Person B knows exactly which picture Person A is looking at, and vice versa.

This is the core experiment in the paper: Can two agents with different "visions" learn to speak the same language without a teacher telling them they are right or wrong?

The Game: "Guess or Reject"

Instead of a teacher grading them, the agents play a game called the Metropolis-Hastings Captioning Game (MHCG). Here is how it works:

The Speaker: Agent A looks at a photo through their glasses and comes up with a secret code (a sequence of numbers) they think describes it.
The Listener: Agent B looks at the same photo through their different glasses. They also generate their own code.
The Decision: Agent B has to decide: "Is Agent A's code a good description of my view of this photo?"
- If Agent B thinks, "Yes, that code fits my picture well," they accept it.
- If they think, "No, that code doesn't fit my picture," they reject it and keep their own code.
Learning: Over thousands of rounds, they swap roles. They only update their "brain" (their text generator) based on the codes that were accepted. They are learning to speak a language that works for both sets of glasses, purely by checking if the other person's guess makes sense to them.

What They Found

The researchers tested this with three scenarios:

1. The Twins (Same Glasses)
When both agents wore the exact same type of glasses, they quickly learned a rich, detailed language. They could describe many different things, and they understood each other perfectly.

2. The Moderate Mismatch (Slightly Different Glasses)
When one agent had high-definition glasses and the other had slightly lower-resolution glasses, they still learned to communicate.

The Result: They developed a smaller vocabulary. They didn't agree on as many specific codes as the twins did.
The Twist: However, the codes they did agree on were very precise. They only kept the descriptions that were so clear that both types of glasses could see them. It was like they agreed to only talk about the "big, obvious things" (like "a dog" or "a car") and ignored the tiny details that only one pair of glasses could see.

3. The Big Mismatch (Very Different Glasses)
When the agents had very different types of glasses (one high-definition, one completely different architecture), communication became much harder.

The Result: They learned very few codes. The codes they did learn were "coarse" (vague).
The Bias: The language became unbalanced. It started to look more like the language of the agent with the "stronger" glasses. The agent with the weaker glasses had to adapt more to the stronger one's way of seeing the world. The language wasn't a fair mix; it leaned heavily toward one side.

The Secret Sauce: The "Listener's Gut Check"

The paper found that the most critical part of this game was the Listener's ability to say "No."

If the researchers forced the listener to accept every code the speaker offered, the agents would just start repeating the same boring code over and over (like saying "1-1-1-1-1" for every picture). The "rejection" step was necessary to force them to find codes that actually carried useful visual information. The listener's internal "gut check" (evaluating the code against their own visual data) was the only thing keeping the language from collapsing into nonsense.

The Big Takeaway

The paper proves that shared symbols can emerge from private, different experiences. You don't need a shared teacher or a shared goal to create a common language. You just need two agents to constantly check if a guess makes sense to their own private view of the world.

However, the more different their views are, the simpler and more one-sided their language becomes. They can still talk, but they have to compromise on the details, and the resulting language often reflects the perspective of the "stronger" viewer more than the weaker one.

Technical Summary: Emergent Communication between Heterogeneous Visual Agents through Decentralized Learning

1. Problem Statement

The paper addresses a fundamental question in emergent communication (EC): What visual information can become shareable when agents possess different private visual representations?

Standard EC paradigms typically rely on a shared external task signal (e.g., a referential game where a listener must identify a target image based on a speaker's message). This shared signal couples agents through a common objective, potentially allowing emergent symbols to optimize for the task rather than being grounded in each agent's private perceptual evidence. The authors argue that to understand how symbols arise from private perception alone, a decentralized setting is required where agents are coupled only through exchanged messages and update their models using local perceptual evidence, without shared rewards, cross-agent gradients, or parameter sharing.

The core challenge is determining whether common symbols can emerge without shared perceptual access and how the similarity (or mismatch) between agents' private visual spaces constrains the content and symmetry of the resulting language.

2. Methodology

2.1 The Metropolis-Hastings Captioning Game (MHCG)

The authors instantiate their decentralized setting using the Metropolis-Hastings Captioning Game (MHCG), adapted for Vision-Language Models (VLMs).

Agents: Two agents (Agent A and Agent B) observe independently augmented views of the same image.
Architecture: Each agent consists of:
- A frozen visual encoder (pretrained ViT-based, e.g., DINOv2 or MAE).
- A randomly initialized text module (BERT-Tiny configuration) acting as both encoder and decoder.
- A ProbVLM adapter that maps modality-specific features into a shared probabilistic latent space, modeling distributions as diagonal generalized Gaussians.
Interaction Protocol:
1. Speaker: Encodes its visual observation and samples a candidate token sequence (caption) $c^*$ .
2. Listener: Generates its own caption $c$ for the same image.
3. Acceptance Decision: The listener evaluates the speaker's proposal $c^*$ against its own caption $c$ using a Metropolis-Hastings (MH) acceptance criterion. The decision is based on a likelihood ratio derived from the listener's own visual features and the text-side density parameters of the captions.
4. Update: If accepted, $c^*$ becomes the training target for the listener; otherwise, $c$ is retained. The roles swap in the next round.
5. Learning: Agents update only their text modules (encoder, decoder, projection layers, and ITM head) and ProbVLM adapters using the accepted captions as fixed targets, minimizing BLIP-style losses (Image-Text Contrastive, Image-Text Matching, and Language Modeling). The visual encoders remain frozen.

2.2 Experimental Setup

Dataset: MS-COCO (train2017 for training, val2017 for evaluation).
Visual Encoder Conditions: Three pairings of frozen encoders were tested:
1. HOMO: Identical encoders (DINOv2 ViT-B/14 for both).
2. HETERO1 (Moderate Heterogeneity): DINOv2 ViT-B/14 vs. DINOv2 ViT-S/14.
3. HETERO2 (Strong Heterogeneity): DINOv2 ViT-B/14 vs. MAE ViT-B/16.
Controls:
- NoCom: No communication (baseline).
- AllAccept: Accepts all proposals ( $r \equiv 1$ ).
- Random Accept: Matches MH acceptance rate but accepts uniformly at random.
- ITM-based: Uses the listener's discriminative image-text matching score instead of the MH likelihood ratio.

2.3 Evaluation Metrics

Cross-Agent Alignment: Top-1 cross-modal retrieval (Image-to-Text and Text-to-Image) and Inter-agent Text-Text Representational Similarity Analysis (RSA).
Visual Information Transfer: $\Delta R^2$ (permutation-corrected linear prediction of visual features from tokens) and Vision-Text RSA.
Visual Specificity: Global-normalized visual radius of shared token sequences in fixed CLS spaces.
Representational Bias: Partial-RSA bias ( $\Delta X$ ) to measure if emergent symbols align more strongly with one agent's visual geometry.

3. Key Contributions

Decentralized VLM Language Game: Introduced a setting where agents with heterogeneous visual encoders learn shared symbols without shared task rewards, cross-agent gradients, or parameter sharing.
MHCG Efficacy: Demonstrated that MHCG produces shared token sequences that preserve visual information across agents, significantly outperforming a no-communication baseline in alignment, prediction, and retrieval tasks.
Characterization of Heterogeneity:
- Encoder Mismatch: All cross-agent metrics decline as encoder mismatch increases.
- Moderate Heterogeneity: Reduces the number of shared sequences but preserves the visual specificity of individual sequences.
- Strong Heterogeneity: Yields fewer, coarser sequences with a directional bias toward one agent's representational structure.
Critical Role of MH Filtering: Showed that listener-side MH likelihood-ratio filtering is essential for preventing degenerate token formation (collapse) and enabling effective visual information transfer, outperforming discriminative matching or random acceptance controls.

4. Results

Performance vs. Baseline: In all encoder conditions, MHCG significantly outperformed the NoCom baseline across all metrics (e.g., in HOMO, cross-agent retrieval was ~92% vs. ~23% for NoCom).
Impact of Mismatch:
- HOMO: Highest performance, symmetric alignment.
- HETERO1: Performance decreased but remained robust. The system formed 39% fewer shared sequences than HOMO, but the per-sequence visual specificity (measured by visual radius) remained comparable.
- HETERO2: Significant performance drop. Shared sequences were fewer and covered much broader, more diverse image sets (coarser concepts).
Representational Bias: Under HETERO2, a clear directional bias emerged ( $\Delta A > 0, \Delta B < 0$ ), where the shared language aligned more strongly with Agent A's (DINOv2 ViT-B) visual geometry. This resulted in asymmetric retrieval performance (Agent B's tokens transferred better to Agent A's space than vice versa).
Ablation Studies:
- AllAccept: Produced high text-text alignment but with degenerate, low-diversity sequences and poor retrieval.
- Random Accept: Failed to transfer visual information, performing near chance.
- ITM-based: Improved over random but remained inferior to MHCG, indicating that the probabilistic MH filter is superior to simple discriminative matching for this decentralized setting.

5. Significance and Claims

The paper claims that shared symbols can arise from local perceptual evaluation alone, without the need for a shared external communicative objective. The study demonstrates that:

Visual Representational Similarity is a key factor shaping the resulting language: it determines both the content (specificity vs. coarseness) and the symmetry (bias) of the emergent symbols.
Decentralized Inference via MH sampling allows agents to negotiate a shared latent space despite having different frozen visual backbones.
Moderate Heterogeneity acts as a selection bottleneck, filtering out concepts that are not invariant across different visual representations, resulting in a smaller but more robust vocabulary of shared concepts.
Strong Heterogeneity leads to a breakdown in symmetry, where the shared language tends to privilege the representational structure of the agent with the more dominant or compatible visual encoder.

The authors position this work as foundational for understanding how private perceptual spaces and shared symbolic systems co-develop through interaction, offering a framework to study language grounding in distributed, heterogeneous multi-agent systems. They explicitly note limitations, including the use of frozen encoders (preventing perception adaptation), a single dataset, and a small vocabulary, suggesting that future work should explore whether shared symbols can reshape private perceptual spaces and how these dynamics scale with larger populations and diverse modalities.