🤖 AI

Learning to Communicate Across Modalities: Perceptual Heterogeneity in Multi-Agent Systems

This paper investigates emergent communication in heterogeneous multi-agent systems, revealing that while multimodal agents achieve class-consistent but less efficient and more uncertain communication compared to unimodal ones, meaning is encoded distributionally rather than compositionally, and cross-system interoperability can be restored through limited fine-tuning despite initial perceptual misalignment.

Original authors: Naomi Pitzer, Daniela Mihai

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Naomi Pitzer, Daniela Mihai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine two people trying to solve a puzzle together, but they are in completely different rooms and can only see different parts of the picture. One person hears a sound (like a dog barking), and the other sees a picture (of a dog). They have never met, they don't speak a common language, and they can't see what the other is seeing. Their only way to connect is to send each other short, binary "beep-boop" messages (like a series of 1s and 0s) to figure out if they are talking about the same thing.

This paper is a study of how these two "agents" (computer programs) learn to talk to each other when they are experiencing the world in totally different ways. Here is what the researchers found, explained simply:

1. The "Noisy Walkie-Talkie" Effect

When the two agents share the same type of input (both hearing sounds), they get really good at compressing their thoughts into very short, efficient messages. It's like two people speaking the same language; they can say a lot with just a few words.

However, when one hears and the other sees, the conversation gets "noisier." To get the same job done, they have to send longer, more detailed messages. It's like trying to describe a visual image to someone who only hears; you have to use more words and be less certain because you are translating between two different worlds. The "hearing" agent has to work harder to explain what it hears so the "seeing" agent can guess the right picture.

2. The Secret Code is a Pattern, Not a Dictionary

You might think that in these computer languages, a specific "1" always means "dog" and a "0" always means "cat." The researchers tested this by flipping the bits (changing a 1 to a 0) in the messages.

They found that the meaning isn't stored in individual bits like a dictionary. Instead, the meaning is distributed across the whole pattern.

The Analogy: Think of a message like a melody. If you change one note in a song, the whole tune might change, but it doesn't mean that specific note is the word "happy." The meaning comes from how all the notes fit together.
The Finding: Some bits are "constant" (they almost always stay the same) and act like the backbone of the message. If you mess with those, the message breaks completely. Other bits are "variable" and wiggle around. Interestingly, in the mixed-modality setup (hearing vs. seeing), the agents became less sensitive to these wiggly bits, suggesting they were focusing only on the most essential, shared information to survive the communication gap.

3. The Sender Stays True to Its Own World

Even though the "Receiver" sees the world differently, the "Sender" (the one making the messages) doesn't try to pretend it sees the world the same way.

The Analogy: Imagine a musician playing a song for a painter. The musician doesn't try to paint the notes; they play the music exactly as they hear it. The painter then has to interpret the music to understand the song.
The Finding: The researchers found that the messages the Sender sent still carried the "fingerprint" of the sound (like the pitch or frequency). Even though the Receiver couldn't hear the sound, the structure of the message still reflected the original sound's properties. The Sender didn't detach from its reality; it just hoped the Receiver could decode it.

4. Learning to Dance with a New Partner

The most exciting part was testing if these agents could switch partners.

The Scenario: They took a "Sender" trained to talk to a "Visual Receiver" and paired it with a brand new "Audio Receiver."
The Result: At first, they couldn't understand each other at all. It was like two people speaking different dialects who had never met. But, after a very short period of "fine-tuning" (practicing together for just a few rounds), they quickly learned to understand each other again.
The Twist: Once they learned to talk to the new partner, they actually got better at talking to the original partner too, though they had to find a middle ground. It showed that these agents are flexible; they can adapt their "language" to fit a new person's way of seeing the world.

The Big Takeaway

This study shows that communication doesn't require everyone to have the exact same view of the world. Even when agents are "blind" to each other's senses, they can build a shared language. However, it costs them more energy (longer messages) and creates more uncertainty. The meaning they create isn't a rigid code, but a flexible pattern that adapts to the gap between their different realities.

Technical Summary: Learning to Communicate Across Modalities

Problem Statement
Current research in emergent communication largely assumes that interacting agents share homogeneous perceptual representations or aligned representational spaces. This assumption overlooks a critical dimension of real-world communication: interlocutors often inhabit different perceptual worlds (e.g., one agent perceives via audio while another perceives via vision) yet must still establish successful communication. This paper investigates how such perceptual heterogeneity and misalignment influence the structure, efficiency, and interoperability of emergent communication protocols in multi-agent systems.

Methodology
The authors extend the multimodal, multi-step referential game framework (originally introduced by Evtimova et al., 2018) from an image-text setting to an audio-image setting.

Task: A Sender and a Receiver must identify a target object among a set of distractors. The Sender observes the object in one modality (e.g., audio of a dog barking), while the Receiver observes it in another (e.g., an image of a dog).
Protocol: The interaction is a multi-step dialogue where the Sender emits a $D$ -dimensional binary message ( $m^s_t$ ). The Receiver decides whether to terminate the dialogue and guess the object or to reply with a message ( $m^r_t$ ).
Agents: The Sender is a feed-forward neural network; the Receiver is a recurrent neural network. Both are trained jointly using RMSprop to minimize a combined objective of classification loss, REINFORCE-based reinforcement loss, and entropy regularisation.
Datasets: Experiments utilize a synthetic "Shapes World" dataset for controlled evaluation and a naturalistic dataset pairing CIFAR-100 images with UrbanSound8K and ESC-50 sounds to assess performance under noisy conditions.
Comparisons: The study compares multimodal systems (audio $\to$ image) against unimodal baselines (audio $\to$ audio).

Key Results

Efficiency and Uncertainty: Perceptual misalignment significantly impacts communication efficiency. While both systems tolerate message compression up to a threshold, multimodal agents require greater information exchange (longer messages) to maintain accuracy compared to unimodal agents. As message length decreases, multimodal systems exhibit degraded accuracy and higher classification entropy (uncertainty), suggesting that misaligned perceptual representations introduce noise into the communication channel.
Protocol Consistency:
- Sender Consistency: Senders develop strong class-consistent messages, converging toward deterministic encodings. This consistency increases as message capacity is constrained.
- Cross-Agent Consistency: Despite the Sender's internal consistency, there is little to no correlation between Sender and Receiver messages. This indicates that agents do not converge on a shared set of signals but rather develop individual protocols.
Grounding and Semantics:
- Bit Perturbation: Meaning is encoded in a distributional rather than compositional manner. Perturbing "constant" bits (those consistently 0 or 1) causes sharp drops in accuracy, whereas perturbing "variable" bits has minimal impact. Crucially, the informational role of a bit depends on its surrounding pattern; no single bit carries a fixed, interpretable meaning independent of context.
- Perceptual Grounding: Even when the Receiver operates on a different modality, the Sender's messages remain grounded in its own perceptual space. t-SNE visualizations show that message embeddings cluster according to low-level perceptual features (e.g., audio frequency), indicating that the communication protocol preserves traces of the Sender's perceptual structure.
Interoperability: Systems trained in different perceptual contexts (e.g., an audio-trained Sender paired with an image-trained Receiver) fail to communicate effectively without adaptation (accuracy drops to random guessing). However, limited fine-tuning (as few as 2–15 epochs) enables successful cross-system communication. Extended fine-tuning reveals a trade-off: while the new pairing improves, performance with the original partner may slightly degrade.

Significance and Claims
The paper positions emergent communication as a framework for studying how agents adapt and transfer representations across heterogeneous modalities. The authors claim that:

Perceptual misalignment creates a "modality gap" that necessitates higher capacity for reliable communication.
Meaning in these emergent systems is distributed across message patterns rather than being compositional or localized to specific bits.
Communication protocols can remain grounded in the Sender's specific perceptual reality even when the Receiver lacks access to that modality.
Agents possess the capacity to rapidly adapt their protocols to new interlocutors with different perceptual realities through minimal fine-tuning.

These findings offer a theoretical basis for understanding how shared understanding might arise across divergent perceptual worlds, with potential implications for robotics, epistemology, and theories of embodiment. The authors suggest future work should extend these analyses to embodied agent systems (e.g., robots) and alternative architectures to develop a broader account of heterogeneous emergent communication.