🤖 machine learning

Emergent Compositional Communication for Latent World Properties

This paper demonstrates that multi-agent systems communicating through a Gumbel-Softmax bottleneck can spontaneously develop discrete, compositional protocols to represent latent physical properties like elasticity and mass from frozen video features, with performance heavily dependent on the perceptual prior and validated on both synthetic and real-world footage.

Original authors: Tomek Kaszyński

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomek Kaszyński

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a room with a friend, but you can't see each other's screens. You both watch a video of a ball bouncing and sliding. Your goal? To figure out which ball is "bouncier" and which one is "slipperier" just by sending each other short text messages.

This paper is about teaching AI agents to do exactly that, but with a twist: they have to invent their own language to describe things they can't actually see.

Here is the story of how they did it, broken down into simple concepts.

1. The Invisible Properties

In the real world, you can't see "friction" or "elasticity" in a single photo. You only know a ball is bouncy if you watch it bounce. You only know a surface is slippery if you watch it slide.

The Challenge: The AI agents are given videos of balls moving. They have to guess the hidden physics rules (like mass or bounciness) and tell their partner.
The Catch: They can't send long, detailed reports. They have to compress all that complex motion into a tiny, discrete message (like a short code of symbols).

2. The "Group Project" Effect (Multi-Agent Pressure)

The researchers found a magic ingredient: having more people in the group.

The Solo Artist: When they tried with just two agents, it was a coin flip. Sometimes they figured out a great system; other times, they just memorized random codes that didn't make sense.
The Orchestra: When they increased the team to four agents, something amazing happened. Every single time, they developed a perfect, organized language.
The Analogy: Imagine a group of four people trying to describe a painting to a fifth person. If they all shout at once, it's chaos. But if they agree that Person A describes the color, Person B describes the shape, and Person C describes the mood, they create a perfect, organized system. The paper shows that splitting the job among multiple independent observers forces the AI to organize its thoughts, rather than just memorizing the answer.

3. The "Discrete Bottleneck" (The Telephone Game)

The agents have to send their messages through a "bottleneck." Think of it like a game of Telephone, but the phone line is very narrow.

They can't send a video file. They can only send a few symbols (like "A-B-C").
Because the line is so narrow, the AI is forced to be efficient. It learns to say, "The first letter tells you about bounciness, and the second letter tells you about slipperiness."
This creates Compositionality: A fancy word meaning the language has parts that mean specific things, and you can mix and match them. Just like in English, "Red" + "Ball" means a red ball, and "Blue" + "Ball" means a blue ball. The AI learned to do this with physics.

4. The "Eyes" Matter (Perception is Key)

The paper tested two different types of "eyes" (AI vision models) to see what they could see:

The Photographer (DINOv2): This model looks at individual frames like a photographer. It's great at seeing static things (like how high a ball is on a ramp).
The Movie Critic (V-JEPA 2): This model watches the whole video and understands motion and time.
The Result:
- If the task was about a ball rolling down a ramp (where you can see the distance), the "Photographer" did great.
- If the task was about two balls colliding (where you have to watch how they speed up or slow down to guess their weight), the "Photographer" failed. The "Movie Critic" succeeded because it understood the story of the movement, not just the picture.
- Lesson: You can't teach an AI to speak about physics if its "eyes" can't see the physics in the first place.

5. The "Surgery" Test (Proof it Works)

How do we know the AI actually learned the rules and didn't just guess? The researchers played a trick.

They took the AI's message and deleted the part that was supposed to talk about "bounciness."
The Result: The AI's partner immediately failed at guessing bounciness but was still perfect at guessing "slipperiness."
The Metaphor: It's like if you took a sentence and erased the word "Red." If the sentence was "The red ball is heavy," and you erased "Red," the listener would still know the ball is heavy, but they wouldn't know the color. This proved the AI had separated the concepts cleanly.

6. Real-World Proof

Finally, they tested this on real video footage of real objects (like a ball bouncing on a real spring), not just computer simulations.

The AI successfully learned to compare the weight of real objects it had never seen before.
It proved that this "invented language" works even when the video is messy and real, not perfect and clean.

The Big Takeaway

This paper shows that if you give AI agents a reason to talk to each other, and you force them to split up the work among a team, they will spontaneously invent a structured, logical language to describe the invisible laws of physics.

It suggests that communication is a teacher. By trying to explain the world to a partner, the AI learns to understand the world better itself. And, crucially, it shows that to understand the movement of the world, you need a model trained on video, not just on still photos.

1. Problem Statement

The paper addresses a fundamental gap in artificial intelligence: Can multi-agent systems develop structured, compositional language to communicate about physical properties that are never directly visible in a single observation?

While humans effortlessly infer and discuss latent properties like elasticity, friction, and mass through temporal dynamics (e.g., how objects bounce or slide), current AI models typically rely on continuous representations for physical prediction. The authors investigate whether agents, driven solely by the pressure to cooperate on a comparison task, can spontaneously discover discrete, compositional protocols where specific message slots encode specific physical properties, even when those properties are invisible in static frames.

2. Methodology

Experimental Setup

Environment: The authors use physics simulations (Kubric/PyBullet) and real-world video (Physics 101 dataset).
- Ramp Physics: Objects slide down ramps (revealing friction) and bounce (revealing elasticity).
- Collision Dynamics: Identical spheres collide; mass ratio and restitution must be inferred solely from post-collision velocity changes over time.
Task: Two sender agents observe video clips of objects with hidden physical properties. They must communicate to a receiver to determine which object has a higher value for specific properties (e.g., "Which ball has higher elasticity?").
Communication Channel: A discrete bottleneck using Gumbel-Softmax. Agents transmit a short message of categorical symbols.
- Constraint: Senders never see the ground-truth property values; they only receive gradients based on the receiver's task loss.
- Structure: Messages are factored (e.g., $K=2$ positions, $V=5$ vocabulary), allowing for potential positional specialization.

Key Architectural Components

Perceptual Backbones (Frozen):
- DINOv2 (ViT-S/L): Image-based self-supervised features (processes frames independently).
- V-JEPA 2 (ViT-L): Video-based joint-embedding predictive architecture (processes spatiotemporal sequences jointly).
Training Regime:
- Iterated Learning: Receivers are periodically reset (every 40 epochs) to force the sender to develop a protocol that is learnable from scratch by any new receiver (simulating cultural transmission).
- Population Pressure: Senders are trained against a population of 3 receivers simultaneously to ensure universal decodability.
- Multi-Agent Scaling: Experiments range from 2 to 4 agents, where each agent observes a subset of the input frames.

Evaluation Metrics

Positional Disentanglement (PosDis): Measures if specific message positions encode specific properties (high values indicate compositionality).
Holdout Accuracy: Performance on unseen combinations of property values (testing generalization).
Causal Intervention: Zeroing out specific message positions to verify if the disruption is selective to the encoded property.

3. Key Contributions

Emergence of Compositional Protocols for Invisible Physics:
The study demonstrates that agents spontaneously develop positionally disentangled protocols (e.g., Position 0 = Elasticity, Position 1 = Friction) when communicating about latent physical properties. This occurs in 54% of 2-agent runs and 100% of 4-agent runs, significantly outperforming holistic (non-compositional) baselines in generalization.
Perception Dictates Communicability:
Through a controlled $2 \times 2$ factorial comparison, the paper proves that the perceptual backbone determines what can be communicated.
- V-JEPA 2 dominates on collision dynamics (where properties are invisible in single frames) because it encodes temporal velocity differences.
- DINOv2 dominates on ramp physics (where properties are spatially visible) but fails on collision dynamics.
- Crucial Finding: Scaling DINOv2 to match V-JEPA 2's parameter count (ViT-L) or increasing frame count does not close the gap. The advantage is strictly due to video-native pretraining objectives, not model capacity.
Multi-Agent Structure as the Driver:
The paper isolates the cause of compositionality. It is not driven by increased bandwidth or temporal coverage.
- Control: A single agent with matched bandwidth (4x5 vocabulary) fails to develop compositionality (35% success).
- Result: Only multi-agent structure (independent agents compressing partial observations) forces the emergence of compositional codes. With 4 agents, compositionality becomes reliable (100% of seeds).
Causal Addressability and Reusability:
The emergent protocols are not just correlated with properties; they are causally addressable.
- Surgical Ablation: Zeroing a specific message position selectively disrupts the corresponding property prediction while leaving others intact.
- Cross-Property Reasoning: A frozen sender trained on "elasticity comparison" can be used by a new receiver to solve "Is A's elasticity > B's friction?" with 93.8% accuracy, proving the interface is reusable and modular.
Real-World Validation:
The mechanism transfers to real camera footage (Physics 101). Agents achieve 85.6% accuracy on unseen real-world objects. Temporal dynamics contribute +11.2% accuracy over static appearance, and the multi-agent scaling effect (90% compositionality at 4 agents) replicates in real video.

4. Key Results

Compositionality Rates:
- 2 Agents: 54% compositional seeds (PosDis > 0.4).
- 4 Agents: 100% compositional seeds (PosDis $\approx$ 0.999).
Generalization: Compositional seeds generalize significantly better to unseen property combinations than holistic seeds (79.3% vs. 75.9% holdout accuracy; $p=0.011$ ).
Backbone Performance (Collision Task):
- V-JEPA 2 (4 agents): 87.4%
- DINOv2 ViT-S (4 agents): 77.7%
- DINOv2 ViT-L (Scale-matched): 74.6% (Worse than ViT-S, proving scale isn't the factor).
Downstream Utility: Frozen discrete messages retain 94% of the predictive power of raw high-dimensional features for outcome prediction, despite a 25x compression ratio. They support action-conditioned planning with physically correct counterfactual reasoning ( $r=0.780$ ).
Boundary Condition: The mechanism fails on categorical tasks (e.g., CIFAR-100 class identification), where agents memorize class codes rather than composing properties. It succeeds only on continuous property variation.

5. Significance and Implications

For World Model Design: The results provide empirical evidence supporting LeCun's (2022) cognitive architecture, suggesting that discrete latent variables within JEPA-style world models can support compositional, causally addressable reasoning. The discrete bottleneck acts as a structured abstraction layer.
For Multi-Agent Systems: The paper argues that distributing perception across independent agents is a more effective design principle for inducing compositionality than simply increasing the bandwidth of a single agent.
For Evaluation: The authors propose emergent communication as a novel evaluation methodology for foundation models. Unlike linear probes (which check if information exists), communication tests if information can be structured, compressed, and reused in a compositional form.
Physical Understanding: The study confirms that video-native pretraining (V-JEPA) is essential for understanding dynamics-only physics, and that this knowledge can be distilled into interpretable, discrete symbols for downstream planning.

In summary, this paper demonstrates that communication pressure, combined with multi-agent structure and adequate video-based perception, is sufficient to induce agents to develop a compositional language for invisible physical laws, creating a robust, queryable interface for physical reasoning.