🤖 machine learning

Better Slots, Better Worlds: Representation Quality & Robustness in Object-Centric World Models

This paper demonstrates that in object-centric world models, planning success and robustness under distribution shifts are primarily driven by high-quality, well-bound object representations and the use of pretrained features, rather than auxiliary inputs or specific inductive biases previously relied upon.

Original authors: Shukrullo Nazirjonov, Sai Prasanna, Anna Manasyan, Georg Martius

Published 2026-08-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shukrullo Nazirjonov, Sai Prasanna, Anna Manasyan, Georg Martius

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a game, like pushing a block into a specific spot. You can't stand there holding its hand every second; instead, you want the robot to learn the rules of the game by watching a pile of old video recordings of other robots playing. This is the world of world models: a way for AI to build a mental map of how the world moves and changes just by watching, so it can plan ahead and solve new problems without needing a human to explain every single step.

But here's the tricky part: how does the robot actually "see" the world in those videos? Does it just look at the whole picture as one big, blurry mess of pixels? Or does it learn to pick out the individual characters—the robot arm, the block, the goal—and treat them as separate actors with their own stories? This paper asks a very specific question: Does it matter if the robot learns to separate these objects clearly? Some researchers think that if a robot can neatly sort the world into "object slots" (like putting different toys into different boxes), it will be a much better planner. Others wonder if that sorting is actually necessary, or if the robot just needs to be really good at guessing what happens next, no matter how messy the picture looks. This matters because if we can figure out the secret sauce for better planning, we can build robots that are smarter, learn faster, and don't get confused when the lighting changes or the objects look different.

The Great Slot Sort-Out

The authors of this paper decided to settle the debate with a controlled experiment. They built a robot planner that uses "object-centric" representations—basically, a system that tries to break a video scene down into a set of "slots," where each slot is supposed to hold the information for one specific object. They compared this approach against two other ways of seeing the world: one that looks at the whole scene as a single blob, and another that looks at the scene as a grid of tiny, frozen puzzle pieces.

Their first discovery was all about quality. They tested the robot planners using different "slot sorters" that were better or worse at separating objects. Imagine one sorter that neatly puts the red block in one box and the blue block in another, and another sorter that accidentally mixes the red block's color into the blue block's box. The researchers found that planning success goes up as the sorting gets cleaner. If the robot can clearly tell the objects apart, it plans better. However, there's a catch: once the sorting is really, really good, making it even more perfect doesn't help much. It's like having a map that is 99% accurate; making it 99.9% accurate doesn't suddenly make you a faster driver.

The second finding was a bit of a surprise. Previous methods used to rely on extra "crutches" to help the robot plan, like giving it a sense of its own body position (proprioception) or hiding parts of the video to force it to guess what was missing. The authors found that if the robot has a really good slot sorter, it doesn't need these crutches anymore. The extra inputs and training tricks were only there to compensate for bad sorting. Once the robot could clearly see the objects, it could plan just as well without the extra help.

Finally, they tested robustness: what happens when the world changes in ways the robot hasn't seen before? They changed the colors of the blocks, the size of the objects, and even the background. The results were clear: the robot using the high-quality object slots was the most resilient. It handled the changes almost as well as a robot that used a very advanced, pre-trained "frozen" vision system (which the researchers call DINO-WM). However, the robot that tried to plan using a single, global view of the whole scene (without separating objects) fell apart completely when the scene changed.

The Takeaway

The paper suggests that the key to a smart, adaptable robot planner isn't just having a complex brain; it's about having a clear way of seeing. Better "slots" (clean object separation) lead to better planning, but only up to a point. Once the objects are well-defined, the robot doesn't need extra tricks to function. Most importantly, seeing the world as distinct, separate objects makes the robot much tougher when the environment changes, far more so than trying to understand the whole scene as one giant, unseparated image. The authors note that while their results are strong in these specific simulations, the real world might have even more surprises, but the path forward seems clear: teach the robot to sort its toys first, and the planning will follow.

Technical Summary: Better Slots, Better Worlds

Problem Statement

World models enable agents to learn environment dynamics from offline trajectories to plan toward arbitrary goals. While object-centric (OC) representations are theoretically promising for improving sample efficiency and generalization by decomposing scenes into causal, compositional structures, their practical utility in planning remains unverified. Prior object-centric world models (OCWMs) suffer from two critical gaps:

Untested Representation Quality: Existing OCWMs treat the slot encoder as a fixed component, often selected based on unsupervised metrics (e.g., FG-ARI, mBO) that reward clean object masks but do not necessarily correlate with downstream planning success.
Unverified Generalization: It is unclear whether the inductive bias of object-centricity actually translates to robustness under distribution shifts (OOD) compared to scene-centric models, as prior evaluations have largely been confined to in-distribution settings.

Methodology

The authors conduct a controlled study of OCWMs for visual model-predictive control (MPC) on two physical manipulation environments: 2D PushT and 3D OGBench-Cube. The study isolates three specific research questions by varying representation quality and architectural components while holding the dynamics model and planner constant.

Experimental Setup

Baseline Framework: The study builds upon C-JEPA, an OCWM that encodes frames into object slots using VideoSAUR and predicts future slots via a transformer dynamics module. C-JEPA relies on auxiliary mechanisms: a masked-slot prediction objective and auxiliary proprioception inputs.
Proposed Model: The authors introduce SlotContrast-WM, replacing VideoSAUR with SlotContrast (a framework using Slot Attention with temporal contrastive objectives) and updating the feature extractor to DINOv3. Crucially, the default SlotContrast-WM configuration removes the slot-masking objective and proprioception tokens to test their necessity.
Baselines: The OCWMs are compared against scene-centric models:
- DINO-WM: Uses frozen DINOv2 patch features.
- LeWM: Uses a global CLS token from an end-to-end trained ViT.
Metrics:
- Representation Quality: Foreground Adjusted Rand Index (FG-ARI) and mean Best Overlap (mBO) on video sequences.
- Planning: Task success rate (SR) relative to a random policy baseline.
- Robustness: Performance under visual (color, scale, shape) and dynamics shifts.

Key Experimental Axes

Quality Sweep: Training SlotContrast-WM on intermediate checkpoints of the SlotContrast encoder to correlate slot quality metrics with planning success.
Component Ablation: Sweeping the number of masked slots and the presence of proprioception tokens on both high-quality (SlotContrast) and low-quality (VideoSAUR) encoders.
Distribution Shift: Evaluating zero-shot planning performance under unseen variations in object appearance, scale, shape, and scene-level factors (background, lighting, camera angle).

Key Results

1. Slot Quality Correlates with Planning Success

There is a strong positive correlation between unsupervised slot-quality metrics and planning success.

On PushT, planning success tracks the emergence of object separation (Pearson $r = 0.96$ for FG-ARI, $r = 0.94$ for mBO).
Saturation Effect: Gains saturate at high slot quality; once slots are well-bound, further improvements in mask quality yield diminishing returns for planning.
On OGBench-Cube, the correlation exists but saturates earlier, likely because the large robotic arm dominates the segmentation metrics while the small, task-relevant cube contributes less.

2. High-Quality Slots Render Auxiliary Mechanisms Unnecessary

The study challenges the necessity of the auxiliary mechanisms (masking and proprioception) found in prior OCWMs like C-JEPA.

SlotContrast-WM (high-quality slots) achieves 84.7% success without proprioception or masking.
Adding proprioception to this configuration provides negligible gain (+0.6 pp).
Conversely, VideoSAUR (low-quality slots) relies heavily on these mechanisms; without proprioception, masking degrades performance significantly. When paired with proprioception, VideoSAUR reaches 85.3%, suggesting these mechanisms act as compensatory shortcuts for weak representations rather than intrinsic drivers of performance.

3. Robustness to Distribution Shifts

OCWM vs. Scene-Centric: The OCWM (SlotContrast-WM) degrades the least under distribution shifts.
The Role of Pretraining: DINO-WM (scene-centric but using frozen pretrained features) remains comparably robust to the OCWM. In contrast, LeWM (end-to-end trained scene-centric) degrades substantially, particularly under scene-level shifts (e.g., background color changes).
Conclusion: Robustness appears driven by the use of frozen pretrained visual representations (whether object-centric or scene-centric) rather than the object-centric inductive bias alone. Both models fail under geometric shape changes that alter contact dynamics.

Significance and Claims

The paper claims to isolate the specific factors that make object-centric world models effective for planning:

Representation Quality is Dominant: Planning success is tightly coupled with the quality of the object-centric representation. Once slots are well-bound, the model does not require complex auxiliary objectives or proprioceptive inputs to perform well.
Pretraining Drives Robustness: The robustness of OCWMs under distribution shift is largely attributed to the use of frozen, pretrained visual features (e.g., DINO). This suggests that the "object-centric" advantage in robustness may be secondary to the quality of the underlying visual foundation.
Refining Inductive Biases: The study demonstrates that for manipulation tasks, a sufficiently high-quality object-centric representation can simplify the world model architecture by removing redundant inductive biases previously thought necessary.

Limitations

The authors note that unsupervised slot metrics (FG-ARI, mBO) are less informative when task-relevant objects are small relative to the scene (as seen in OGBench-Cube), motivating the need for task-aware quality measures. Future work is suggested to test these findings in more diverse environments with varying object counts, scales, and dynamics.