💻 computer science

SlotDiT: Object-Centric Representations for Diffusion Transformers

This paper introduces SlotDiT, a text-guided Diffusion Transformer that utilizes object-centric slot-based latent representations to achieve competitive video generation quality and significantly improved task-completion rates in robotic applications compared to traditional VAE-based approaches.

Original authors: Gjergj Plepi, Sven Behnke

Published 2026-09-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gjergj Plepi, Sven Behnke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled to understand the world not just as a blur of pixels, but as a collection of distinct things that move and interact. For years, artificial intelligence systems designed to generate video or plan robot actions have relied on a method that treats every image as a massive grid of colored dots. While this approach produces stunningly realistic pictures, it often fails when a robot needs to figure out how to pick up a cup or slide a block across a table. The problem is that these systems see the world in high-definition detail but lack a clear mental map of the individual objects within it. They know what a scene looks like, but they struggle to understand what is happening inside it. This gap between seeing and understanding has limited the ability of machines to follow simple instructions or plan complex movements in real-world environments.

A team of researchers at the University of Bonn has proposed a different way for machines to see. Instead of forcing an artificial intelligence to process every single pixel of a video, they taught it to break a scene down into a small number of distinct, movable parts. They call these parts "slots." Imagine looking at a busy kitchen counter and instantly recognizing the coffee mug, the toaster, and the fruit bowl as separate entities, rather than a chaotic mess of shapes and colors. This new system, which the researchers named SlotDiT, uses this object-focused approach to guide a powerful video-generating computer model. By focusing on the objects themselves rather than the background noise, the system can predict how a scene will change over time with much greater accuracy, especially when given a simple text instruction like "move the red block to the blue bowl."

The researchers built their system to work in two main stages. First, the computer learns to look at a video frame and separate it into these individual object slots. It identifies the entities in the scene and assigns each one a compact digital representation. Once the scene is broken down, the system uses a text instruction to guide a prediction of what will happen next. Instead of trying to guess the color of every pixel in the future, the model simply predicts how these few object slots will move and change. It then stitches these predictions back together to create a video of the future. The team tested this method against older systems that relied on the traditional, pixel-heavy approach. They ran experiments on four different datasets, ranging from simple computer simulations of tabletop games to complex, real-world footage of robots performing household tasks.

The results showed a clear divide between seeing the world in detail and understanding it structurally. The older systems, which focused on high visual fidelity, often produced videos that looked smooth and realistic to the human eye. However, when asked to follow a specific instruction, these systems frequently failed. They might generate a beautiful video of a block sliding, but the block would end up in the wrong place or fail to interact with the target object as requested. In contrast, the new SlotDiT system, while sometimes producing videos that were slightly less visually perfect, consistently succeeded at the actual task. On a dataset called CLIPort, where a robot must place a specific block into a specific bowl, the new system achieved a success rate of 73.0 percent, far outperforming the next best method, which only reached 50 percent. Even in more complex, real-world scenarios involving household chores, the object-focused system maintained a higher rate of success than its pixel-focused rivals.

Beyond just getting the job done, the new approach proved to be significantly faster and more efficient. Because the system only has to track a handful of object slots instead of thousands of pixels, it requires far less computing power to generate predictions. In their tests, the researchers found that the new system could generate predictions more than five times faster than the standard high-definition models. This speed advantage is crucial for robotics, where a machine needs to think and react in real-time. The study suggests that for robots to be truly useful, they do not necessarily need to see the world in perfect high definition; they need to see it in a way that highlights the objects that matter. By prioritizing the structure of the scene over the fine details of the image, the researchers have shown that machines can become much better at following instructions and planning their own actions.

The work also highlights a surprising truth about artificial intelligence: looking better does not always mean thinking better. The researchers explicitly found that the systems that produced the most visually impressive videos were often the worst at solving the tasks. This indicates that the current standard for judging video generation—how realistic it looks—might be misleading when the goal is to build machines that can interact with the physical world. The study concludes that giving robots an object-centric view of the world is a powerful way to improve their ability to plan and control their movements. While the system still has limitations, such as needing a fixed number of objects to track, the findings offer a promising path forward. It suggests that the future of robotic intelligence may lie not in making machines see more, but in helping them understand what they are seeing.

Technical Summary: SlotDiT: Object-Centric Representations for Diffusion Transformers

Problem Statement

Text-conditioned latent diffusion models have demonstrated significant success in video generation and show promise for robotic applications such as planning, policy learning, and world modeling. However, existing approaches predominantly rely on pixel-level or Variational Autoencoder (VAE)-based latent representations. These representations are typically learned with reconstruction-driven objectives (e.g., Stable Diffusion VAE), prioritizing high visual fidelity over explicit semantic structure. Consequently, the latent space often lacks the structured, object-level decomposition necessary for effective downstream tasks in robotics, where task completion and planning matter more than pixel-perfect reconstruction. While object-centric representations have proven beneficial for dynamics modeling and control, their integration as latent spaces for diffusion-based generative modeling remains unexplored.

Methodology

The authors propose SlotDiT, a text-guided Diffusion Transformer (DiT) that operates within a compact, object-centric slot-based latent space. The framework consists of two primary stages:

1. Learning Slot-Based Representations

The first stage involves learning an object-centric decomposition of the scene.

Encoder: A frozen DINOv2 vision transformer (ViT) extracts visual features from input frames.
Slot Attention: A Slot Attention module iteratively refines a set of latent vectors (slots) by attending to the visual features. This enforces competition among slots to represent distinct scene entities.
Decoder: A two-stage decoder (MLP-based broadcast decoder followed by a CNN-based image decoder) reconstructs the scene from the slots, optimizing for both feature and image reconstruction losses.
Output: Each frame is represented as a set of $N_S$ object-centric slots, where each slot corresponds to a specific entity in the scene.

2. Slot-based Text-Conditioned Diffusion Transformer

The second stage trains the generative model within the learned slot space.

Architecture: SlotDiT utilizes a Diffusion Transformer backbone composed of identical blocks containing bidirectional self-attention, text-to-slot cross-attention, and MLPs.
Conditioning: The model is conditioned on a language instruction (encoded via a T5 encoder) and the observed scene context (initial slot set).
Generation Process: Given a reference image and instruction, the model autoregressively denoises future slot trajectories. It operates on a fixed-length temporal window, receiving context slots and text embeddings, while injecting Gaussian noise into future frames.
Training: The model follows the Diffusion Forcing Transformer (DFoT) framework, utilizing a noise-as-masking paradigm where each frame is corrupted with an independent noise level. The objective is to predict the velocity ( $v$ -prediction) of the denoising process.
Inference: Generation proceeds autoregressively using a DDIM sampler. Classifier-free guidance is employed to improve sample quality and instruction following by combining conditional and unconditional predictions.

Key Contributions

SlotDiT Model: The introduction of a text-guided Diffusion Transformer that operates in an object-centric latent space, enabling structured generative modeling for robotic environments.
Controlled Comparative Study: A systematic investigation comparing slot-based representations against established VAE-based (SD-VAE, ImageVAE, VA-VAE, VideoVAE) and semantics-aligned (DINOv2-based) latent spaces within a unified DiT framework. This isolates the impact of the latent space design while keeping architecture and training procedures constant.
Empirical Validation: Demonstration that object-centric structure serves as a powerful inductive bias, yielding competitive video generation quality while significantly improving task-success rates in text-guided video generation and robot control tasks.

Results

The authors evaluated SlotDiT on four robotic datasets (LanguageTable-Synthetic, CLIPort, LanguageTable-Real, and Bridgev2) covering both simulated and real-world environments.

Task Success vs. Visual Fidelity: A critical finding is the decoupling of visual quality metrics from task success. While VAE-based baselines often achieved superior perceptual scores (LPIPS, FVD, JEDi), they frequently failed to follow instructions or complete tasks. For instance, on CLIPort, DiT + VideoVAE achieved the highest visual fidelity among VAE baselines but only a 57.1% task success rate, whereas SlotDiT achieved an 87.8% task success rate. Similarly, on LanguageTable-Synthetic, DiT + VideoVAE outperformed SlotDiT in visual metrics but achieved a significantly lower task success rate (55.8% vs. 79.7%).
Robot Control: In closed-loop and open-loop robot control evaluations, SlotDiT outperformed all baselines, including object-centric Transformer predictors (TextOCVP) and VAE-based DiTs. On LanguageTable-Synthetic, SlotDiT nearly matched the performance of an oracle using ground-truth future slots, suggesting the generated trajectories preserve essential task-relevant information.
Efficiency: The compact nature of the slot representation (e.g., 10 slots per frame vs. 256 tokens for VAEs) resulted in substantial computational efficiency. SlotDiT achieved a 5.68× speedup in inference throughput compared to the DiT + SD-VAE baseline, primarily due to the reduced token count during the diffusion sampling stage.
Robustness and Controllability: SlotDiT demonstrated robustness to out-of-distribution scene configurations (e.g., increased object counts) and unseen instruction templates. It also exhibited strong controllability, successfully adapting generated trajectories to modified text instructions while maintaining scene consistency.

Significance and Claims

The paper claims that object-centric structure is a powerful inductive bias for diffusion-based generative modeling in robotic environments.

The authors argue that while high-fidelity visual reconstruction is valuable for general video generation, it is not the primary requirement for robotic planning and control. Instead, a latent space that explicitly decomposes scenes into semantic entities (slots) provides a more efficient and effective representation for modeling task-relevant dynamics. The results suggest that standard video generation metrics do not fully capture the utility of a latent space for downstream robotic tasks. By prioritizing structured, object-level semantics over pixel-level fidelity, SlotDiT bridges the gap between generative modeling and practical robotic application, offering a computationally efficient alternative that improves both planning accuracy and task completion rates.