Technical Summary: TextOCVP

Problem Statement

Autonomous agents require the ability to understand and forecast future scene states to plan effectively in complex environments. While object-centric models with structured latent spaces have shown promise in modeling object dynamics and interactions, they face significant limitations. Existing approaches often struggle to scale beyond simple synthetic datasets, fail to integrate external guidance (such as natural language), and lack mechanisms to control predictions based on high-level semantic intent. Furthermore, current methods often rely on holistic scene representations that ignore scene compositionality, making them less robust to novel configurations and harder to interpret.

Methodology

The authors propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. The architecture decomposes the prediction process into three main stages:

1. Scene Parsing (Object-Centric Decomposition):
   The model parses an observed reference frame ($X_1$) into a set of permutation-invariant object representations called slots ($S_1$). The paper implements two variants based on different decomposition backbones:

   • TextOCVP$^{SAVi}$: Uses the SAVi framework (Kipf et al., 2022) with a CNN feature extractor and Slot Attention.
   • TextOCVP$^{DINO}$: Extends the DINOSAUR framework (Seitzer et al., 2023) using a DINOv2 vision transformer for feature extraction, better suited for complex real-world scenes.
     In both variants, Slot Attention iteratively updates slots based on visual features, encouraging them to compete to represent distinct objects.

2. Text-Conditioned Object-Centric Predictor:
   A transformer-based predictor forecasts future object states ($\hat{S}_{2:T+1}$) conditioned on the initial slots and a text description ($C$).

   • Text Encoding: The text caption is encoded into token embeddings using a transformer encoder (e.g., T5).
   • Architecture: The predictor mirrors a transformer decoder. It employs self-attention to model spatio-temporal relations between object slots and a text-to-slot cross-attention mechanism. This cross-attention explicitly integrates textual guidance into the slot representations, allowing the model to align specific objects with described actions or attributes.
   • Autoregression: The model predicts future slots autoregressively. A residual connection from the current slots to the predicted next slots is used to improve temporal consistency.

3. Video Rendering:
   The predicted slots are decoded back into video frames ($\hat{X}_{2:T+1}$).

   • The SAVi variant uses a CNN-based Spatial Broadcast Decoder to render object images and masks, which are combined via a weighted sum.
   • The DINO variant uses a two-stage decoder: an MLP-based Spatial Broadcast Decoder reconstructs patch features, followed by a CNN decoder to generate the final high-resolution frames.

Training Strategy:
The model is trained in two stages. First, the scene parsing and rendering modules are trained to reconstruct video frames (minimizing reconstruction loss). Second, the text-conditioned predictor is trained on paired video-text data using a combined loss of image reconstruction error ($L_{Img}$) and slot alignment error ($L_{Slot}$), employing autoregressive training rather than teacher forcing to better model long-term dynamics.

Key Contributions

• TextOCVP Architecture: A novel object-centric model that integrates textual guidance into the video prediction process via a text-to-slot attention mechanism, enabling the forecasting of future states aligned with natural language instructions.
• Structured Latent Space: The use of slot representations allows for precise control over the forecasting process, disentangling object dynamics from the background and other objects.
• Performance and Robustness: The model demonstrates superior performance compared to existing text-conditioned video prediction baselines, particularly in scenes with multiple moving objects. It exhibits strong robustness to novel scene configurations, including varying numbers of objects and unseen colors.
• Interpretability and Controllability: The structured latent space provides interpretability, revealing how textual tokens influence specific object representations. The model allows for fine-grained controllability, where modifying the text instruction changes the predicted motion of specific objects while maintaining scene consistency.

Experimental Results

The authors evaluated TextOCVP on two datasets: CATER (synthetic 3D object motion) and CLIPort (robot manipulation tasks).

• Quantitative Performance: TextOCVP outperformed several baselines, including MAGE, SEER, and non-object-centric variants (Non-OC), across standard metrics (PSNR, SSIM, LPIPS, JEDi). Notably, it achieved the best scores on perceptual metrics (LPIPS and JEDi), indicating higher visual quality and motion realism.
• Generalization: In robustness tests involving unseen colors and increased object counts (8 objects vs. 6 in training), TextOCVP showed significantly smaller performance degradation compared to holistic models like MAGEDINO.
• Controllability: Qualitative and quantitative analyses confirmed that the model can successfully adapt predictions to diverse instructions, such as changing target objects, actions, or the number of moving entities.
• Efficiency: TextOCVP demonstrated faster inference times (approx. 40% lower latency) compared to diffusion-based or token-heavy autoregressive baselines like MAGE, due to operating on a compact set of object slots rather than dense spatial tokens.

Significance and Claims

The paper claims that TextOCVP represents a significant step toward controllable object-centric manipulation in simulated robotic environments. By combining structured object-centric representations with language guidance, the model addresses the scalability and controllability limitations of previous object-centric methods.

The authors emphasize that the structured latent space offers:

1. Precise Control: The ability to align language instructions with specific objects for accurate, per-object motion forecasting.
2. Robustness: Superior generalization to novel scene compositions compared to holistic scene representations.
3. Interpretability: The transparency of the attention mechanisms allows for understanding how textual cues drive specific object dynamics.

The work suggests that such structured, language-guided prediction is a promising foundation for more efficient planning, reasoning, and decision-making in autonomous systems, though the authors acknowledge current limitations regarding visual artifacts and long-term temporal consistency in complex, real-world scenarios.