Technical Summary: DiLA: Disentangled Latent Action World Models

1. Problem Statement

Latent Action Models (LAMs) aim to learn world models from unlabeled video by inferring abstract actions between consecutive frames, serving as surrogates for explicit control signals. However, existing LAMs face a fundamental trade-off, termed the "LAM Trade-off," between action abstraction and generation fidelity.

• The Dilemma: To ensure actions are transferable and abstract, models typically impose strong predictive bottlenecks (e.g., Vector Quantization or variational bottlenecks). While these encourage abstraction, they often disrupt the intrinsic manifold of the latent action space, leading to over-simplification and degraded video generation quality. Conversely, relaxing these bottlenecks improves generation but results in entangled representations where latent actions capture irrelevant visual details rather than pure dynamics.
• Current Limitations: Most existing approaches prioritize abstraction at the expense of generation quality, often adopting disjoint, two-stage training paradigms where the Forward Dynamics Model (FDM) is discarded in favor of separate, pre-trained diffusion models. Others rely on explicit separation of motion (e.g., optical flow) from content, but fail to jointly optimize these processes within a single self-supervised framework.

2. Methodology: DiLA

The authors propose DiLA (Disentangled Latent Action), a novel world model that resolves the LAM Trade-off via content-structure disentanglement. The core insight is that disentanglement and latent action learning are co-evolving: the predictive bottleneck inherent in latent action learning acts as a driving force for disentanglement.

Architecture

DiLA processes video sequences through two specialized pathways, utilizing a DINOv2 encoder for feature extraction and a pre-trained RAE decoder for visualization.

1. Structure Pathway (Dynamics):

   • Goal: Isolates motion-related spatial layouts (positions, shapes) to learn latent actions invariant to visual content.
   • Mechanism: A structure encoder compresses tokens into structure embeddings ($s_t$). An Inverse Dynamics Model (IDM) computes abstract latent actions ($z_t$) from the temporal difference of these embeddings ($\Delta s_t$).
   • Constraint: A strict information bottleneck forces the IDM to predict only structural dynamics. The Forward Dynamics Model (FDM), implemented as a lightweight Spatial-Temporal Transformer, predicts the next structural state ($\hat{s}_{t+1}$) based on the current state and latent action.
   • Driving Force: The bottleneck compels the model to distill motion dynamics into $z_t$ while offloading static visual details to the content pathway, as high-entropy content details cannot be effectively compressed into the low-dimensional latent action without impeding prediction accuracy.

2. Content Pathway (Appearance):

   • Goal: Extracts and memorizes temporal-invariant visual features (colors, textures, landmarks).
   • Mechanism: A content encoder compresses tokens into embeddings ($c_t$), which are processed by a Mamba architecture to aggregate historical context. This mimics Slow Feature Analysis, accumulating static features as they are revealed.
   • Role: It maintains visual details for high-fidelity generation and handles belief updates regarding occluded backgrounds, allowing the structure pathway to specialize strictly in physical dynamics.

3. Fusion Decoder:

   • Combines the predicted structural state ($\hat{s}_{t+1}$), the content memory ($c^{mem}_t$), and the initial visual embedding ($e_0$) to reconstruct the target visual embedding ($\hat{e}_{t+1}$).
   • Conditioning on $e_0$ is crucial for supplying high-frequency details lost during compression.

Training Objectives

DiLA is trained using a self-supervised teacher-forcing paradigm on video sequences without ground-truth action labels. The total loss ($L_{total}$) is a weighted combination of:

• Visual Latent Prediction ($L_e$): Reconstructing the target embedding.
• Structure Prediction ($L_s$): Minimizing the error in predicting the next structural state.
• Latent Action Consistency ($L_z$): Ensuring consistency between forward and backward inferred actions.
• Regularization ($L_{reg}$): Enforces group symmetry (inverse temporal transitions yield opposite action vectors), constrains the vector norm to maintain a compact manifold, and regularizes variance to maximize information entropy.

3. Key Contributions

1. Unified Framework: DiLA is the first disentangled latent action world model that reconciles the trade-off between latent action abstraction and generation fidelity via content-structure disentanglement.
2. Co-evolution Mechanism: The paper demonstrates that the predictive bottleneck acts as a driving force to isolate spatial layouts from appearance, and conversely, this separation facilitates the learning of highly abstract latent actions.
3. Performance: Extensive experiments show DiLA achieves superior results in video generation quality, cross-embodiment action transfer, and visual planning compared to state-of-the-art baselines.
4. Interpretability: The model learns a structured, interpretable latent action space that supports downstream visual planning and effectively separates motion dynamics from static appearance.

4. Experimental Results

Experiments were conducted on large-scale datasets including Something-Something-v2 (SSv2), RT-1 (robotics), RECON (outdoor navigation), and LoopNav (Minecraft).

• Video Generation Quality: On SSv2 and RT-1, DiLA outperformed leading baselines (LAPA, Moto, AdaWorld, Villa-X) in both SSIM (structural consistency) and LPIPS (perceptual similarity). Notably, removing the content pathway ("DiLA w/o content") resulted in significantly lower generation quality, validating the necessity of the dual-pathway architecture.
• Action Transfer: DiLA successfully transferred latent actions across different embodiments (e.g., human-to-robot) and domains (virtual-to-real navigation). It maintained high generation quality while transferring complex semantic actions like "picking up" objects across viewpoint changes.
• Disentanglement Validation:
  • Rebinding Experiments: By fusing structure from one video with content from another, DiLA generated sequences that preserved the source's spatial dynamics while inheriting the reference's appearance.
  • Motion Isolation: Freezing the structure embedding while evolving the content memory resulted in a completely static sequence, confirming that the content pathway encodes no motion information.
• Ablation Studies:
  • Removing the IDM/FDM components caused the model to fail at disentanglement, with structure embeddings retaining redundant content details.
  • Replacing the continuous bottleneck with discrete (VQ) or Gaussian (VAE) priors led to inferior generation quality and slower convergence, suggesting that rigid priors disrupt the intrinsic low-dimensional manifold of latent actions.
• Visual Planning: In the VP2 benchmark, DiLA adapted for Model Predictive Control (MPC) outperformed the baseline AdaWorld, particularly in tasks like "Push Button," demonstrating its utility in robotic control.

5. Significance and Claims

The paper claims that DiLA establishes a unified framework that simultaneously achieves high-level action abstraction and high-fidelity generation, advancing the frontier of self-supervised world model learning.

• Resolution of the Trade-off: The authors argue that disentangling structure from content is the key to resolving the tension between abstraction and generation. The predictive bottleneck is not merely a constraint but a mechanism that drives the model to learn pure transition dynamics.
• Robustness: The learned latent action space forms a continuous, semantically structured manifold that supports transfer across different embodiments and environments without requiring explicit supervision.
• Limitations: The authors modestly acknowledge that the inherent abstraction of latent actions may sacrifice fine-grained control precision, which could lead to instability in downstream video-to-control tasks. They also note that the current disentanglement separates spatial layouts from visual details but does not yet fully support the decoupling of multi-object dynamics. Additionally, the approach remains susceptible to autoregressive compounding errors in long-horizon video generation.

In summary, DiLA posits that by leveraging the predictive bottleneck to drive content-structure disentanglement, it is possible to learn world models that are both abstract enough for planning and transfer, and detailed enough for high-fidelity generation.