Technical Summary: Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio

1. Problem Statement

Generative video foundation models (VFMs) possess strong compositional priors derived from internet-scale data, enabling them to understand temporal coherence, object interactions, and physical plausibility. However, when these models are adapted for robotic control as World-Action Models (WAMs) or Video-Action Models (VAMs), they often lose these compositional generalization abilities after finetuning on robotic action data. The authors term this discrepancy the Video-Action Generalization (VAG) gap.

Existing policies frequently fail under compositional generalization (e.g., recombining seen objects and receptacles into unseen task sequences) because they overfit to visual correlations in the training data rather than leveraging the underlying generative priors. While standard design choices for VAMs (such as joint training vs. two-stage training, or varying noise levels) have been explored, they yield a fragmented performance landscape with no clear emergent explanation for why certain configurations succeed or fail in Out-of-Distribution (OOD) scenarios.

2. Methodology

2.1 Systematic Design Space Analysis

The authors systematically evaluated a class of VAMs that utilize latent features from pre-trained video foundation models (specifically Cosmos-Predict 2.5-2B) and learn a latent inverse-dynamics action head via flow matching. They investigated the design space across two primary axes:

• Training-time configurations: Comparing parameter-efficient adaptation (LoRA) against full finetuning, and joint training (optimizing video and action losses simultaneously) against two-stage training (finetuning the backbone first, then freezing it for action head training).
• Inference-time parameters: Varying the video noise level ($\sigma_v$) at which features are extracted and the prediction horizon ($T$).

The analysis revealed that while In-Distribution (ID) performance was often saturated, OOD performance was highly sensitive to these choices, yet no single configuration consistently closed the gap.

2.2 The Temporal Ratio (TR)

To explain the failure modes, the authors introduced the Temporal Ratio (TR), an attention-based metric quantifying how strongly the action head relies on predicted future latent rollouts relative to the anchored current frame.

• Definition: TR is calculated as the ratio of attention mass assigned by the action head to future frame tokens ($V_+$) versus current frame tokens ($V_0$):
  $$TR = \frac{\sum_{q \in A} \sum_{v \in V_+} A^\ell(q,v)}{\sum_{q \in A} \sum_{v \in V_0} A^\ell(q,v)}$$
  where $A$ represents action tokens and $\ell$ is a specific layer in the action head.
• Physical Interpretation: In the video denoising formulation, the first frame ($t=0$) is injected as a clean, high-fidelity observation anchor at every step. Future frames are denoised from noisy latents and represent the model's "imagined" evolution. A high TR indicates the policy is planning based on these imagined futures; a low TR indicates the policy is reacting to the immediate observation.

2.3 Key Observations on TR

• Predictive Power: TR acts as a direct predictor of compositional generalization capacity. Models with higher TR during planning phases correlate with better OOD success.
• Task Phase Dynamics: TR is not static; it fluctuates based on the task phase. It rises during "planning phases" (e.g., selecting an object or receptacle) where the policy must bind language and goals to future states. It drops during "precise manipulation phases" (e.g., grasping or placement) where the policy must anchor to the current frame for accuracy.
• Failure Mode Diagnosis: The VAG gap often stems from a mismatch: the video backbone may predict a correct compositional future, but the action head ignores it (low TR) and follows a memorized ID trajectory. Conversely, the action head may follow a hallucinated future if TR is high but the video prediction is poor.

2.4 TR-Adaptive Guidance

Based on these findings, the authors propose an inference-time adaptive guidance method that dynamically amplifies compositional video conditioning signals precisely when the policy relies on future rollouts.

• Mechanism: The method computes two counterfactual guidance directions:
  1. Language Guidance: Isolates instruction-dependent action velocity by comparing conditional vs. null-text video features.
  2. Plan Guidance: Isolates long-horizon temporal structure by comparing standard horizon features vs. extended-horizon features.
• Adaptive Scaling: The strength of these guidance signals ($w_{lang}, w_{plan}$) is scaled dynamically based on the current TR. The weight increases when TR rises above an episode baseline (indicating a planning phase) and is capped to prevent over-guidance. This allows the model to leverage compositional reasoning during planning while relaxing guidance to preserve local precision during manipulation.

3. Key Results

3.1 Simulation (LIBERO Benchmark)

Evaluated on the LIBERO suites (Spatial, Object, Goal, Long) and their compositional OOD counterparts:

• Baseline Performance: Prior WAMs and VAMs showed saturated ID performance (>95%) but collapsed in OOD scenarios (often <15% success).
• Design Space Findings: Joint training with LoRA and intermediate video noise levels ($\sigma_v \approx 0.883$) yielded the best unguided OOD performance (55.7% average).
• Impact of TR-Adaptive Guidance: The proposed method improved average OOD success to 59.4%, a >5x improvement over prior VAM baselines, while maintaining ID performance comparable to the best unguided models. Flat guidance (without TR adaptation) degraded ID performance, confirming the necessity of phase-aware guidance.

3.2 Real-World Evaluation (Bimanual YAM)

Evaluated on a bimanual robot performing 24 multi-task scenarios (dishes, snacks, bottles, etc.) with compositional OOD test scenes:

• Performance: The proposed VAM outperformed baselines ($\pi_0$, $\pi_0.5$, Cosmos-Policy).
• Guidance Efficacy: Adaptive guidance increased the average success rate from 71.7% (unguided) to 83.3%.
• Failure of Standard Approaches: Extracting features at the fully noisy step ($\sigma_v=1.0$), as done in prior VAMs, resulted in near-zero success on real-world tasks, highlighting the need for structured, partially denoised latents.

4. Significance and Claims

The paper claims that the VAG gap is not merely a result of insufficient data or model capacity, but a mechanistic issue regarding how the action head routes information between the current observation and predicted futures.

• Diagnostic Tool: The Temporal Ratio provides a runtime, interpretable metric to diagnose whether a policy is effectively utilizing its generative video priors.
• Mechanism for Improvement: The work demonstrates that compositional generalization requires a dual condition: (1) the video backbone must predict a useful future, and (2) the action head must actively rely on that future during planning phases.
• Practical Mitigation: TR-Adaptive Guidance offers a practical, inference-time intervention that leverages this intrinsic attention pattern to improve OOD generalization without retraining the model or sacrificing local manipulation precision.

The authors conclude that their framework clarifies the behavior of VAMs and provides a pathway to mitigating the OOD generalization gap in robotic control, though they acknowledge limitations regarding the physical plausibility of the underlying video model and the added inference cost.