Beyond Task Success: Behavioral and Representational Diagnostics for WAM and VLA

Technical Summary: Beyond Task Success: Behavioral and Representational Diagnostics for WAM and VLA

Problem Statement
Vision-language-action (VLA) policies have become a dominant paradigm for general-purpose robot manipulation, mapping observations and instructions directly to actions. However, most VLAs are fundamentally reactive predictors that do not explicitly model how scenes evolve. World-Action Models (WAMs) have emerged as an alternative, coupling action generation with future prediction (e.g., imagining future states or predicting future observations). While prior work has compared WAMs and VLAs based on final task success rates, it remains unclear whether future prediction in WAMs leads to behaviorally meaningful improvements beyond simply increasing success rates. Specifically, it is unknown if WAMs merely add a prediction module or if they fundamentally change robot behavior (e.g., motion smoothness, target selectivity) and internal representations in ways that are actionable for control.

Methodology
The authors introduce a model-agnostic diagnostic framework that evaluates WAMs and VLAs through two complementary lenses: behavioral rollout diagnostics and sparse-autoencoder (SAE) feature-space analysis.

1. Behavioral Rollout Analysis:
   Instead of relying solely on success rates, the framework measures:

   • Action Dynamics: Smoothness (delta, acceleration, jerk), chunk-boundary consistency, and low-motion failure rates (to distinguish smooth failures from successful control).
   • Object Interaction: Target-object progress (final and accumulated), progress reversals, and distractor stability (unintended motion of non-target objects).
   • Runtime Cost: Inference latency, throughput, and GPU memory usage.

2. Feature-Space Analysis (SAEs):
   To characterize internal representations, the authors train Sparse Autoencoders (SAEs) on latent activations (visual, future, and action streams) from various models. They define a set of metrics to classify features into three categories:

   • Memorized: Specific to particular episodes or objects.
   • Reactive-General: Describes the current observation or execution phase (e.g., grasping).
   • Predictive-General: Provides evidence about future states or actions (e.g., high future consistency, horizon stability, or action predictiveness).
   • Key Metrics: Future Consistency Score (FCS), Horizon Stability (HS), and Action Predictiveness (AP) are introduced to specifically detect future-oriented structure.

Experimental Setup
The study evaluates 7 policies across two benchmarks (LIBERO and RoboTwin2.0):

• VLAs: $\pi0$, $\pi0.5$, X-VLA.
• WAMs:
  • Joint/Parallel: Cosmos Policy (predicts future and action together).
  • Sequential: LingBot-VA (imagines future states before acting).
  • Auxiliary/Training-only: FastWAM, VLA-JEPA (use future prediction only during training).

Key Results

1. Success Rates Mask Behavioral Differences:
   On LIBERO, where success rates are high for most models, behavioral diagnostics reveal significant differences. WAMs generally produce smoother control (lower acceleration, jerk, and boundary jumps) and exhibit better object-level behavior, including stronger target progress and reduced disturbance to distractor objects.

2. Architecture Dependence:

   • Sequential WAMs (LingBot-VA): Show the clearest predictive structure in SAE analysis (highest percentage of predictive features in future streams) and achieve the best object-level metrics (lowest wrong-object disturbance, highest progress AUC).
   • Joint WAMs (Cosmos): Exhibit strong object selectivity and low distractor disturbance but encode future information in a more entangled manner (high memorized/reactive features rather than distinct predictive features).
   • Auxiliary WAMs (FastWAM, VLA-JEPA): Show mixed results. FastWAM compresses future information, resulting in behavior closer to direct VLAs with higher distractor disturbance. VLA-JEPA retains more predictive structure and cleaner object interaction than FastWAM.
   • Direct VLAs: Dominate in reactive features but show low predictive structure. They often exhibit higher progress reversals and distractor disturbance compared to WAMs.

3. Runtime Trade-offs:
   WAMs incur a non-trivial deployment cost. Sequential WAMs (LingBot-VA) have the highest inference latency due to explicit future imagination. Joint WAMs are faster than sequential but slower than direct VLAs. Auxiliary WAMs are more efficient but may sacrifice the behavioral benefits of explicit future modeling.

4. Feature-Behavior Correlation:
   SAE analysis reveals that models with higher proportions of predictive-general features (particularly in future streams) correlate with better target-object selectivity and smoother execution. Removing predictive SAE statistics from behavioral probes significantly degrades the prediction of task-level outcomes (success, transport, target progress) but has mixed effects on low-level smoothness metrics.

Significance and Claims
The paper argues that task success alone is an insufficient metric for evaluating robotic policies. The authors claim that:

• WAMs offer behavioral advantages beyond success rates, specifically in motion smoothness, target selectivity, and distractor stability.
• These behavioral improvements are linked to the presence of future-oriented internal representations. Sequential WAMs expose the clearest predictive structure, while joint and auxiliary designs vary in how they preserve or compress this information.
• There is a fundamental trade-off between the actionability of future representations and inference efficiency. While explicit future modeling improves control quality, it increases computational cost.
• Future WAM design should focus on preserving actionable future representations (e.g., via lightweight inference-time imagination or latent prediction) rather than treating future prediction solely as a training auxiliary loss, to balance performance with deployability.

The authors modestly note that their evidence regarding the link between representations and behavior is associative rather than causal, and their findings are currently limited to simulation benchmarks and public checkpoints. They emphasize that future work must address how to make future prediction efficient enough for closed-loop control without losing its behavioral benefits.