Technical Summary: VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation

1. Problem Statement

Contact-rich manipulation tasks (e.g., wiping surfaces, inserting plugs) rely heavily on local interaction states such as deformation, pressure, slip, and friction. These cues are often temporally sparse, transient, or occluded in visual observations, rendering vision-centric policies unreliable when execution requires tactile-informed adjustments.

Existing visual-tactile policies typically feed tactile observations directly into action prediction but fail to fully exploit tactile information. A primary limitation is the temporal imbalance in information availability: visual observations provide dense, scene-level information across most frames, whereas tactile deformation evolves and provides force feedback only during brief contact phases. Consequently, neural networks trained jointly tend to favor visual evidence, leaving tactile signals underutilized. Furthermore, current approaches often treat tactile prediction indirectly (e.g., via planning modules) rather than coupling it directly with action generation to model the dynamics of contact evolution.

2. Methodology: VT-WAM

The authors propose VT-WAM (Visual-Tactile World Action Model), a unified framework that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a flow matching framework. The architecture consists of three modality-specific experts (Visual, Tactile, Action) connected by novel attention mechanisms.

Core Components

1. Asymmetric Mixture-of-Transformers (MoT) Attention
To address the need for both global context and local contact dynamics while maintaining efficient inference, VT-WAM employs an asymmetric readout strategy:

• Visual Anchor: Action tokens attend only to the first-frame visual tokens to establish global scene context. This enables a "visual-cache" inference mode where future visual prediction is removed to reduce latency.
• Temporal Tactile Dynamics: Action tokens attend to the full tactile sequence to capture contact evolution.
• Implementation: During training, the model optimizes visual, tactile, and action predictions jointly. During inference, future visual tokens are masked out, while the tactile expert continues to predict future deformation, and the action expert attends to this evolving tactile latent sequence alongside the static visual anchor.

2. Contact-Gated Action-Visual-Tactile Attention Guidance (AVTAG)
To prevent the model from defaulting to visual dominance during joint training, VT-WAM introduces an auxiliary training objective.

• Mechanism: AVTAG computes the relative attention weights of action queries toward visual keys ($p_v$) versus tactile keys ($p_t$).
• Loss Function: A training-only hinge ranking loss is applied specifically to contact-phase action tokens (identified by pronounced tactile deformation). The loss penalizes cases where $p_v > p_t$, encouraging the action queries to prioritize tactile evidence when physical interaction is informative.
• Effect: This guides the model to rely on tactile dynamics during contact phases without altering the inference-time architecture.

Training and Inference

• Objective: The model is trained with a joint flow matching objective ($L_{Flow}$) minimizing the error between predicted and target velocity fields for visual, tactile, and action tokens. When enabled, the total loss includes the AVTAG term ($L_{Train} = L_{Flow} + \lambda_{AVTAG}L_{AVTAG}$).
• Inference Modes:
  • Joint Inference: Denoises visual, tactile, and action tokens together (used for analysis).
  • Visual-Cache Inference: Used for real-world control. The model keeps the current visual observation as a fixed anchor and denoises only tactile and action latents, avoiding the computational cost of predicting future visual frames.

3. Key Contributions

1. Formulation of VT-WAM: The authors formulate a Visual-Tactile World Action Model that couples tactile deformation dynamics directly with action prediction through joint visual-tactile-action flow matching, rather than using tactile data merely as an auxiliary input.
2. Architectural Innovations: The introduction of Asymmetric MoT Attention (enabling a visual anchor with full temporal tactile access) and Contact-Gated AVTAG (enforcing tactile reliance during contact phases) to address information imbalance and inference efficiency.
3. Empirical Validation: Extensive evaluation on six real-world contact-rich tasks demonstrates that modeling tactile deformation as interaction dynamics significantly improves success rates compared to visual-only and standard tactile-input baselines.

4. Experimental Results

The authors evaluated VT-WAM on a 7-DoF xArm7 robot equipped with a wrist camera and Xense tactile sensors across two regimes: Surface-Interaction Tasks (Wipe Board, Wipe Vase, Peel Cucumber) and Constrained Insertion Tasks (Insert Plug, Swipe Card, Insert Tube).

• Overall Performance: VT-WAM achieved an average success rate of 71.67%, outperforming the strongest baseline, Fast-WAM (45.00%), by 26.67%. It also surpassed OmniVTLA (35.84%) and RDP (34.17%).
• Task-Specific Gains:
  • Surface-Interaction: VT-WAM achieved 81.67% (vs. 56.67% for Fast-WAM), demonstrating the value of modeling tactile evolution for sustained contact.
  • Constrained Insertion: VT-WAM achieved 61.67% (vs. 33.33% for Fast-WAM), highlighting the utility of tactile dynamics for fine alignment under occlusion or tight constraints.
• Prediction Quality: In joint inference mode, VT-WAM demonstrated superior tactile prediction quality (lower $L_2$ deformation error and higher directional consistency) compared to baseline models like exUMI and UVA.
• Ablation Studies:
  • Tactile Dynamics: Allowing action queries to attend to the full tactile sequence (vs. only the first frame) significantly improved performance (e.g., 70% vs. 40% on Wipe Vase), confirming that temporal evolution is critical.
  • AVTAG: The contact-gated guidance improved success rates further (e.g., 85% vs. 70% on Wipe Vase), proving that explicitly guiding attention to tactile evidence during contact phases is essential for robustness.

5. Significance and Claims

The paper claims that modeling tactile deformation as temporal interaction dynamics is a more effective approach for contact-rich manipulation than treating tactile data as a static policy input. By coupling tactile prediction with action generation, VT-WAM enables the policy to react to local contact evolution (e.g., re-establishing contact after a slip) that is invisible to the camera.

The authors position VT-WAM as a step forward from existing World Action Models (WAMs) that primarily model visual dynamics. They argue that the proposed Asymmetric MoT Attention successfully balances the need for global visual context with the necessity of local tactile dynamics, while AVTAG ensures the model does not ignore sparse but critical tactile signals.

The paper concludes modestly, noting that while VT-WAM focuses on individual-task specialization for precise tactile modeling, multi-task training and scaling laws remain unexplored directions for future research.