Tactile Modality Fusion for Vision-Language-Action Models

1. Problem Statement

Vision-Language-Action (VLA) models have achieved significant success in generalizing robot policies through semantic reasoning. However, these models predominantly rely on visual perception, which is insufficient for contact-rich manipulation tasks (e.g., peg-in-hole insertion, cable plugging).

• Limitations of Vision: Vision alone cannot capture critical physical dynamics such as contact forces, surface friction, compliance, and shear, especially when occlusion occurs or millimeter-scale adjustments are required.
• Limitations of Existing Fusion Methods: Current attempts to integrate tactile signals into VLAs often use feature concatenation (appending tactile tokens to the input sequence) or require large-scale multimodal pretraining. These approaches increase computational costs, extend sequence lengths, risk degrading performance as context grows, and often require training separate tactile encoders, making them inefficient for post-training adaptation.

2. Methodology: TacFiLM

The authors propose TacFiLM, a lightweight modality fusion approach that integrates tactile signals into pretrained VLA models without increasing input token counts or requiring extensive retraining.

Core Architecture

• Base Model: The method builds upon OpenVLA-OFT, which utilizes a fused SigLIP and DINOv2 visual backbone, an MLP projector, and a decoder-only LLaMA 2 7B language model.
• Fusion Mechanism (FiLM): Instead of concatenating tactile tokens, TacFiLM uses Feature-wise Linear Modulation (FiLM).
  • Process: Tactile images are encoded by a pretrained tactile representation model (e.g., T3 or Sparsh) to produce a pooled embedding ($z$).
  • Conditioning: This embedding is projected via an MLP to generate scale ($\gamma$) and shift ($\beta$) parameters.
  • Application: These parameters are applied to the intermediate visual features ($F^n$) within the Vision Transformer (ViT) blocks of the VLA backbone before the multi-head self-attention layers.
  • Formula: $\text{FiLM}(F^n | \gamma, \beta) = F^n \odot (1 + \gamma) + \beta$.
• Training Strategy: The approach employs parameter-efficient fine-tuning (LoRA). The majority of the VLA backbone is frozen, and only the FiLM conditioning layers and LoRA adapters are trained. This preserves the pretrained visual-language priors while injecting tactile sensitivity.

Tactile Encoders

The method is agnostic to the specific encoder but evaluates two pretrained frameworks:

1. T3: A framework with sensor-specific encoders and a shared transformer trunk.
2. Sparsh: A self-supervised learning framework (using MAE, IJEPA, or DINO objectives).

• Selection: Experiments identified Sparsh-DINO as the superior encoder for downstream policy performance.

3. Key Contributions

1. Novel Fusion Architecture: Introduction of TacFiLM, which conditions visual features on tactile embeddings via FiLM, avoiding token sequence expansion.
2. Efficiency: A post-training fine-tuning paradigm that leverages pretrained tactile representations, eliminating the need for task-specific encoder training or massive multimodal pretraining.
3. Empirical Validation: Comprehensive real-robot experiments demonstrating that FiLM-based fusion outperforms concatenation-based baselines in success rates, execution time, and force control.

4. Experimental Results

The authors evaluated the method on 700+ rollouts across diverse insertion tasks (Circle, Square, Pentagon pegs, USB/HDMI cables) with varying clearances (2mm and 3mm).

Performance Metrics

• Success Rate: TacFiLM achieved 100% success on in-distribution 3mm clearance tasks and maintained 86.67% on out-of-distribution (OOD) tasks, significantly outperforming the vision-only baseline (OpenVLA-OFT) and the concatenation baseline (TactileConcat).
• Direct Insertion: TacFiLM showed a massive improvement in "direct insertions" (successful on the first attempt without recovery adjustments). For example, on the 3mm Circle-Peg task, TacFiLM achieved 36.67% direct insertions compared to 3.33% for the vision-only baseline.
• Force Stability: TacFiLM significantly reduced the average maximum interaction force. On the 3mm Circle-Peg task, it exerted 7.64 N compared to 14.94 N for the vision-only model, indicating better sensitivity to contact dynamics.
• Execution Time: Tasks were completed faster due to fewer recovery attempts.

Robustness and Ablation

• Visual Degradation: Under dimmed lighting (80% reduction) and partial frame freezing (50% updates), TacFiLM maintained 100% success, whereas the vision-only baseline dropped to 73.33%. This proves tactile signals compensate for unreliable visual inputs.
• Integration Depth: Ablation studies showed that applying FiLM to all ViT blocks yielded the best overall balance, though applying it to early blocks improved direct insertion rates slightly.
• Generalization: TacFiLM generalized well to OOD tasks (e.g., HDMI cable plugging), improving success rates by 50% over the next-best baseline.

5. Significance

• Bridging the Gap: TacFiLM effectively bridges the gap between high-level semantic reasoning (VLAs) and low-level physical interaction (tactile control) without sacrificing the efficiency of large language models.
• Safety and Efficiency: By reducing interaction forces and completion times, the method makes robotic manipulation safer and more efficient, crucial for real-world deployment in unstructured environments.
• Scalability: The lightweight nature of the approach (no token expansion, frozen backbone) suggests a scalable path for integrating other sensory modalities (e.g., proprioception, audio) into foundation models without prohibitive computational costs.

In conclusion, TacFiLM demonstrates that feature-level conditioning is a superior strategy to token concatenation for integrating tactile feedback into VLA models, resulting in more robust, force-aware, and efficient robotic manipulation policies.