Technical Summary: A Causality-aware Infer-diagnose-refine Framework for Test-time Modality Adaptation in VLA Models

1. Problem Statement

Vision-Language-Action (VLA) models predict sequential robot actions based on language instructions, visual observations, and proprioceptive states. While effective, a critical challenge remains in how these models fuse modalities during execution. Robot manipulation involves dynamic phases (e.g., long-distance movement vs. close-range interaction) where the importance of visual observations fluctuates over time.

Existing data-driven VLA models typically rely on static, learned multimodal fusion, which may not adapt to these dynamic shifts. Recent studies indicate that visual importance varies across environments, model architectures, and specific motion-transition phases. However, estimating and exploiting this dynamic visual importance for frozen VLA models at test time—without retraining—remains an open problem. Current test-time adaptation methods often target specific failure modes (e.g., language-conditioning collapse) rather than providing a general mechanism to dynamically adjust modality reliance based on causal effects.

2. Methodology: The IDR Framework

The authors propose Infer-Diagnose-Refine (IDR), a model-agnostic, training-free framework designed to refine action predictions at test time by estimating the causal effects of visual observations. The framework operates in three stages:

A. Infer (Counterfactual Inference)

The framework constructs counterfactual scenarios to decouple multimodal inputs. Given a frozen VLA model $\pi_{\theta^*}$ and inputs $(v_t, s_t, l)$ at timestep $t$:

1. Factual Action: Compute the base action $a_{base,t} = \pi_{\theta^*}(v_t, s_t, l)$.
2. Counterfactual Actions: Apply zero-padding interventions to isolate modalities:
   • $a_{no\_img,t} = \pi_{\theta^*}(0_v, s_t, l)$ (Visual observation masked).
   • $a_{no\_prop,t} = \pi_{\theta^*}(v_t, 0_s, l)$ (Proprioceptive state masked).
     This allows the model to infer how the action would change if a specific modality were absent.

B. Diagnose (Causal Effect Quantification)

The framework diagnoses the dynamic importance of visual observations by measuring the distributional shift between factual and counterfactual outputs:

1. Deviation Calculation: Compute action deviations $\Delta_{img,t} = a_{base,t} - a_{no\_img,t}$ and $\Delta_{prop,t} = a_{base,t} - a_{no\_prop,t}$.
2. Magnitude Estimation: Quantify the causal effect magnitude using the $L_2$ norm:
   • $E_{img,t} = \|\Delta_{img,t}\|_2$
   • $E_{prop,t} = \|\Delta_{prop,t}\|_2$
     A large $E_{img,t}$ indicates high reliance on vision (e.g., during close-range interaction), while a small value suggests suppressed visual importance (e.g., during long-distance movement).

C. Refine (Gated Residual Fusion)

The diagnosed causal effects guide a Gated Residual Fusion mechanism to refine the base action:

1. Dynamic Gating: A gate $g_t$ is activated only when visual importance is low ($E_{img,t} < \tau$, where $\tau$ is a threshold). This prevents over-correction when the model is already relying heavily on vision.
   • $g_t = \mathbb{I}[E_{img,t} < \tau]$
2. Bounded Regularization: To prevent high-frequency control jitter, a bounded proprioceptive regularizer scales the proprioceptive correction relative to the visual importance.
3. Final Action: The refined action is formulated as:
   $$a_{final,t} = a_{base,t} + g_t \cdot (\alpha \Delta_{img,t} + w_{prop,t} \cdot u_{prop,t})$$
   Where $\alpha$ controls visual refinement strength, and $w_{prop,t}$ dynamically scales the proprioceptive term.

3. Key Contributions

1. IDR Framework: A training-free, model-agnostic framework that estimates visual importance via causal inference and dynamically refines actions at test time.
2. Causality-Aware Action Refiner: A specific implementation utilizing zero-padding interventions for counterfactual inference, norm-based quantification for causal diagnosis, and gated residual fusion for selective action refinement.
3. Comprehensive Validation: Extensive experiments across four VLA backbones ($\pi_{0.5}$, OpenVLA-OFT, X-VLA, VLA-Adapter), multiple simulation benchmarks (LIBERO, SIMPLER, CALVIN), and real-world dual-arm manipulation tasks.

4. Experimental Results

Simulation Benchmarks

• LIBERO: IDR improved performance across all four tested backbones. Notable gains were observed on VLA-Adapter (+1.40% average) and $\pi_{0.5}$ (+1.25%). X-VLA, which is already proprioception-dominant, showed smaller gains (+0.45%), confirming that IDR targets underutilized visual cues.
• SIMPLER & CALVIN: IDR demonstrated robustness under distribution shifts (different backgrounds, lighting, and unseen environments). In the SIMPLER "Variant Aggregation" setting, IDR improved performance by +6.33%. On CALVIN, it increased the average completed subtask length from 4.22 to 4.44.

Real-World Experiments

Evaluated on a dual-arm ARX5 platform with tasks including "Grab Cola," "Fold Clothes," "Sweep Trash," and "Organize Table."

• Performance: The average success rate increased from 56.5% to 75.3% (+18.8%).
• Efficiency: Despite requiring three forward passes per step, the average completion time for successful trials decreased from 53.6s to 41.5s due to more task-directed actions and reduced unnecessary motion.
• Long-Horizon Tasks: On the "Organize Table" task, IDR achieved a 33.3% full-completion rate compared to 0% for the baseline.

Ablation Studies

• Intervention Strategy: Zero-padding outperformed Gaussian noise and mean-value padding, providing the cleanest counterfactual signal.
• Hyperparameters: Performance peaked at a correction scale $\alpha=0.08$ and a threshold $\tau$ approximating the mean causal effect. Uniform intervention (without gating) degraded performance.
• Modality Direction: Applying refinement guided by proprioception (for a proprioception-dominant model) resulted in severe performance drops, confirming that visual cues are the primary driver for task-level correction even in proprioception-heavy models.

5. Significance and Claims

The paper claims that visual importance in VLA models is structured but not fixed; it varies significantly across architectures, environments, and execution phases. The significance of IDR lies in its ability to:

• Diagnose these dynamic shifts without retraining the model.
• Refine actions by explicitly compensating for moments where the model underutilizes visual information (e.g., during long-distance movements where the model might rely too heavily on proprioception).
• Generalize across diverse VLA backbones and deployment settings, offering a practical strategy for improving frozen VLA models in real-world scenarios.

The authors acknowledge a primary limitation: the requirement of three forward passes per control step increases inference latency. They suggest future work could involve distilling these online causal diagnostics into the model during training to enable implicit adaptation without test-time overhead.