Technical Summary: When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models

Problem Statement

Vision-Language-Action (VLA) models have demonstrated strong performance in language-conditioned robotic manipulation by mapping visual observations and natural language instructions to action sequences. However, their robustness to linguistic variation remains poorly understood. While multilingual robustness has been explored in language and vision-language models, VLA models face unique challenges: they generate continuous action streams in a closed-loop setting where language-induced errors can accumulate over time, irreversibly impacting long-horizon execution.

Current VLA systems often assume English-only instructions. When translated into non-English languages, these models exhibit severe performance degradation. Existing approaches to mitigate this, such as uniform cross-lingual alignment (e.g., CLAIM), assume that language shifts are evenly distributed across execution steps. This paper challenges that assumption, positing that language influence is highly non-uniform over time, and that treating multilingual robustness as a static, global alignment problem is insufficient for embodied agents.

Methodology

1. Systematic Multilingual Evaluation

The authors translated the LIBERO benchmark into ten languages (English, Chinese, Japanese, Korean, French, Spanish, Arabic, Thai, Portuguese, and Vietnamese) to conduct the first systematic multilingual evaluation of VLA models. They evaluated two distinct architectures: OpenVLA-OFT and $\pi0.5$.

2. Step-wise Language Sensitivity Analysis

To understand where and why failures occur, the authors introduced a step-wise analysis framework:

• Deviation Measurement: They measured the cosine distance between hidden representations generated by non-English instructions versus English instructions at each execution step.
• Gradient-Based Sensitivity: To identify language-critical steps without requiring English references during inference, they computed the text–image gradient ratio. By calculating the gradients of the predicted action with respect to language tokens ($g_{lang}$) and visual tokens ($g_{vis}$), they derived a sensitivity score $r_t = g_{lang} / (g_{vis} + \epsilon)$.
• Finding: Steps with high gradient ratios (high reliance on language) correlated strongly with steps exhibiting large deviations from English representations. This confirmed that language influence is temporally localized, with specific "hotspots" driving task failure.

3. Step-wise Inference-Time Intervention

Based on the sensitivity analysis, the authors proposed a step-wise alignment strategy that operates at inference time without retraining:

• Offline Phase: Construct a reference set of hidden representations from English training trajectories, indexed by their pre-computed language sensitivity scores.
• Online Phase:
  1. Retrieval: For a given non-English step, retrieve the $K$ nearest English reference representations based on cosine similarity.
  2. Gating: Determine if the current step is language-critical by checking if the proportion of retrieved neighbors belonging to the top sensitive steps exceeds a threshold $\tau$.
  3. Alignment: If the step is critical, compute a similarity-weighted average of the retrieved English references and update the current hidden representation: $h_{aligned} = h_t + \alpha I_t (\bar{h}_t - h_t)$.
  4. Execution: Pass the aligned representation to subsequent layers to generate the action.

4. Data-Efficient Training

The authors also explored using step-wise sensitivity to guide fine-tuning. Instead of updating all steps, they restricted parameter updates to the identified language-critical steps, comparing this against full fine-tuning and random step selection.

Key Results

Multilingual Performance Degradation

Without intervention, VLA models suffer significant performance drops under non-English instructions. Success rates dropped by 30–50% compared to English across various tasks (Spatial, Object, Goal, Long). For instance, on the "Goal" task, OpenVLA-OFT's success rate dropped from 97.9% (English) to as low as 6.4% (Arabic).

Effectiveness of Step-wise Intervention

• Inference-Time: The proposed step-wise alignment significantly outperformed baselines. For OpenVLA-OFT, average success rates improved from ~51.7% (Baseline) to 68.8% (Step-wise shift) across non-English languages. For $\pi0.5$, rates improved from ~56.7% to 80.3%.
• Comparison: Uniform alignment (Average shift) yielded unstable results and sometimes degraded performance. English Chain-of-Thought (EN-CoT) provided limited and inconsistent gains, highlighting the difficulty of applying reasoning-based prompts to action-generating policies.
• Visualization: Hidden representation distributions showed that step-wise alignment effectively shifted non-English representations toward the English reference, but only at critical steps.

Training Efficiency

When applied to fine-tuning, restricting updates to language-critical steps (50% of steps) achieved performance comparable to full fine-tuning (85.5% vs. 86.8% average success on OpenVLA-OFT) while requiring fewer updates. Random step selection performed significantly worse (44.9%), confirming that the identified steps capture meaningful language-action dependencies.

Failure Analysis

Analysis of 100 failure cases revealed that multilingual degradation is driven primarily by errors in instruction-to-visual grounding.

• 53% of failures occurred at the Navigate primitive (identifying the target location/receptacle).
• 18% occurred at Grasp.
• Failures were less frequent in later-stage primitives like Place, Open/Close, or Push.
  This indicates that language-critical steps are closely tied to semantic target selection rather than low-level motor control.

Significance and Claims

The paper claims that language robustness in VLA models is fundamentally a step-wise control problem rather than a global representation alignment issue.

1. Temporal Structure: The authors demonstrate that language influence is highly non-uniform; certain steps dominate overall task failure while others are largely language-agnostic.
2. Targeted Intervention: Uniform alignment is insufficient and potentially harmful. Selective intervention at language-sensitive steps is necessary to improve robustness without introducing noise into language-agnostic steps.
3. Efficiency: Step-wise sensitivity analysis enables both effective inference-time correction and data-efficient training, allowing models to recover performance gains with fewer updates.
4. Broader Implication: The work highlights the need for temporally structured analysis and intervention in future embodied agent systems, moving beyond static, holistic views of language robustness.

The authors note limitations, acknowledging that experiments were conducted in simulation and that the transfer of these findings to real-world robotic systems with additional noise and uncertainty remains an open question. They also suggest that as VLA architectures evolve, the nature of step-wise sensitivity may change, requiring future adaptation of these strategies.