💬 NLP

Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models

This paper presents the first systematic study revealing a significant performance gap in Vision-Language-Action models when processing non-English instructions despite multilingual language backbones, and proposes a Multilingual Principal Component Alignment fine-tuning strategy to effectively mitigate this issue.

Original authors: Hanyang Chen, Hongliang Li, Jiarui Cao, Yang Li, Yang Jiang, Haonan Wen, Kaiyu Huang, Shengnan Guo, Huaiyu Wan

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanyang Chen, Hongliang Li, Jiarui Cao, Yang Li, Yang Jiang, Haonan Wen, Kaiyu Huang, Shengnan Guo, Huaiyu Wan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "English-Only" Robot

Imagine you are teaching a robot to cook. You have a very smart robot brain (a Vision-Language-Action model, or VLA) that can see the kitchen, understand your words, and move its arms to grab a spoon or turn on a stove.

The researchers found a major problem: Most of these robots are like students who only studied in English. Even if the robot's "brain" (the underlying language model) knows how to speak French, Chinese, or Arabic, the robot itself doesn't know how to act on those languages.

When you give the robot a command in English ("Pick up the cup"), it works perfectly. But if you say the exact same thing in French ("Prends la tasse"), the robot often freezes, gets confused, or grabs the wrong object. It's as if the robot suddenly forgot how to move its hands just because you changed the language.

The Experiment: Testing the Robot's "Ear"

The researchers wanted to see if this was true. They took existing robot training data (which was almost entirely in English) and created a "multilingual test."

Translation: They took English instructions and translated them into four other languages: Chinese, French, Russian, and Arabic.
The Mix-Up: They also created "code-switching" instructions, where a sentence is mostly in one language but has a few key words (like "cup" or "pick") in English.
The Test: They asked the robots to perform tasks in a simulated kitchen using these different languages.

What They Found: The "Multilingual Gap"

The results were clear: There is a huge gap.

The "English-Only" Brain: Robots trained on English data performed terribly when asked to speak other languages. Their success rate dropped significantly, sometimes almost to zero.
The "Multilingual" Brain: Some robots were built on top of language models that already knew many languages (like Qwen-VL). These robots did better, especially with Chinese, but they still struggled compared to English.
The Visual Trap: The problem got worse when the visual scene was confusing. If two different tasks looked exactly the same (like putting a bowl on a plate vs. turning on a stove), the robot relied heavily on the language. If the language was in a foreign tongue, the robot got completely lost.
The Power of Keywords: Interestingly, if the robot heard a sentence in French but the word "cup" was still in English, it did much better. It's like the robot has a "keyword trigger" that helps it figure out what to do, even if it doesn't understand the rest of the sentence.

Why Does This Happen? (The "Translation" Problem)

The researchers looked inside the robot's "brain" to see what was going wrong. They found two main issues:

Confused Understanding: Sometimes, the robot simply didn't understand the foreign instruction. It heard "turn on the stove" in French but thought you said "put the bowl on the plate" because the visual scene looked similar.
The "Action Head" Mismatch: This is a technical part, but think of it like a translator at the end of the line. The robot's brain understands the words, but the part that actually moves the arms (the "Action Head") is tuned to English. When the language changes, the "signal" sent to the arms gets garbled.
- Analogy: Imagine a conductor (the brain) speaking French to an orchestra, but the musicians (the robot arms) only know how to read sheet music written for English speakers. The music gets played wrong.
- They found that some types of "conductors" (specific designs called GR00T and π) were better at handling this mix-up than others.

The Solution: "Principal Component Alignment" (MPCA)

The researchers didn't just point out the problem; they offered a fix. They proposed a method called Multilingual Principal Component Alignment (MPCA).

The Analogy: Imagine the robot's brain has a "map" of how words relate to actions. In English, the map is clear. In French, the map is shifted to the left, so the robot walks in the wrong direction.
The Fix: MPCA is like a compass that realigns the French map so it points in the same direction as the English map. It doesn't require retraining the whole robot from scratch. It just takes the existing knowledge and "rotates" the foreign language data so it fits the robot's existing action system.

The Takeaway

The paper concludes that we cannot assume robots will work globally just because their language models are multilingual. If we want robots to work in real-world, multilingual environments, we need to specifically train and align them to understand that different languages should lead to the same physical actions.

They showed that with a simple adjustment (MPCA), we can significantly improve a robot's ability to follow instructions in languages other than English, making them more ready for the real world.

Technical Summary: Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models

Problem Statement

Vision-Language-Action (VLA) models have emerged as a promising paradigm for developing generalist robot policies by integrating visual perception, natural language understanding, and action generation. However, a critical limitation exists in current VLA systems: they are predominantly trained and evaluated using English instructions. While the underlying Large Language Models (LLMs) and Vision-Language Models (VLMs) often possess multilingual capabilities, it remains unclear whether these capabilities transfer effectively to the action generation space during the alignment process.

The central problem addressed is the potential "multilingual gap"—the degradation in robot performance when instructions are provided in languages other than English. This gap poses a significant barrier to real-world deployment in global environments where robots must understand diverse linguistic inputs. The paper investigates whether VLAs truly retain multilingual understanding after being aligned with robot actions and seeks to identify the underlying causes of performance degradation in non-English settings.

Methodology

1. Multilingual Evaluation Framework

The authors propose a systematic evaluation framework consisting of three stages:

Multilingual Dataset Construction: Existing robot manipulation benchmarks (LIBERO and SimplerEnv) were extended by translating English instructions into four target languages: Chinese (zh), French (fr), Russian (ru), and Arabic (ar). Two interaction settings were defined:
- Multilingual Instructions: Direct translations preserving original semantics.
- Code-Switching Instructions: Mixed-language expressions where key verbs and nouns are substituted from English into the translated text, simulating common multilingual communication scenarios.
- A pipeline using Cloud Translation APIs and LLMs (for Named Entity Recognition and semantic consistency checking via "LLM-as-a-Judge") was employed to generate high-quality instruction variants.
Multilingual Evaluation Adapter: A lightweight adapter was developed to seamlessly swap language inputs in the evaluation pipeline without modifying model architectures or environment logic, ensuring a fair comparison across languages.
Evaluation Protocol: Task success rates were measured across different languages. Performance was quantified using Relative Performance, calculated as the difference between the success rate in a multilingual setting and the original English setting.

2. Experimental Setup

Models: Several representative VLA models were evaluated, including $\pi_0.5$ , OpenVLA-OFT, ABot-M0, Cosmos Policy, and various Qwen-VL-based models with different action head designs (OFT, FAST, $\pi$ , GR00T).
Benchmarks: Experiments were conducted on the LIBERO and SimplerEnv simulation environments, covering tasks such as object rearrangement, tool use, and spatial manipulation.

3. Proposed Solution: Multilingual Principal Component Alignment (MPCA)

Motivated by the finding that multilingual instructions cause representation shifts, the authors propose MPCA, a fine-tuning approach.

Mechanism: MPCA utilizes Principal Component Analysis (PCA) to identify the principal component subspace of the language embeddings. It then aligns the projected multilingual representations to this subspace.
Goal: To reduce the distribution shift between English and non-English instruction representations, thereby improving the model's ability to generalize across languages.

Key Findings and Results

1. Existence of a Significant Multilingual Gap

Experiments revealed a substantial performance drop for models trained primarily on English when evaluated on non-English instructions.

Base Model Dependency: Models based on English-centric VLMs (e.g., PaliGemma, Prismatic) showed severe degradation, often performing similarly to models with no instructions. In contrast, models based on multilingual VLMs (e.g., Qwen-VL) demonstrated better generalization, particularly for Chinese, though gaps persisted for other languages.
Visual Context Amplification: The gap was most pronounced in environments with similar visual inputs (e.g., LIBERO-Goal), suggesting that language understanding becomes the critical differentiator when visual cues are ambiguous.

2. Impact of Instruction Structure and Action Heads

Keyword Importance: Code-switching instructions (containing English keywords) resulted in smaller performance drops compared to fully translated instructions. This indicates that key verbs and nouns are crucial for VLA understanding.
Action Head Design: The architecture of the action head significantly influences multilingual robustness.
- Diffusion-based heads (GR00T-style, $\pi$ -style) maintained better performance, likely because they retain semantic information from the base VLM and adapt better to distribution shifts.
- Token-based heads (FAST-style) showed worse performance, potentially due to semantic information loss during tokenization.
- MLP-based heads (OFT-style) showed intermediate performance, struggling to adapt to the distribution shift caused by multilingual inputs.

3. Root Cause: Representation Shift

Representation analysis (visualizing pooled embeddings) confirmed that the multilingual gap stems from representation shifts.

In models with poor multilingual performance, embeddings for non-English instructions were far removed from English embeddings in the latent space.
In models with better performance (e.g., Qwen3-VL-π), English and Chinese embeddings were closely aligned, correlating with better generalization.
The failure modes were categorized into: (1) Instruction Misunderstanding (generating completely wrong actions due to language confusion) and (2) Execution Failure (understanding the intent but failing to generate the correct action sequence).

4. Effectiveness of MPCA

The proposed MPCA strategy effectively reduced the multilingual performance gap. By aligning multilingual representations to the principal component subspace, MPCA improved task success rates in non-English settings, demonstrating its potential as a practical solution for enhancing multilingual robustness.

Significance and Contributions

The paper claims the following contributions:

First Systematic Study: It presents the first comprehensive investigation into multilingual instruction following in VLA models, moving beyond the English-only assumption prevalent in current research.
Framework and Benchmark: It introduces a multilingual evaluation framework and dataset extensions for LIBERO and SimplerEnv, enabling controlled comparisons across languages and interaction settings.
Empirical Insights: It reveals a significant multilingual gap in current systems and provides detailed analysis on how base model capabilities, action head designs, and instruction structures influence this gap.
Practical Solution: It proposes MPCA, a simple yet effective fine-tuning method that leverages representation alignment to mitigate the multilingual gap, offering a pathway toward more robust, globally deployable embodied agents.

The authors conclude that while VLAs show promise, their multilingual capabilities are currently underdeveloped and require specific architectural and training interventions to match the performance of their English counterparts.