💻 computer science

Rethinking VLM Representation for VLA Initialization

This paper investigates optimal strategies for initializing Vision-Language-Action (VLA) models from pretrained Vision-Language Models (VLMs), revealing that preserving the original VLM representation while employing staged LoRA-based training with robot-data pretraining yields the most effective action learning performance.

Original authors: Weifeng Lin, Siyuan Huang, Hao Li, Tingwei Chen, Ruichuan An, Xinyu Wei, Jianbo Liu, Hongsheng Li

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weifeng Lin, Siyuan Huang, Hao Li, Tingwei Chen, Ruichuan An, Xinyu Wei, Jianbo Liu, Hongsheng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do chores, like picking up a cup or folding laundry. To do this, you give the robot a "brain" that already knows a lot about the world—like how to recognize a cat, understand a sentence, or know what a kitchen looks like. This brain is called a Vision-Language Model (VLM).

The paper asks a simple but tricky question: How should we tweak this pre-trained brain so it becomes a good "robot brain" (a VLA model) without breaking the good stuff it already knows?

The authors treat this like a controlled experiment in a kitchen, testing three main ingredients to see what makes the perfect robot recipe.

1. The "Specialized Training" Ingredient (Embodied VQA)

Imagine your robot brain is a generalist chef who knows how to cook anything. You want to teach it to be a robot chef. You could give it a crash course in specific skills, like "how to hold a spoon" or "where the soap is."

The Experiment: The researchers gave the robot brain different types of "homework" (called VQA, or Visual Question Answering) based on seven different skills, like understanding space, knowing where objects are, or predicting camera angles.
The Surprise: Just giving the robot more homework didn't always help.
- If the robot was already good at a task, extra homework helped.
- If the robot was struggling with a different type of task, the same homework actually made it worse.
- The Sweet Spot: The best results came from a specific combo: teaching the robot where things are (Grounding) and what the robot itself is doing (Egocentric Understanding). Adding too many other skills (like predicting camera angles or planning complex steps) actually confused the robot, like trying to learn five new languages at once.

2. The "How Hard We Study" Ingredient (Update Strategy)

Once the robot starts its homework, how much should we change its brain?

The "Full Rewrite" (Full Fine-tune): This is like telling the robot, "Forget everything you knew before. Rewrite your entire brain to focus only on these new robot tasks."
- Result: This often backfired. The robot learned the new tasks but forgot the general knowledge that helped it understand the world in the first place. It became a specialist who couldn't think outside the box.
The "Gentle Nudge" (LoRA): This is like giving the robot a small set of sticky notes (adapters) to stick on its brain. It learns the new robot tasks without erasing its old knowledge.
- Result: This worked much better. The robot kept its general smarts but learned the specific robot skills. The paper found that preserving the original brain was more important than completely reshaping it.

3. The "Real-World Practice" Ingredient (Robot Data Pretraining)

Finally, the researchers asked: Should we teach the robot using just pictures and questions, or should we also show it videos of real robots moving?

The Finding: Showing the robot videos of real movements (robot data) helped, but only if done carefully.
The Best Recipe: The winning strategy was a two-step process:
1. First, gently nudge the brain with the "Where things are" and "What I'm doing" homework (using the gentle LoRA method).
2. Second, let the robot practice with real movement data, again using the gentle nudge method.
- Trying to do everything at once (mixing all the homework and real practice together) was less effective than doing it in stages.

The Big Takeaway

The paper concludes that building a good robot brain isn't about throwing everything at it. It's about balance:

Don't erase the past: The original "general knowledge" the robot had is actually very useful for learning new robot tasks. Don't wipe it out.
Pick the right homework: Only teach the robot the specific skills that match the job it needs to do. Teaching it everything at once creates confusion.
Go slow: Use gentle updates (like sticky notes) rather than a total brain rewrite, and introduce new skills in stages.

In short, the best way to turn a smart general AI into a smart robot is to add new skills carefully without deleting the old ones.

Technical Summary: Rethinking VLM Representation for VLA Initialization

Problem Statement

Vision-Language-Action (VLA) models typically initialize their policy backbones from pretrained Vision-Language Models (VLMs) to inherit visual-language representations. However, it remains unclear what specific characteristics of a pretrained VLM representation constitute an effective initialization for VLA tasks. While recent works have explored stronger backbones and action modules, the choice of initialization is often treated as a default step rather than a controlled design variable. This paper addresses the fundamental question: What kind of pretrained VLM representation makes a useful VLA initialization?

The authors argue that simply adopting a stronger VLM or applying broad embodied adaptation does not guarantee better action policies. Instead, they frame VLA initialization as a controlled representation-design problem that must balance the injection of action-relevant signals with the preservation of the original, transferable VLM knowledge.

Methodology

The study investigates VLA initialization along three primary axes using a two-stage training pipeline:

Stage 1 (Representation Shaping): Adapting a base VLM on embodied Visual Question Answering (VQA) data or robot trajectory data.
Stage 2 (Action Learning): Initializing a VLA policy from the Stage 1 checkpoint and training it on action trajectories.

The three axes of investigation are:

1. Capability-Level Embodied VQA Domains

The authors organize embodied VQA data into seven capability-oriented domains to test which signals transfer positively to action learning:

Spatial: Relative/absolute relations, orientation, distance.
Grounding: Spatial referring, localizing objects/regions.
Plan & Reasoning: Decomposing goals, preconditions, step ordering.
Camera Prediction: Estimating intrinsics/extrinsics/viewpoint changes.
Egocentric Understanding: Hand/gripper position, held objects, reachability.
Temporal Understanding: Action ordering, event boundaries, causality.
Action Next-Token Prediction (Action-NTP): Treating action trajectories as language tokens.

They evaluate single-domain adaptation, pairwise compositions, and multi-domain compositions.

2. Parameter-Update Strategy

The paper compares two strategies for Stage 1 adaptation to analyze the trade-off between specialization and preservation:

Full Fine-tune (Full FT): Updates all parameters, aggressively reshaping the representation.
LoRA (Low-Rank Adaptation): Updates only a small set of adapter parameters, preserving most original VLM weights and limiting representation drift.

3. Robot-Data Pretraining

The study examines the role of action-side supervision by using robot trajectory data (AgiBot-World-Beta) as a pretraining signal. They compare:

Robot-data-only pretraining.
Joint robot-data and VQA pretraining.
Staged pretraining (VQA adaptation followed by robot-data pretraining).

Experimental Setup

Base Models: Qwen3-VL-4B (primary), Qwen3-VL-2B, and PaliGemma2-3B.
Architectures: OpenVLA-OFT style (MLP head) and $\pi_0$ -style (Diffusion Expert).
Benchmarks: Libero-10 (long-horizon tabletop), SimplerBridge (real-to-sim), and RoboCasa GR1 Tabletop (high-dimensional humanoid).
Metrics: Mean success rate over three evaluation seeds.

Key Results

1. The Critical Role of Pretrained Representations

Training VLA policies from scratch results in a performance drop of over 20% across all benchmarks compared to initializing from a pretrained VLM. This confirms that the original pretrained VLM representation is a primary source of action performance.

2. Conditional Utility of Embodied VQA

Embodied VQA adaptation does not yield uniform gains; its effectiveness depends on the downstream bottleneck and the specific capability injected:

Benchmark Dependency: On Libero-10, most domains improve performance. On SimplerBridge (real-to-sim), most domains degrade performance relative to the baseline, with only Grounding showing slight gains under the Diffusion Expert.
Capability Specificity: Grounding, Egocentric Understanding, and Action-NTP provide the most robust transfer signals. Other domains (e.g., Spatial, Plan & Reasoning) show inconsistent or negative transfer depending on the benchmark.
Non-Additivity: Combining domains is not simply additive. The strongest improvement comes from the pairwise composition {Grounding + Egocentric Understanding}. Adding more domains (e.g., a 7-domain uniform composition) often leads to saturation or performance drops, suggesting that broader coverage can dilute useful supervision or introduce interference.

3. Preservation vs. Specialization (LoRA vs. Full FT)

LoRA Superiority: LoRA consistently provides a stronger VLA initialization than Full Fine-tuning. Full FT often falls below the baseline, indicating that aggressively reshaping the representation for VQA tasks can overwrite information useful for downstream action learning.
Base Model Strength: The advantage of LoRA is more pronounced with stronger base VLMs. As the base model weakens (e.g., from Qwen3-VL-4B to PaliGemma2-3B), the gains from LoRA diminish, and Full FT degradation becomes more severe. This suggests that adapter-based adaptation is most effective when there is a high-quality pretrained representation to preserve.

4. Robot-Data Pretraining and Staged Adaptation

Action-Side Signals: Robot-data pretraining improves initialization, but its interaction with VQA supervision is complex.
Competition in Joint Training: Jointly training on robot data and VQA using LoRA can result in performance lower than robot-data-only training, likely due to objective competition within the limited adapter capacity.
Staged Approach: The best performance is achieved via a staged route: first adapt the VLM with {Grounding + Egocentric Understanding} VQA, then continue with LoRA-based robot-data pretraining. This approach reaches the highest success rates (55.2% on RoboCasa), suggesting that sequential injection of signals mitigates competition while preserving the core VLM representation.

Significance and Claims

The paper reframes VLA initialization from a default backbone selection or data-scaling step into a controlled representation-design problem. The authors claim their findings provide practical guidance for constructing effective VLA initializations:

Signal Injection: Effective initialization requires injecting action-relevant signals (specifically Grounding and Egocentric Understanding) that align with the downstream bottleneck.
Representation Preservation: It is crucial to preserve the pretrained VLM representation that remains useful for action learning. Over-reshaping the representation via Full Fine-tuning is detrimental.
Adaptation Strategy: Constrained adapter-based updates (LoRA) are superior to full-parameter updates for balancing specialization with preservation.
Staged Training: A staged training pipeline (VQA adaptation followed by robot-data pretraining) is more effective than joint training, as it avoids signal competition and leverages the strengths of both perception-side and action-side supervision.

The authors conclude that these insights offer a nuanced understanding of the design space for VLA initialization, moving beyond the assumption that "more embodied data" or "stronger backbones" automatically lead to better policies.