💬 NLP

EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

The paper proposes EmbodiedMidtrain, a method that bridges the gap between Vision-Language Models (VLMs) and Vision-Language-Action Models (VLAs) by using a data engine to curate VLA-aligned VLM data for mid-training, thereby significantly improving downstream robot manipulation performance across various backbones.

Original authors: Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong

Published 2026-04-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class chef (the Vision-Language Model or VLM). This chef has read every cookbook, watched every cooking show, and can describe a tomato in poetic detail. They are an expert at talking about food and recognizing ingredients.

Now, you want to hire this chef to actually cook a meal in a busy, chaotic kitchen where they have to grab a knife, chop an onion, and stir a pot without dropping anything. This is the job of a Vision-Language-Action Model (VLA).

The problem? The chef is great at theory, but terrible at practice. If you just hand them a recipe and say, "Go cook," they might knock over the salt shaker or cut their finger because their brain is wired for describing food, not moving a robotic arm to grab it.

This paper introduces EmbodiedMidtrain, a clever "cooking school" designed to bridge that gap. Here's how it works, broken down simply:

1. The Problem: The "Library vs. Workshop" Gap

The authors realized that the data used to train the "chefs" (VLMs) is like a massive, dusty library full of books about food. The data needed for the "robot cooks" (VLAs) is like a messy, real-world workshop with actual ingredients and tools.

The Gap: The library data is huge and diverse, but most of it is useless for the workshop. A book about the history of the tomato is great for a quiz, but it doesn't teach you how to hold a knife.
The Mistake: Previous attempts tried to just throw the chef into the workshop and hope they learned. But because the chef started with the wrong mindset (thinking in books, not in actions), they struggled to learn quickly.

2. The Solution: The "Smart Scout" (The Data Engine)

Instead of forcing the chef to read every book in the library before entering the workshop, the authors built a Smart Scout.

How the Scout Works: Imagine a scout who has a special pair of glasses. They can look at a book in the library and instantly say, "This one is about chopping onions? Keep it! This one is about the history of ketchup? Put it back."
The Magic: The authors trained a tiny, lightweight AI (the "proximity estimator") to act as this scout. It scans the massive library of general data and picks out only the specific pages that look most like the real-world workshop tasks.
The Result: Instead of a random mix of books, the chef now gets a curated reading list that is 90% about hands-on cooking, spatial reasoning, and tool use.

3. The Process: "Mid-Training"

This is the "Mid-training" part of the name.

Start: You have the brilliant chef (the pre-trained VLM).
The Detour: Before sending them to the robot workshop, you send them to a special "Mid-training" camp.
The Camp: In this camp, they only study the curated list the Scout picked out. They practice visualizing how to grab a cup, how to avoid obstacles, and how to move their "hands."
The Finish: Now, when you finally send this chef to the robot workshop to learn the specific robot tasks, they are already 80% there. They don't have to unlearn their "bookish" habits; they are already thinking like a robot.

4. Why It's a Big Deal

The paper shows that this method is a game-changer for three reasons:

Small Fish, Big Pond: They took a relatively small robot model (1.1 billion parameters) and, using this method, made it perform better than massive, expensive models (7 billion+ parameters) that were trained with way more data. It's like a small, well-trained apprentice beating a giant, untrained novice.
It Works Everywhere: They proved that the "curated list" works even if you swap the chef. The data selected for one type of robot brain worked perfectly for a different type of robot brain, too.
Faster Learning: Because the robot started with the right "muscle memory" from the mid-training, it learned the final tasks much faster. The performance gap between the new method and the old method got wider the longer they trained, proving it wasn't just a lucky start—it was a fundamentally better foundation.

The Takeaway

EmbodiedMidtrain is like realizing that to teach a human to play soccer, you shouldn't just give them a dictionary of sports terms. Instead, you should first show them a highlight reel of only the best soccer moves, filtered out from a million hours of random sports footage.

By filtering the data to match the specific "embodied" reality of robots, the authors created a shortcut that makes robots smarter, faster, and more capable without needing to build bigger, more expensive brains.

1. Problem Statement

Vision-Language-Action Models (VLAs) are the current frontier in robotics, unifying perception, language understanding, and action generation. Most VLAs are built by fine-tuning pre-trained Vision-Language Models (VLMs). However, a fundamental distributional gap exists:

VLM Data: Broad, diverse, and often text-centric (e.g., captioning, visual QA, document understanding).
VLA Data: Compact, specialized, and grounded in physical interaction (robotic manipulation trajectories).

Simply fine-tuning a general-purpose VLM on embodied data often yields suboptimal results because the VLM's internal representations are not well-aligned with the specific reasoning required for embodied control. Current approaches either fail to transfer VLM capabilities effectively or require massive amounts of in-domain robot data, which is scarce.

2. Methodology: EmbodiedMidtrain

The authors propose EmbodiedMidtrain, a mid-training framework designed to bridge the VLM-VLA gap before the final VLA fine-tuning stage. The core idea is to curate a specific subset of VLM data that is highly aligned with the VLA domain to create a stronger initialization.

The pipeline consists of three main components:

A. Data Distribution Analysis

The authors first characterize the gap using Maximum Mean Discrepancy (MMD) and t-SNE visualization. They found:

VLA data forms compact, separated clusters distinct from the broad VLM distribution.
However, the gap is not uniform; some VLM samples are naturally closer to the VLA domain than others.
This suggests that sample-level selection is more effective than coarse dataset-level mixing.

B. Proximity-Based Data Selection Engine

To identify the most relevant VLM samples, the authors developed a lightweight Proximity Estimator:

Feature Extraction: They use the frozen last hidden states of a VLM as feature representations ( $\phi(x)$ ).
Classifier Training: A lightweight, learnable binary classifier (a linear layer + sigmoid) is trained to distinguish between VLA samples (positive) and VLM samples (negative).
- Theoretically, the optimal classifier output approximates the density ratio $p_{VLA}(x) / (p_{VLA}(x) + p_{VLM}(x))$ , serving as a continuous "proximity score."
Selection: All candidate VLM samples are scored. The top- $K$ $K$ samples with the highest proximity scores are selected to form a curated mid-training mixture.
- This mixture preserves diversity (spanning both general and embodied-oriented sources) but shifts the distribution toward the VLA domain.

C. Mid-Training and Fine-Tuning

Mid-Training: The base VLM is mid-trained on the curated mixture. This adapts the model's representations to be more "embodied-ready" without architectural changes.
VLA Fine-Tuning: The mid-trained VLM is then used as the backbone for standard VLA fine-tuning (cascading with an action decoder) on downstream robot manipulation tasks.

3. Key Contributions

Proximity-Based Mid-Training Pipeline: Introduced a novel data engine that learns a proximity estimator to score and select VLM samples based on their alignment with the VLA domain, rather than relying on hand-crafted metrics or random sampling.
Efficient Performance Gains: Demonstrated that mid-training a small model (e.g., 1.1B parameters) with this curated data can outperform much larger off-the-shelf models (3B–30B) and expert VLAs on robotics benchmarks.
Cross-Backbone Transferability: Showed that data selected using features from one VLM (InternVL3.5-1B) effectively improves a different architecture (Qwen3VL-2B), indicating the selection captures generalizable alignment signals.
Analysis of Training Dynamics: Revealed that the benefits of mid-training emerge immediately at the start of fine-tuning and widen over time, proving it provides a fundamentally better initialization rather than a transient advantage.

4. Experimental Results

Experiments were conducted on three simulated manipulation benchmarks: Calvin ABC-D, SimplerEnv Bridge, and Libero-10.

Performance: The mid-trained InternVL3.5-1B (1.1B params) achieved:
- Calvin: 36.5% success rate (vs. 39.0% for the baseline without mid-training), surpassing larger expert models like OpenVLA (7.7B) and $\pi_0$ (3.1B).
- SimplerEnv: 56.3% (vs. 36.5% baseline), outperforming larger models like Qwen3VL-8B and Paligemma-2.
- Libero-10: 54.2% (vs. 39.0% baseline).
Efficiency: The method achieved these results with significantly fewer training samples (approx. 4.1M vs. 25.6M for baselines) and a smaller model scale.
Ablation Studies:
- Random Selection: Randomly sampling data performed significantly worse, confirming that selection is critical.
- Proximity Metrics: The learned estimator outperformed hand-crafted alternatives like feature-space distance and perplexity-based metrics.
- Data Composition: The selected data favored spatial reasoning tasks (e.g., RefSpatial) over text-centric tasks (e.g., book cover QA), while maintaining high overall data diversity.

5. Significance and Conclusion

EmbodiedMidtrain addresses a critical bottleneck in robotics foundation models: the misalignment between general pre-training data and specific embodied tasks.

Paradigm Shift: It moves away from the "brute force" approach of training larger models on more data, instead focusing on data quality and distribution alignment.
Scalability: The method is lightweight, scalable, and requires no architectural modifications to existing VLMs or VLAs.
Generalizability: By proving that proximity-based selection works across different model architectures, it offers a general recipe for preparing vision-language backbones for embodied intelligence.

The authors conclude that mid-training with carefully selected, distribution-aligned data provides a superior initialization for downstream VLA learning, enabling smaller, more efficient models to achieve state-of-the-art performance in robot manipulation. All code, data, and models are released for future research.