🤖 AI

Learning Action Priors for Cross-embodiment Robot Manipulation

This paper proposes a two-stage training framework that pretrains a lightweight action module on unconditioned motion trajectories to learn cross-embodiment temporal priors, which are then transferred to Vision-Language-Action models via decoder reuse and latent distillation to achieve faster convergence, higher success rates, and improved generalization in data-scarce real-world manipulation tasks.

Original authors: Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li, Zhiwu Lu, Mingyu Ding

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li, Zhiwu Lu, Mingyu Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blind" Robot

Imagine you are hiring a new apprentice to learn how to cook. In current robot training methods (called Vision-Language-Action or VLA models), you hand the apprentice a massive cookbook filled with pictures of ingredients and written recipes (the Vision and Language parts).

However, the cookbook says nothing about how to actually move your hands. The apprentice has never held a knife, never felt the weight of a pan, and has no idea how to chop, stir, or flip. They have to figure out the physics of cooking at the exact same time they are trying to read the recipe.

The result? The apprentice is confused. They might chop the air, drop the pan, or freeze up because they are trying to learn two incredibly hard things at once: "What does the recipe say?" and "How do I move my muscles?"

This paper argues that current robots suffer from this exact problem. They are great at understanding language and seeing images, but their "muscles" (the action module) are starting from zero, with no prior knowledge of how to move.

The Solution: "Blindfolded" Practice First

The authors propose a two-step training method. Instead of throwing the robot into the kitchen with a recipe immediately, they let the robot practice moving without any recipes or pictures first.

Think of it like a dance student. Before they learn a specific dance routine to music (the recipe), they spend time in the studio just practicing the basic steps, balance, and rhythm in silence. They learn how their body moves, how to turn, and how to stop.

The Two Stages:

Stage 1: The "Blindfolded" Movement Class
- What happens: The robot is fed only data about how robot arms move (trajectories). It sees no pictures and hears no instructions.
- The Analogy: It's like a blindfolded apprentice practicing the physical motions of stirring, lifting, and placing objects over and over again. They aren't trying to cook a specific dish; they are just learning the "feel" of moving.
- The Result: The robot builds a "muscle memory" or a Motion Prior. It learns the general rules of physics: "If I move my arm this way, the object goes there." It learns the smooth flow of movement.
Stage 2: The Cooking Class
- What happens: Now, the robot is given the cookbook (images and language instructions).
- The Analogy: The apprentice is no longer a total beginner. They already know how to hold a knife and move their arm smoothly. Now, they just need to learn which knife to use and when to chop based on the recipe.
- The Result: Because the robot already knows how to move, it learns the new tasks much faster. It doesn't waste time figuring out basic physics; it focuses on understanding the instructions.

How They Connect the Two Stages

The paper uses three clever tricks to make sure the "movement practice" helps the "cooking class":

Reusing the Muscle Memory: The part of the robot that learned to move in Stage 1 is reused in Stage 2. It's like the apprentice keeping the same hands and muscles they practiced with, rather than getting a new set of hands.
The "Ghost" Teacher: In the beginning of Stage 2, the robot is told, "Remember how you moved in Stage 1? Try to move like that." This keeps the robot from going crazy or moving erratically while it's still learning the new recipes.
The "Memory Token": Robots often forget what they did a second ago. The authors created a way to compress the robot's recent history (what it did and saw) into a single, tiny "summary token." It's like the apprentice having a sticky note that says, "I just grabbed the cup, now I need to lift it." This helps the robot make better decisions in long tasks.

Why This Matters (The Results)

The researchers tested this on 13 different tasks using different types of robots (some in simulation, some real robots).

Faster Learning: The robots learned new tasks much faster because they didn't have to relearn how to move from scratch.
Better at Hard Tasks: The biggest win was on "long-tail" tasks—tasks where there is very little data (like a real-world robot trying to stack cups with only 50 examples). Without the prior, the robot would freeze or move jerkily. With the prior, it moved smoothly and successfully.
Stability: The training process was much more stable. The "gradients" (the math signals that tell the robot how to improve) were less chaotic, meaning the robot didn't "unlearn" what it knew.

The Bottom Line

This paper suggests that before teaching a robot what to do (the goal), we should teach it how to move (the motion). By giving the robot a "physical education" class before its "academic" class, it becomes a much better, faster, and more reliable worker, especially when it has to work with different types of robots or in situations where it hasn't seen many examples before.

Technical Summary: Learning Action Priors for Cross-embodiment Robot Manipulation

1. Problem Statement

Current Vision-Language-Action (VLA) models typically construct robotic policies by attaching an action generation module to a pre-trained Vision-Language Model (VLM) backbone and optimizing them jointly. While the VLM backbone inherits rich visual and linguistic priors from large-scale foundation model training, the action module is often initialized randomly or with parameters from unrelated modalities. Consequently, the action module must learn physical motion dynamics and temporal structure from scratch.

This creates a structural imbalance: early VLA training must simultaneously discover temporal action dynamics, learn cross-modal alignment (vision/language to action), and model the action distribution. In cross-embodiment settings, where robots have heterogeneous action spaces and motion distributions, this challenge is amplified. The lack of an explicit action prior leads to unstable gradients, slow convergence, and poor performance, particularly on data-scarce, long-tail real-world tasks where the action head cannot rely on sufficient downstream supervision to learn valid motion patterns.

2. Methodology

The authors propose a two-stage training framework that decouples the learning of physical motion structure from cross-modal policy alignment.

Stage 1: Learning Action Priors (Action-Only)

Before any visual or language inputs are introduced, a lightweight encoder-decoder action module is trained solely on unconditioned action trajectories (state-action sequences).

Architecture: A Transformer-based encoder-decoder.
Objective: The encoder compresses a sequence of interleaved proprioceptive states and actions into a compact latent embedding ( $z$ ). The decoder reconstructs the original action chunk from this embedding using a flow-matching objective.
Input: Only low-level trajectory data (no vision or language).
Goal: To learn a robust, generalizable prior over the continuous action distribution and temporal motion structure. The encoder learns to capture macroscopic motion semantics (e.g., translating, grasping) rather than isolated atomic actions.

Stage 2: Action-Prior-Guided VLA Training

The learned action module is transferred to the VLA training phase through three mechanisms:

Decoder Reuse: The Stage 1 decoder is reused as the VLA action head, initialized with weights that already model plausible robot motion.
Latent Alignment Distillation: The Stage 1 encoder acts as a "teacher." During early VLA training, the VLM predicts a latent action embedding ( $z'$ $z^{'}$ ) which is aligned with the encoder's ground-truth embedding ( $z$ $z$ ) via a distillation loss ( $L_{align}$ $L_{a l i g n}$ ). This anchors the VLM's action features to the learned motion latent space.
- Decay Strategy: A monotonically decaying weight $\lambda(k)$ is applied to the distillation loss. This allows the model to benefit from the prior early on but relaxes the constraint later to allow end-to-end refinement based on visual and linguistic inputs.
History Compression: The Stage 1 encoder is reused as a history compressor. It maps past state-action trajectories into a single latent token ( $z_{hist}$ ), which is injected into the VLM input. This provides temporal context without the token overhead of raw history sequences.

The total training objective combines the primary prediction loss ( $L_{pred}$ ) with the decaying distillation and reconstruction losses.

3. Key Contributions

Identification of a Structural Bottleneck: The paper identifies the absence of action priors as a key limitation in standard VLA training, particularly for cross-embodiment generalization.
Two-Stage Framework: Introduction of a dedicated action prior learning stage that equips the action module with temporal motion structure before cross-modal alignment.
Flow-Matching Encoder-Decoder: A specific architecture that learns compact, motion-aware embeddings and transfers them via decoder reuse and latent distillation.
Efficient History Integration: Demonstrating that the action encoder can serve as a compact history compressor, enabling history-aware decision-making at negligible computational cost.
Empirical Validation: Extensive evaluation across 13 diverse cross-embodiment tasks (simulated and real-world) showing faster convergence, higher success rates, and improved stability on long-tail tasks.

4. Experimental Results

The method was evaluated on 13 tasks across three benchmarks: LIBERO (4 suites), RoboCasa (5 tasks), and a Real-world Franka platform (4 tasks). All models were trained jointly on heterogeneous embodiments without environment-specific fine-tuning.

Performance Gains: Compared to a baseline without action priors ("No Action Prior"), the proposed method ("Action-State Prior + History") improved the overall average success rate from 55.3% to 68.0%.
Long-Tail Real-World Performance: The gains were most pronounced on data-scarce real-world tasks. For example, on the "Stack Cups" task, success rates increased from 25% to 75%, and "Grasp Coke" improved from 5% to 35%.
Convergence: Training dynamics analysis revealed that action priors reduced the initial prediction loss by nearly 8x and the initial gradient norm by 16x, leading to significantly faster and more stable convergence.
Scaling: Increasing the amount of action-only data in Stage 1 (scaling from ~565k to ~2.3M transitions) further improved downstream VLA performance, suggesting the approach is scalable.
Efficiency: Stage 1 training took only ~2 hours (vs. 20 hours for Stage 2) due to the absence of image/language tokens and larger batch sizes, making the overhead negligible relative to the performance gains.

5. Significance and Claims

The paper claims that the proposed framework addresses a fundamental imbalance in VLA training by providing the action module with a "motion prior" analogous to the semantic priors VLMs possess.

Decoupled Learning: By separating the learning of "how to move" (Stage 1) from "how to see and act" (Stage 2), the method prevents the action head from destabilizing the foundation backbone during early training.
Robustness in Data-Scarce Settings: The action prior is particularly valuable when downstream supervision is sparse (e.g., real-world tasks with few demonstrations), as it prevents the policy from learning unstable or jerky motions.
Generalization: The approach enables a single policy to generalize across heterogeneous embodiments (e.g., from simulated tabletop robots to humanoid robots and real Franka arms) by learning a unified, structured representation of motion dynamics.
Scalability: The framework suggests that scaling action-only datasets can directly yield stronger action priors, which transfer effectively to downstream VLA performance, offering a path toward more data-efficient robot learning.

The authors conclude that this two-stage approach yields smoother trajectories, faster convergence, and substantially stronger performance on real-world, data-scarce tasks compared to standard end-to-end VLA training.