LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer

1. Problem Statement

The primary goal of robotics foundation models is to create a "generalist" policy capable of controlling diverse robot embodiments (different hardware, grippers, camera placements) without requiring per-robot fine-tuning. While Vision-Language-Action (VLA) models have made significant progress by leveraging pre-trained Vision-Language Models (VLMs), they currently suffer from a critical limitation: poor zero-shot cross-embodiment transfer.

• The Bottleneck: Existing VLAs are tightly coupled to their training embodiments. Even with large-scale multi-embodiment pre-training, they typically fail to generalize to new robots with minor hardware differences (e.g., a different gripper or wrist camera location) without costly, embodiment-specific fine-tuning.
• The Root Cause: Standard VLA training often forces pre-trained VLMs to predict raw, continuous motor commands (joint angles or end-effector poses) represented by arbitrary or abstract tokens. This creates a distributional mismatch between the VLM's pre-training (natural language) and the fine-tuning objective (motor control), causing the model to lose its generalizable semantic representations and overfit to specific hardware configurations.

2. Methodology: Language-Action Pre-Training (LAP)

The authors propose LAP, a pre-training recipe that bridges the gap between high-level semantic understanding and low-level motor control by representing actions directly in natural language.

Core Concept: Language-Actions

Instead of predicting raw continuous values or abstract tokens, LAP trains the VLM to predict language-actions: structured natural language descriptions of the net effect of a robot's movement.

• Format: Actions are parsed from continuous end-effector trajectories into templates like: "Move forward 5 cm," "Tilt right 30 degrees," or "Rotate clockwise 45 degrees."
• Derivation: These are generated deterministically from robot data using fixed coordinate conventions and templates, requiring no manual annotation and no learned tokenizer.
• Benefit: This keeps the supervision signal within the VLM's pre-training distribution (natural language), preserving semantic grounding and enabling transfer across different physical embodiments.

Model Architecture: LAP-3B

To address the inefficiency of autoregressive language generation for real-time control, the authors instantiate LAP as LAP-3B, a hybrid architecture inspired by $\pi_0.5$:

1. VLM Backbone: A PaliGemma-3B model trained via Cross-Entropy Loss to predict language-action tokens. This component learns the generalizable representation.
2. Action Expert: A lightweight, diffusion-based (flow-matching) module that predicts continuous, high-frequency control signals.
3. Knowledge Insulation: Gradients from the Action Expert are blocked from flowing back into the VLM backbone. This ensures the VLM is trained purely on language supervision, preventing the "catastrophic forgetting" of pre-trained visual-semantic knowledge.
4. Inference: At runtime, the VLM generates a low-frequency language-action plan, which the Action Expert executes as a continuous trajectory at 25 Hz.

Training Data & Co-training

• Datasets: Trained on Open X-Embodiment (OXE) and MolmoAct datasets.
• Co-training: The framework naturally supports co-training with Visual-Question-Answering (VQA) tasks (e.g., "Motion Prediction"), where the model predicts the language-action describing the displacement between two image frames. This further unifies the learning objective.

3. Key Contributions

1. LAP Framework: A novel pre-training recipe that represents low-level robot actions as natural language, solving the distributional mismatch in standard VLA training.
2. LAP-3B: The first VLA to achieve substantial zero-shot transfer to previously unseen robot embodiments without any embodiment-specific fine-tuning.
3. Efficiency: Demonstrates that language-action representations enable 2.5x fewer demonstrations and fewer gradient steps for fine-tuning to new robots compared to state-of-the-art baselines.
4. Scalability: Shows favorable scaling behavior where performance improves monotonically with model size (4B to 27B), whereas baseline methods saturate or degrade.

4. Experimental Results

The authors evaluated LAP-3B across four robot embodiments (one seen, three unseen) and ten real-world manipulation tasks.

• Zero-Shot Performance:
  • On unseen embodiments (Custom Franka, YAM, Kinova), LAP-3B achieved an average success rate of >50%.
  • This represents a ~2x improvement (approx. 30% absolute gain) over the strongest prior baselines (e.g., $\pi_0.5$, OpenVLA, X-VLA).
  • Crucially, all open-sourced baseline VLAs collapsed to ~0% success on unseen robots, while LAP-3B maintained robust performance.
• Fine-Tuning Efficiency:
  • In simulation (LIBERO benchmark), LAP-3B converged to near-optimal performance in a fraction of the steps required by baselines.
  • On real robots, LAP-3B achieved comparable task performance using 2.5x fewer demonstrations than baseline policies.
• Representation Analysis:
  • t-SNE Visualization: LAP-3B's learned feature representations showed significant overlap between training and unseen embodiments, indicating embodiment-agnostic learning. Baselines formed distinct, non-overlapping clusters.
  • Prediction Error: LAP-3B maintained lower action prediction error on held-out unseen datasets throughout training, suggesting more stable and generalizable dynamics.
• Scaling: As model size increased from 4B to 27B parameters, LAP-3B showed consistent loss reduction and performance gains, while the baseline ($\pi_0.5$-replicated) showed early saturation.

5. Significance and Impact

• Paradigm Shift: The paper challenges the prevailing approach of using abstract tokens or raw coordinates for robot control. It demonstrates that natural language is a superior interface for aligning motor control with pre-trained semantic knowledge.
• Democratization of Robotics: By enabling zero-shot transfer, LAP reduces the barrier to deploying foundation models on new hardware. It eliminates the need for expensive, data-hungry fine-tuning for every new robot type.
• Unified Representation: LAP unifies action prediction and VQA, suggesting a path toward truly generalist agents that can reason, communicate, and act within a single, coherent language-based representation space.
• Future Directions: The authors note that while LAP works well for single-arm manipulators, it is easily adaptable to bimanual robots and can leverage less precise data sources (e.g., human pose tracking, internet videos) due to the tolerance of language descriptions to noise.

In summary, LAP provides a simple yet powerful solution to the "embodiment gap" in robotics, proving that representing actions in natural language allows foundation models to generalize across the physical world without retraining.