💻 computer science

One-Policy-Fits-All: Geometry-Aware Action Latents for Cross-Embodiment Manipulation

The paper proposes One-Policy-Fits-All (OPFA), a framework that learns a shared geometry-aware latent action space to enable a single policy to effectively manipulate diverse robot embodiments through end-to-end co-training, significantly improving data efficiency and cross-embodiment skill transfer.

Original authors: Juncheng Mu, Sizhe Yang, Hojin Bae, Feiyu Jia, Qingwei Ben, Boyi Li, Huazhe Xu, Jiangmiao Pang

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juncheng Mu, Sizhe Yang, Hojin Bae, Feiyu Jia, Qingwei Ben, Boyi Li, Huazhe Xu, Jiangmiao Pang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of robots how to do chores. You have a robot with a simple two-finger clamp (like a clothespin), another with a three-finger gripper, and a third with a highly dexterous, human-like hand with five fingers and many joints.

The Problem:
Currently, if you want to teach the "clothespin" robot to pick up a banana, you have to collect hundreds of hours of video data just for that specific robot. If you then want to teach the "human-like hand" to do the same thing, you have to start from scratch and collect another huge pile of data. It's like hiring a chef who only knows how to cook with a wooden spoon, then hiring a new chef who only knows how to cook with a spatula, and forcing them to relearn everything from zero. They can't share their knowledge because their "hands" are built so differently.

The Solution: One-Policy-Fits-All (OPFA)
The paper introduces a new framework called OPFA (One-Policy-Fits-All). Think of OPFA as a universal translator and a shared brain for robots.

Here is how it works, using a simple analogy:

1. The "Shape-Shifting" Language (Geometry-Aware Latent Representation)

Instead of teaching the robots to move their specific joints (e.g., "move finger 1 up, move finger 2 down"), OPFA teaches them to think in shapes and spaces.

The Analogy: Imagine you are trying to describe a dance move to a friend.
- Old Way: You say, "Move your left knee up 3 inches, then rotate your right ankle 15 degrees." This only works if your friend has legs exactly like yours.
- OPFA Way: You say, "Reach out and grab the air in a circle." This instruction is about the shape of the movement, not the specific body parts.
How it works: The system looks at the robot's hand and converts its physical movements into a "point cloud" (a 3D map of where the hand can reach). It then compresses this 3D map into a single, abstract "idea" or latent code. This code represents the intent of the action (e.g., "grasp the object firmly") without caring if the robot has 2 fingers or 20.

2. The Universal Translator (Unified Decoder)

Once the robot has this abstract "idea" of what to do, it needs to translate it back into specific movements for its own body.

The Analogy: Think of the "idea" as a universal recipe (e.g., "Make a cake").
- Old Way: You need a different recipe book for every type of oven (gas, electric, solar). If you want to use a new oven, you need a whole new book.
- OPFA Way: You have one master translator. The robot takes the "universal recipe" (the latent code) and the translator instantly figures out, "Oh, your oven needs the dial set to 350, while that robot's oven needs it at 180."
The Magic: This translator is unified. It doesn't need to be retrained for every new robot. It can take the same "grasp" idea and instantly figure out how to move a simple gripper OR a complex human-like hand to achieve the same result.

3. The Super-Learning Effect (Cross-Embodiment Co-Training)

Because all robots are now speaking this "universal shape language," they can learn from each other instantly.

The Analogy: Imagine a classroom where a student with a bicycle, a student with a skateboard, and a student with a unicycle are all learning to navigate a maze.
- Old Way: They practice separately. The skateboarder never learns from the bicyclist's experience.
- OPFA Way: They all share a single "Maze Map" (the latent space). If the bicyclist learns a shortcut, the skateboarder and unicyclist instantly understand the concept of that shortcut, even though their vehicles are different. They just have to figure out how to apply it to their specific wheels.

Why This is a Big Deal

The researchers tested this on 11 different types of robot hands (from simple clamps to complex 5-finger hands) and found amazing results:

Data Efficiency: Usually, teaching a new robot takes hundreds of examples. With OPFA, if you give a new robot just 8 examples (a tiny amount of data), it performs almost as well as a robot that has seen 72 examples. It's like learning a language by reading a few pages instead of a whole dictionary.
Zero-Shot Transfer: A robot trained on data from a "two-finger" hand can successfully perform tasks in a workspace it has never seen, simply because it learned the geometry of the task from the other robots.
No Custom Tuning: You don't need to write special code or retrain the system for every new robot you buy. You just plug it in, and the "Universal Translator" handles the rest.

In Summary:
OPFA stops robots from being stuck in their own "body silos." It teaches them to think about what they are doing (the shape and goal) rather than how their specific body parts move. This allows a robot with a simple gripper to learn from a robot with a fancy hand, making robot learning faster, cheaper, and much more adaptable to the real world.

1. Problem Statement

Robot manipulation learning faces a critical bottleneck: data scarcity and poor scalability across different robot embodiments (end-effectors).

Morphological Diversity: Robots range from simple parallel-jaw grippers to highly articulated dexterous hands (5+ fingers). These differences result in vastly different action spaces (dimensions and kinematic structures), making it difficult to jointly train a single policy on data from multiple sources.
Data Inefficiency: Collecting large-scale demonstration data for every new robot is costly and time-consuming.
Limitations of Existing Methods: Current cross-embodiment approaches often rely on:
- Unifying action spaces to a standard (e.g., end-effector pose), which fails for complex dexterous hands.
- Training separate, embodiment-specific decoders. This prevents the sharing of geometric knowledge between robots and leads to overfitting in few-shot scenarios (where data for a new robot is scarce).

2. Methodology: One-Policy-Fits-All (OPFA)

OPFA proposes a general framework that enables end-to-end co-training of diverse embodiments without manual annotation or embodiment-specific decoder tuning. The framework operates in two main stages:

A. Geometry-Aware Latent Representation (GaLR)

The core innovation is learning a shared latent action space that captures the geometric structure of the end-effector's reachable states.

Input Generation: Instead of using raw joint angles directly, the system samples reachable states (joint angles) and converts them into 3D point clouds representing the physical configuration of the hand/gripper using forward kinematics.
Encoder Architecture:
- 3D Convolution (KPConv): Extracts multi-scale local geometric features from the point cloud.
- Geometric Transformer: Processes "superpoints" (downsampled points) to capture global gesture information.
- Semantic Positional Embedding: A novel embedding scheme that assigns a 2D index (finger-level and segment-level) to each point. This allows the model to understand the structural hierarchy (e.g., "thumb base," "index finger tip") across different hands, enabling embodiment-agnostic geometry reasoning.
Output: The encoder produces a unified latent vector ( $z \in \mathbb{R}^{d_{latent}}$ ) that represents the action in a shared geometric space, regardless of the number of joints in the specific robot.

B. Unified Latent Retargeting Decoder

To recover specific actions from the shared latent space:

Universal Hand Model: The decoder is trained on a hypothetical "universal hand" containing all physically meaningful joints across all supported embodiments.
Retargeting: During inference for a specific robot $m$ , the decoder predicts the full universal joint set, and only the joints relevant to robot $m$ are extracted.
Training: The system is trained end-to-end using an RMSE loss between the predicted joint angles and the ground truth, without needing separate decoders for each robot.

C. Policy Integration

The pretrained GaLR encoder and unified decoder are integrated into downstream policy architectures (e.g., Diffusion Policy 3 - DP3).

Training: The policy predicts the latent action $z$ (GaLR) instead of raw joint angles.
Inference: The policy outputs $z$ , which is passed through the unified decoder to generate the specific joint angles for the target robot.

3. Key Contributions

Geometry-Aware Latent Representation (GaLR): A novel method to align action spaces across diverse end-effectors by encoding reachable states as 3D point clouds, capturing geometric commonalities without manual annotation.
Unified Latent Retargeting Decoder: A single decoder capable of handling arbitrary degrees of freedom (DOF) and structural variations, eliminating the need for embodiment-specific tuning and enabling effective knowledge transfer.
End-to-End Cross-Embodiment Co-training: A framework that allows joint training of data from 11 different embodiments (grippers and dexterous hands), significantly improving data efficiency and generalization.

4. Experimental Results

The authors evaluated OPFA on 11 different end-effectors (ranging from 2-finger grippers to 5-finger hands) across 14 manipulation tasks (including pouring, sweeping, and tool use) in both simulation and the real world.

Cross-Embodiment Generalization:
- In spatial generalization tests (training on one region, testing on another), OPFA achieved success rates >90% on tasks like bucket-lifting and banana-picking, whereas baselines failed completely or performed poorly.
- Compared to "Naive Co-training" (separate decoders), OPFA improved success rates by >50% in many cross-embodiment settings.
Few-Shot Learning:
- OPFA demonstrated remarkable data efficiency. When introducing a new robot with only 8 demonstrations, OPFA achieved performance comparable to a model trained with 72 demonstrations.
- In contrast, naive co-training often degraded performance as sample sizes increased due to action conflicts between different geometries.
Real-World Performance:
- On real-world tasks (e.g., manipulating deformable tissues, using tools like brooms/syringes), OPFA consistently outperformed baselines.
- Notably, on the "Pot" task (pouring), naive co-training failed due to structural conflicts, while OPFA successfully transferred fine-grained control skills.

5. Significance

Scalability: OPFA provides a scalable solution to the data scarcity problem in robotics. It allows developers to pool data from heterogeneous robots to train a single, versatile policy.
Rapid Adaptation: The ability to adapt a policy to a new robot with minimal data (few-shot) drastically reduces the cost of deploying robots in new environments or with new end-effectors.
Beyond Grippers: Unlike previous works focused on simple grippers, OPFA successfully bridges the gap for dexterous hands, handling high-dimensional action spaces and complex kinematic structures.
Foundation for Generalist Robots: By decoupling policy learning from specific hardware, OPFA moves the field closer to "One-Policy-Fits-All" generalist robots capable of operating across diverse physical forms.