💻 computer science

Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots

This paper proposes a bridging action representation based on relative wrist translation within a head-camera frame, combined with a vision-language-action model, to effectively transfer diverse human manipulation skills to bi-manual robots by overcoming the limitations of noisy 6DoF pose estimation and fundamental embodiment differences.

Original authors: Sijin Chen, Kaixuan Jiang, Haixin Shi, Yanhui Wang, Weiheng Zhong, Haosheng Li, Bo Jiang, Yuxiao Liu, Xihui Liu

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sijin Chen, Kaixuan Jiang, Haixin Shi, Yanhui Wang, Weiheng Zhong, Haosheng Li, Bo Jiang, Yuxiao Liu, Xihui Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a clumsy robot how to perform complex kitchen chores, like opening a microwave, stacking cups, or wiping a counter. You have two options: spend thousands of dollars recording the robot doing these tasks (which is slow and expensive), or record humans doing them (which is cheap, easy, and abundant).

The big problem is that humans and robots are built very differently.

Humans have flexible fingers that can twist, turn, and grip in infinite ways.
Robots in this study have "parallel grippers"—basically two stiff pincers that just open and close.

If you try to copy a human's hand movements directly onto a robot, it's like trying to teach a dog to play the piano by showing it a human's fingers. The robot gets confused because the "twisting" motions humans use don't make sense for its stiff pincers. The robot ends up flailing around or doing the wrong thing.

The Solution: "Translation as a Bridging Action"

The researchers came up with a clever trick called a "Bridging Action." Instead of trying to copy the entire human hand (including the confusing twists and turns), they decided to only copy the straight-line movement of the wrist.

Think of it like this:

The "Noisy" Way (Old Method): Trying to copy the exact 3D rotation and angle of a human hand. This is like trying to translate a poem word-for-word without understanding the meaning; it sounds right but makes no sense in the new language.
The "Bridging" Way (New Method): Ignoring the rotation and only looking at where the hand moves in space (left, right, up, down). This is like translating the meaning of the poem rather than the specific words.

By focusing only on translation (moving from point A to point B) and ignoring the rotation (twisting), the researchers found a "common language" that both humans and robots can speak.

How They Built the "Translator"

To make this work, they built a special AI model (a Vision-Language-Action model) that acts like a universal translator. Here is how it works in three simple steps:

The "Bridging" Signal: They extract the human's wrist movement as a simple 3D path (like drawing a line on a map). This is the "bridge" because both humans and robots can understand "move forward" or "move left," even if they move differently.
The "Interleaved" Training: The AI is trained on a mix of data. Sometimes it sees a human moving their hand (and only learns the path). Sometimes it sees a robot moving its arm (and learns the full path plus the pincer grip). The AI is smart enough to handle missing pieces of information, kind of like a chef who can cook a meal even if they are missing one specific spice, by using what they have.
The "Binding" Trick: To make sure the robot actually learns to move its pincers correctly, the AI is forced to practice predicting the robot's full movement using the simple human path as a guide. It's like a student learning to drive by first practicing steering on a simulator, then switching to a real car but keeping the same steering logic.

What They Found

They tested this on 15 different tasks, from opening a microwave to stacking cups. Here is what happened:

Without the Bridge: If they just tried to copy the robot's own limited data, it failed almost everything.
With the Bridge (Human Data): When they used the "translation-only" human data to teach the robot, the robot suddenly got much better. It could open microwaves and stack cups it had never seen before.
The "Rotation" Problem: When they tried to use the full, noisy human hand rotations (the "Noisy" way), the robot's movements became distorted and clumsy, like a puppet with tangled strings. The simple "translation" method was far superior.
The "Upper Limit": They also tested what would happen if they fed the robot perfect robot data but treated it like human data (ignoring the robot's specific camera angles). The robot got even better, suggesting that if we can get cleaner data, this method could be even more powerful.

The Bottom Line

The paper proves that you don't need to perfectly mimic a human's body to teach a robot. You just need to find the shared movement (the path the hand takes) and ignore the parts that don't fit (the twisting fingers).

By using this "Bridging Action," the researchers successfully transferred skills from cheap, abundant human videos to a robot, allowing the robot to learn complex tasks without needing thousands of expensive robot demonstrations. It's a way of saying: "Don't worry about how the human holds the cup; just teach the robot where to put it."

Technical Summary: Translation as a Bridging Action

Problem Statement

The paper addresses the challenge of transferring novel manipulation skills from human data to bi-manual robots equipped with parallel grippers. While human action data is abundant, diverse, and cost-effective for scaling robot learning, direct transfer is hindered by two primary issues:

Noisy Estimation: Human actions are typically derived from hand-pose estimators, which introduce significant noise, particularly in rotation estimates.
Embodiment Mismatch: The contact patterns of human fingers differ fundamentally from parallel robot grippers. Consequently, wrist rotations (6DoF) extracted from human data are often semantically misaligned with the physical constraints of a robot gripper, leading to distorted behaviors when directly replayed.

Prior approaches often treat humans as just another bi-manual 6DoF embodiment, learning rotation-inclusive action signals. The authors argue this is sub-optimal for parallel gripper robots.

Methodology

1. Bridging Action Representation

Instead of learning full 6DoF wrist poses, the authors propose a bridging action representation based solely on relative wrist translations ( $a_{3D-wrist}$ ).

Definition: The action is defined as the relative translation of the wrist within the initial head-camera frame. This frame is shared by both humans (wearing head-mounted cameras) and robots (equipped with head cameras).
Rationale: Since both humans and robots act based on what they perceive, relative translation in the observation frame is physically meaningful, robust to noisy rotation estimates, and embodiment-agnostic.
Formulation: For a time step $t$ and future window $k$ , the action is calculated as:
$a_{3D-wrist}^{t+i} = \Delta W^{3D} = \text{trans}((T_{w \leftarrow c}^t)^{-1} W_w^{t+i}) - \text{trans}((T_{w \leftarrow c}^t)^{-1} W_w^t)$
where $W$ represents the wrist pose and $T$ the camera pose.

2. Unified Model Architecture

The authors build upon a $\pi_0$ -like Vision-Language-Action (VLA) model with flow matching. To handle heterogeneous data sources (human vs. robot) with different available action components, they introduce an interleaved action sequence:

Sequence Order: $a_{3D-wrist} \rightarrow a_{6D-eef} \rightarrow a_{gripper}$ .
Handling Missing Data: The model uses attention masking to handle variable-length inputs. For human data, 6DoF end-effector ( $a_{6D-eef}$ ) and gripper signals may be missing or masked; for robot data, all components are present.
Training Objectives:
- Flow Matching: Used to generate action chunks. The loss is computed only on the available action components for a given data source.
- Interleaved Supervision: Crucially, during human-robot co-training, the model is trained to predict $a_{3D-wrist}$ even when the ground truth is $a_{6D-eef}$ (and vice versa). This "binding" strategy explicitly links the bridging representation to executable robot actions.

3. Three-Stage Training Strategy

Stage I (Pre-training): The model is pre-trained on ~600 hours of large-scale human action data (EgoDex + outsourced + in-lab). Only the bridging signal ( $a_{3D-wrist}$ ) is supervised, as rotation and contact patterns are unreliable at this scale.
Stage II (Co-training): The model is co-trained on generalized robot pick-and-place data (~~72 hours) and task-specific in-lab human data (~~3 hours/task). All action components ( $a_{3D-wrist}$ , $a_{6D-eef}$ , $a_{gripper}$ ) are active, with random substitution of $a_{3D-wrist}$ for $a_{6D-eef}$ to enforce the bridging connection.
Stage III (Few-shot Post-training): The model is fine-tuned on a small set of real-robot demonstrations (10 trajectories/task) to improve data efficiency and final performance.

Key Results

The method was evaluated on 15 novel bi-manual manipulation tasks (e.g., opening microwaves, wiping surfaces, stacking cups) using the ByteMini robot.

Skill Transfer: Training solely on robot pick-and-place data failed to generalize to complex tasks. However, co-training with human data using the bridging action significantly improved both task progress and success rates.
Translation vs. 6DoF: Co-training with 6DoF human actions resulted in noisy, distorted behaviors. The translation-only bridging action yielded stable, natural poses aligned with task requirements (e.g., aligning with microwave handles).
Scalability: Large-scale human-only pre-training (Stage I) substantially improved downstream performance, confirming the scalability of the bridging representation.
Data Efficiency: Models pre-trained on human data required fewer robot demonstrations (few-shot) to achieve high performance compared to models trained from scratch.
Alignment: The training loss for the bridging action ( $a_{3D-wrist}$ ) aligned closely with the loss for executable 6DoF actions ( $a_{6D-eef}$ ) during co-training, suggesting a shared objective landscape.
Upper Bound: When the embodiment gap was removed (treating robot data as human data with translation-only supervision), performance improved further, indicating the bridging representation itself is highly effective and the remaining gap is largely due to visual/embodiment differences and noise.

Limitations and Failure Cases

The authors note that the translation-only approach limits transfer in tasks requiring precise rotational adjustments during contact-rich manipulation (e.g., inserting a straw into a cup or opening a drawer). In these cases, the robot often demonstrates correct intent but fails at the final contact step due to the lack of rotational supervision. Additionally, picking up thin objects remains challenging, attributed to observation gaps and human action noise.

Significance and Claims

The paper claims to demonstrate that:

Robust Transfer: A translation-based bridging action is a superior medium for transferring manipulation knowledge from humans to parallel-gripper robots compared to noisy 6DoF pose estimates.
Scalability: This representation effectively leverages large-scale, diverse human data to pre-train policies, reducing reliance on expensive robot data collection.
Unified Learning: An interleaved action sequence with attention masking allows a single model to learn from heterogeneous data sources with missing action components, facilitating efficient human-robot co-training.

The work suggests that by focusing on the shared perceptual-motor signal (relative translation) rather than the specific embodiment details (finger joints or full 6DoF poses), robot learning can be scaled more effectively using abundant human data.