🤖 machine learning

Cross-Embodiment Transfer via Behavior-Aligned Representations

This paper proposes using behavior-aligned representations, particularly end-effector traces, within vision-language-action models to unify diverse cross-embodiment data and significantly improve robot transfer performance, achieving a 28% increase in real-world task completion after simulation pre-training.

Original authors: Ajay Sridhar, Jensen Gao, Jonathan Yang, Jean Mercat, Suneel Belkhale, Dorsa Sadigh

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ajay Sridhar, Jensen Gao, Jonathan Yang, Jean Mercat, Suneel Belkhale, Dorsa Sadigh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do chores. In the world of machine learning, there's a popular idea that if you feed a model a massive amount of data from all over the place, it becomes smarter than if you only fed it data from a single, specific source. This is the same reason why a human who travels the world and learns many languages can often adapt to new situations better than someone who has only ever lived in one small town. For robots, this means we want to train them on data collected from many different types of robots—some with long arms, some with short ones, some with grippers that look like claws, others like pincers. This is called "cross-embodiment" learning. The goal is to create a "universal" robot brain that can learn from a library of experiences from many different machines and then apply that knowledge to a brand-new robot it has never seen before. However, there's a big problem: a camera on a tall robot sees the world from a high angle, while a short robot sees it from the floor. Their "hands" move differently, and their action commands are written in different "languages." Trying to mix these messy, mismatched datasets is like trying to teach a student to drive by showing them videos of cars, trucks, and bicycles all at once without explaining the differences. It's confusing, and often the robot just gets stuck.

This paper, titled "Cross-Embodiment Transfer via Behavior-Aligned Representations," tackles that confusion by asking a clever question: instead of trying to force the robots to look or move exactly the same, can we teach them to speak a common "behavior language"? The authors, a team from Stanford and the Toyota Research Institute, propose that we can bridge the gap between different robots by focusing on the story of the action rather than the specific mechanics. They suggest using "behavior-aligned representations"—which are basically simplified summaries of what is happening, like a text description of the movement ("move left and down"), a box drawn around the object being picked up, or a line tracing the path the robot's hand takes through the air.

The researchers tested this idea using a new simulation benchmark they built called RoboCasa-X. They trained their robot brains (specifically a type of model called a Vision-Language-Action model) on huge datasets from three different source robots. Then, they tried to transfer that knowledge to four different target robots that were not in the training data. They compared training with just raw data against training with these extra "behavior summaries" added in.

The results were quite revealing. The study found that adding these behavior summaries significantly helped the robots transfer skills to new bodies. Among the different types of summaries they tested, the "end-effector traces"—which are essentially the 2D paths the robot's hand draws in the camera's view—were the most powerful tool. When the robots were trained with these traces, they became much better at adapting to new hardware. In fact, when they took a policy pre-trained on simulation data and tested it on real-world robots, the use of these representations improved the task completion progress by 28%.

Interestingly, the paper also discovered that you don't necessarily need the robot to "think" about these traces while it is actually doing the task. The models learned best when they were trained to predict these traces alongside the actions, but during the actual job, simply predicting the action directly worked just as well. This suggests the traces helped the robot learn the underlying logic of the task during training, acting like a training wheel that could be removed later. Furthermore, the team showed that these representations could even help robots learn from data that didn't include action commands at all (action-free data), as long as the behavior summaries were present.

The authors are careful to note that while their method works well in their specific simulation and real-world tests, it relies on the robots having some level of similarity in their tasks and camera setups. They didn't prove this works for every possible robot mismatch, but their experiments strongly suggest that giving robots a shared "behavioral vocabulary" is a promising way to make them more adaptable. By focusing on the "what" and "how" of the movement rather than the specific "who" (the robot model), they found a way to make the vast, messy library of robot data actually useful for building smarter, more flexible machines.

Technical Summary: Cross-Embodiment Transfer via Behavior-Aligned Representations

Problem Statement

Recent advancements in robot manipulation have been driven by large-scale imitation learning across diverse robot embodiments. However, achieving significant cross-embodiment transfer—where a policy trained on data from one set of robots generalizes to a new, unseen robot platform—remains a persistent challenge.

The primary obstacle is the substantial variation in observation spaces (e.g., camera poses, visual inputs) and action spaces (e.g., joint configurations, end-effector kinematics) across different embodiments. While prior works have attempted to address this by explicitly aligning observations (e.g., via camera pose normalization or inpainting) or action spaces, these methods often require significant manual effort, specific knowledge of the deployment-time robot during training, or are difficult to scale. Consequently, performance often fails to significantly exceed training solely on data specific to the target embodiment.

Methodology

The authors propose a framework that implicitly aligns heterogeneous datasets using behavior-aligned representations. The core hypothesis is that while raw observations and actions vary across robots, intermediate representations describing the behavior of the task (e.g., "move left," "grasp object at [x,y]") can remain invariant across embodiments. By training Vision-Language-Action (VLA) models to predict these representations alongside actions, the policy can leverage a unified reasoning space.

Framework and Formalization

The approach is formulated within a language-conditioned imitation learning setting. The policy $\pi_\theta$ is trained to map observations $o$ and language instructions $l$ to actions $a$ .

Behavior-Aligned Representations ( $z$ ): The authors define a set of representations $z = (z^{(1)}, \dots, z^{(K)})$ that are computable from the dataset, useful for predicting optimal behavior, and invariant across embodiments.
Training Objective: The model is trained to autoregressively predict a sampled subset of representations $\tilde{z}$ followed by actions. The total loss combines the behavior cloning loss (action prediction) with auxiliary losses for predicting the representations:
$\mathcal{L}_{total} = \mathbb{E}[\ell(\pi_\theta(\cdot | o, l, \tilde{z}), a) + \sum \lambda_k \ell_{rep}^{(k)}(\pi_\theta(\cdot | o, l), z^{(k)})]$
During inference, the model can be configured to predict actions directly or condition on predicted representations.

Specific Representations Studied

The paper evaluates three specific types of behavior-aligned representations:

Bounding Boxes: 2D object bounding boxes for manipulatable objects (extracted via Grounding DINO).
Language Motions: Textual descriptions of the immediate motion (e.g., "move left and down"), derived from proprioceptive state changes.
End-Effector (EE) Traces: Sequences of future 2D positions of the robot's end-effector in the image frame.

Benchmark: RoboCasa-X

To rigorously evaluate these hypotheses, the authors introduce RoboCasa-X, a simulation-based benchmark extending RoboCasa.

Source Data: Pre-training on diverse data from three source robots (IIWA, Kinova3, UR5e) with varying grippers.
Target Data: Transfer evaluation on three target robots (Panda, Jaco, Panda-OG) with limited target demonstrations (50 per task).
Tasks: Four manipulation tasks (e.g., "Pick and Place Counter to Sink," "Turn On Sink Faucet") featuring significant variation in layouts, textures, and object poses.

Key Contributions

Implicit Alignment via Representations: The paper demonstrates that behavior-aligned representations offer a scalable, implicit method for unifying cross-embodiment data, avoiding the need for explicit observation or action space alignment.
Empirical Analysis of Representations: Through extensive ablation studies, the authors identify that end-effector traces are the most impactful representation for cross-embodiment transfer, followed by language motions and bounding boxes.
Scaling Laws: The study reveals that the utility of these representations scales with the size of the prior cross-embodiment dataset; larger datasets (e.g., 3,000 demonstrations) yield greater benefits from representation alignment than smaller ones.
Action-Free Transfer: The framework enables transfer from action-free datasets. The authors show that pre-training on data where only representations are predicted (without action labels) still improves downstream performance when fine-tuned on action data, suggesting the representations induce a useful internal state.
Sim-to-Real Validation: The approach is validated on real-world robots, demonstrating that representations can bridge the sim-to-real gap while simultaneously handling cross-embodiment transfer.

Experimental Results

Simulation Results (RoboCasa-X)

Performance Gains: Using behavior-aligned representations consistently improves success rates over baselines with no representations.
- End-Effector Traces: Provided the largest individual boost.
- Combined Representations: Using "Joint Reps" (training a single model to predict any subset of representations) generally achieved the highest overall success rates across most tasks and target robots (Panda, Jaco).
- Exception: For the Panda-OG target (which has a unique gripper not seen in pre-training), end-effector traces alone outperformed combined representations, suggesting the optimal representation choice depends on the specific embodiment gap.
Scaling: Representations provided a +19% improvement when scaling from 900 to 3,000 prior demonstrations, whereas the benefit was smaller with single-embodiment data.
Inference: Predicting representations during inference provided only marginal gains over direct action prediction, supporting the hypothesis that the primary benefit is learned during training (implicit alignment).
Action-Free Data: Pre-training on action-free data (predicting only representations) improved performance by 14% over training from scratch and 11% over using full action data without representations.

Real-World Results

Sim-to-Real Transfer: The authors adapted models pre-trained on RoboCasa-X (XP-3K) to two real robots: Franka Research 3 (FR3) and ViperX 300 S.
Significant Improvement: Despite significant domain gaps in visual observations and action spaces (e.g., different control frequencies and blocking behaviors), the "Joint Reps" approach improved task completion progress by 28% compared to baselines without representations.
Comparison: Representations were more beneficial in the real-world setting than in simulation, likely due to the larger domain gap making implicit alignment more critical.

Significance and Claims

The paper claims that behavior-aligned representations are a crucial mechanism for enabling scalable cross-embodiment learning. By providing a common "language" of behavior (traces, boxes, language) that is invariant to the specific robot hardware, these representations allow policies to leverage massive, heterogeneous datasets that would otherwise be unusable due to embodiment mismatches.

The authors position this work as a step toward understanding "positive transfer" in cross-embodiment learning. They emphasize that while their current setting involves significant alignment (e.g., consistent camera distributions in simulation), the framework provides a starting point for exploring transfer in less structured, more misaligned environments. The work suggests that future research should investigate how different levels of misalignment influence the efficacy of specific representations and explore a broader range of behavior-aligned signals beyond object-centric manipulation.