💻 computer science

DexFormer: Cross-Embodied Dexterous Manipulation via History-Conditioned Transformer

DexFormer is an end-to-end, history-conditioned transformer policy that enables a single model to generalize across diverse dexterous hand embodiments by inferring morphology and dynamics from temporal context, achieving strong zero-shot transfer without requiring separate training for each robot.

Original authors: Ke Zhang, Lixin Xu, Chengyi Song, Junzhe Xu, Xiaoyi Lin, Zeyu Jiang, Renjing Xu

Published 2026-02-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ke Zhang, Lixin Xu, Chengyi Song, Junzhe Xu, Xiaoyi Lin, Zeyu Jiang, Renjing Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot hand to pick up a cup. Now, imagine you have three different robot hands: one looks like a human hand, one looks like a spider, and one looks like a mechanical claw.

In the past, if you wanted a robot to use all three, you had to teach them separately. You'd have to sit down with the "human" hand, teach it how to move, then start over with the "spider" hand, and again with the "claw." It was like hiring three different tutors for three different students, even though they all needed to learn the same lesson: "Pick up the cup."

DexFormer is a new method that changes this. It's like hiring one super-smart tutor who can instantly figure out which student they are talking to just by listening to how that student moves, and then teaching them all at the same time.

Here is how it works, broken down into simple concepts:

1. The Problem: "One Size Does Not Fit All"

Robot hands come in all shapes and sizes. Some have 3 fingers, some have 5, some have 12 joints, and some have 20. They move differently, weigh different amounts, and react to touch in unique ways.

The Old Way: If you built a robot controller for a 5-finger hand, it would fail miserably on a 3-finger hand. You had to rebuild the brain for every new hand.
The Barrier: This made robots expensive and slow to adapt. If a factory bought a new type of hand, they had to start training from scratch.

2. The Solution: The "History-Conditioned" Brain

The researchers built a system called DexFormer. Think of it as a robot brain that doesn't just look at the current moment (like "I see a cup"), but looks at the last few seconds of what happened (like "I moved my thumb left, then right, then felt a bump").

The Analogy: Imagine you walk into a room and see a stranger. You don't know who they are. But if you watch them walk for five seconds, you might notice they limp, or they walk with a long stride, or they carry a heavy bag. You can guess their physical traits just by watching their history of movement.
How DexFormer does it: It watches the robot hand's recent movements and "feels" the hand's shape and weight without being told what hand it is. It infers the "morphology" (the shape and physics) on the fly.

3. The Training: The "Randomized Gym"

To teach this brain to be so smart, the researchers didn't just train it on three specific hands. They created a massive, procedurally generated gym.

They took standard robot hands and randomly tweaked them: making fingers longer, heavier, or lighter, or changing how stiff the joints were.
They trained the AI on 300 different versions of these hands all at once.
The Result: The AI learned a "universal rule" for grasping. It didn't memorize specific hand shapes; it learned how to adapt to any shape it encountered.

4. The "Zero-Shot" Magic

The most impressive part is what happens when they test the robot on hands it has never seen before.

They took the trained AI and gave it a Leap Hand, an Allegro Hand, and a Rapid Hand. These were specific models the AI hadn't been explicitly trained on.
The Result: The AI worked immediately. It didn't need to be re-tuned or re-programmed. It looked at the new hand, watched it move for a split second, figured out its physics, and started picking up objects successfully. This is called "zero-shot transfer."

5. Real-World Testing

They didn't just keep this in a computer simulation. They put the system on a real robot arm with a real Leap Hand.

They used cameras to see the hand and the object (just like in the simulation).
The robot successfully lifted a cube and moved it to a target spot.
This proves the "brain" learned in the computer world could actually work in the messy, real world.

Summary

DexFormer is like a universal remote control for robot hands. Instead of needing a different remote for every TV (robot hand), you have one remote that learns the TV's settings just by watching how the buttons are pressed. It allows a single robot brain to control many different types of hands, making robots more flexible, cheaper to build, and easier to upgrade.

What the paper does NOT claim:

It does not claim this works for every possible robot hand in existence (only the ones tested and similar variations).
It does not claim the robot can do complex tasks like surgery or cooking yet; it focuses specifically on grasping and moving objects.
It does not claim this solves the problem of the robot understanding what the object is (like knowing a cup is for drinking); it only solves the problem of how to hold it regardless of the hand's shape.

Technical Summary: DexFormer

Problem Statement

Dexterous manipulation remains a significant challenge in robotics due to the need for coherent control of high-degree-of-freedom (DoF) hands and arms under complex, contact-rich dynamics. A primary barrier to scalability is embodiment variability: different dexterous hands possess distinct kinematics, inertias, actuator responses, and contact dynamics. Consequently, existing methods often require training separate policies for each hand or relying on shared action spaces with per-embodiment decoder heads. This specificity makes learning and sim-to-real transfer brittle, frequently necessitating repeated system identification, retuning, or additional real-world data collection for every new platform. Furthermore, existing approaches to bridge the sim-to-real gap—such as system identification (SysID), domain randomization, or residual modeling—either lack explicit mechanisms to compensate for embodiment-specific dynamics at test time or depend on collecting embodiment-specific real-world data, failing to achieve true zero-shot cross-embodiment transfer.

Methodology

The authors propose DexFormer, an end-to-end, dynamics-aware cross-embodiment policy built on a modified Transformer backbone. The core innovation is a history-conditioned approach that allows the policy to infer morphology and dynamics implicitly from temporal context, rather than relying on explicit embodiment identifiers or separate decoder heads.

Key Technical Components:

Shared Action Space: The authors define a canonical, morphology-invariant action space ( $D_F$ dimensions). Joints with the same functional role (e.g., thumb abduction, index flexion) are mapped to fixed indices across different embodiments. Lower-DoF hands utilize zero-padding for unused dimensions, while higher-DoF hands occupy more indices. This allows a single policy to output actions for heterogeneous hands.
Embodiment Generation: To train a morphology-agnostic policy, the authors construct a broad distribution of hand embodiments. They start with canonical hands (Allegro, LEAP, RAPID) and procedurally randomize morphology-related physical parameters (link lengths, masses, inertias) while preserving kinematic graph topology and actuation structure. This creates a "morphology-rich" training set ( $E_{train}$ ).
History-Conditioned Transformer: The policy conditions on a finite history window ( $H$ $H$ ) of observation-action pairs.
- Input: The history is tokenized and processed by a Transformer encoder with causal self-attention masks to prevent future information leakage.
- Inference: The final embedding serves as a compact summary of the history, implicitly inferring the latent morphology and dynamics of the current hand. This embedding is passed through an MLP action head to produce the control distribution.
- Observation Space: Includes object pose/quaternion, target pose, previous actions, joint positions/velocities, fingertip/palm states, contact forces, and object point clouds.
Training Framework: The authors utilize a distributed training framework (Isaac Lab) where randomized variants of the same canonical hand are grouped on single GPUs, while distinct canonical types are assigned to different GPUs. Distributed Data Parallel (DDP) with all-reduce gradient synchronization ensures a single shared policy is trained across this diverse morphology distribution.

Key Contributions

DexFormer Architecture: Introduction of a history-conditioned Transformer policy trained on a large scale of hand embodiments that performs implicit morphology inference for adaptive control without explicit embodiment encoding.
Scalable Training Pipeline: Development of a large-scale morphology-randomization pipeline and a distributed training framework specifically designed for cross-embodiment dexterous hand training.
Zero-Shot Generalization: Demonstration that a single morphology-agnostic policy can generalize to unseen dexterous embodiments and multiple real-world hands (Leap, Allegro, Rapid) without manual retargeting, explicit morphology encodings, or separate policy heads.

Experimental Results

The authors evaluated DexFormer through extensive simulation and real-world experiments:

Baselines Comparison: DexFormer consistently outperformed LSTM and GRU-based baselines in grasping success rates across LEAP, Allegro, and RAPID hands. For example, on the canonical LEAP hand, DexFormer achieved an 83.25% success rate compared to 66.81% for LSTM and 58.91% for GRU.
Zero-Shot Transfer: The policy demonstrated strong zero-shot generalization to 32 unseen variants of each canonical hand and the canonical hands themselves, which were held out during training.
Ablation Studies:
- Temporal Context: Using a 5-step history significantly improved performance over a 1-step history, particularly for Allegro and LEAP hands.
- Morphological Diversity: Increasing the number of distinct training embodiments (from 25 to 100 per canonical hand) substantially improved zero-shot performance, confirming that exposure to heterogeneity enhances generalization.
- Parallelism: Scaling the number of parallel environments per GPU generally improved performance, though some instability was noted for the RAPID hand, attributed to morphology statistic imbalances.
- Robustness: The policy showed resilience to joint locking, with performance degrading gracefully based on the anatomical importance of the locked joint.
Real-World Deployment: The policy was successfully deployed on a real-world Franka arm with a LEAP hand. Using offline distillation from 2000 expert demonstration episodes, the system achieved zero-shot transfer from simulation to reality, successfully lifting objects to target positions.

Significance and Claims

The paper claims that DexFormer establishes a scalable foundation for cross-embodiment dexterous manipulation. By leveraging temporal context to implicitly infer physical structure, the approach eliminates the need for embodiment-specific tuning or explicit kinematic modeling. The results suggest that dynamics-aware temporal inference is an effective mechanism for cross-embodiment generalization. The authors posit that this work bridges the gap between large-scale Reinforcement Learning (RL) training and history-conditioned models, offering a path toward unified manipulation policies applicable across diverse robotic embodiments and deployment scenarios.

The authors remain modest regarding limitations, noting that current training is limited to specific object geometries and that higher-DoF embodiments (like RAPID) still show modest performance drops in zero-shot settings. They identify future directions in foundation-style models, multi-expert distillation, and improved gradient aggregation strategies for mixed kinematic topologies.