💻 computer science

SPIDER: Scalable Physics-Informed Dexterous Retargeting

SPIDER is a scalable, physics-informed retargeting framework that transforms kinematic-only human motion data into dynamically feasible robot trajectories, significantly improving policy learning efficiency and success rates across diverse humanoid and dexterous hand embodiments.

Original authors: Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi, Jitendra Malik, Francois Hogan

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaoyi Pan, Changhao Wang, Haozhi Qi, Zixi Liu, Homanga Bharadhwaj, Akash Sharma, Tingfan Wu, Guanya Shi, Jitendra Malik, Francois Hogan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to perform a complex task, like pouring a cup of tea or playing a guitar. You have a library of thousands of videos showing humans doing these things perfectly. But there's a problem: robots and humans are built very differently.

If you simply copy a human's hand movements onto a robot, the robot might fail. Why? Because humans have muscles and balance that robots don't, and robots have rigid joints and different finger shapes. A human movement that looks smooth on video might cause a robot to crash into a table, drop the cup, or get stuck because the physics don't add up.

This is the problem the paper SPIDER solves.

The Core Idea: The "Physics Translator"

Think of SPIDER as a super-smart translator that doesn't just translate words, but translates physics.

The Input (The Human Script): You give SPIDER a video or motion-capture data of a human doing a task. This is just "kinematic" data—it tells you where the hand went, but not how hard it pushed or if it actually touched the object correctly.
The Problem (The Embodiment Gap): If you try to run this "human script" on a robot, the robot might try to walk through a wall or grab a cup with a grip that breaks the cup. It's like trying to drive a Formula 1 car using the steering instructions for a bicycle.
The SPIDER Solution (The Simulation Lab): SPIDER takes that human script and runs it through a virtual physics lab (a simulator). It asks: "If a robot tried to do this, would it work?"
- If the answer is no, SPIDER tweaks the movement.
- It uses a method called "Sampling" (trying thousands of tiny variations at once) to find a version of the movement that obeys the laws of physics.
- It uses "Virtual Contact Guidance" (a clever trick) to make sure the robot's fingers actually stick to the object the way the human did, rather than slipping or grabbing the wrong spot.

The Creative Analogy: The "Ghost Hand" and the "Sticky Tape"

Imagine you are trying to teach a clumsy robot arm to pick up a delicate egg.

The Human Demo: You show a video of a human gently picking up the egg.
The Robot's Confusion: The robot tries to copy the hand shape, but its fingers are too big and stiff. It smashes the egg.
SPIDER's "Ghost Hand": SPIDER creates a "ghost" version of the robot in a computer. It tries the human movement, sees the smash, and says, "Nope, that's illegal physics." It tries again, and again, millions of times in parallel, until it finds a way for the robot to hold the egg without breaking it.
The "Sticky Tape" (Virtual Contact): Sometimes, the robot doesn't know where to touch the egg. SPIDER puts a piece of invisible "virtual tape" between the robot's finger and the egg in the simulation. This tape gently pulls the robot's finger toward the correct spot on the egg. As the robot gets closer to the right answer, the tape gets weaker, letting the robot learn the natural grip on its own.

Why This is a Big Deal

The paper claims three major wins:

It's Fast: Traditional methods to fix these movements (like Reinforcement Learning) are like trying to teach a dog to dance by having it practice for years. SPIDER is like a coach who shows the dog the move once and corrects it instantly. The paper says SPIDER is 10 times faster than these older methods.
It Works Everywhere: They tested SPIDER on 9 different types of robots (from tiny dexterous hands to full-sized humanoid robots) and 6 different datasets of human movements. It worked for all of them, turning human data into robot data that actually works.
It Creates a Massive Library: Because it's so fast, SPIDER can take a small amount of human video and turn it into 2.4 million frames of high-quality robot data. This is like turning a single recipe book into a massive cookbook that teaches robots how to cook thousands of dishes.

The Result

Instead of spending years and millions of dollars collecting data by physically moving robots around (which is slow and expensive), researchers can now use SPIDER to take existing human videos, run them through this "physics translator," and instantly get a library of safe, feasible, and successful robot movements.

The paper concludes that this allows robots to learn complex skills much faster, bridging the gap between how humans move and how robots can actually move in the real world.

Technical Summary: SPIDER - Scalable Physics-Informed Dexterous Retargeting

Problem Statement

Learning dexterous and agile control policies for humanoid robots and dexterous hands typically requires large-scale, embodiment-specific demonstrations. However, collecting such robot data is prohibitively expensive due to hardware constraints and task complexity. While abundant human motion data exists (from motion capture, videos, and VR), a fundamental "embodiment gap" prevents its direct application to robots. This gap arises from mismatches in morphology, dynamics, and actuation, as well as the absence of critical dynamic information (e.g., force and torque) in human demonstrations. Consequently, direct kinematic transfer often results in dynamically infeasible motions, incorrect contact sequences, and physical violations (e.g., penetration). The central challenge is to efficiently and reliably transform kinematic-only human demonstrations into dynamically feasible robot trajectories that respect physical constraints and contact requirements at scale.

Methodology

The paper proposes SPIDER (Scalable Physics-Informed DExterous Retargeting), a sampling-based framework that bridges the gap between human kinematics and robot dynamics. The pipeline converts reconstructed human-object interaction trajectories into physically feasible robot trajectories through the following components:

1. Physics-Based Retargeting Formulation

The problem is formulated as a constrained optimization task. Given a kinematic reference trajectory $x^{ref}_{0:T}$ (comprising robot joint positions, base transformations, and object poses), the system optimizes a control sequence $u_{0:T-1}$ to minimize the distance to the reference trajectory and control effort, subject to the robot's state transition dynamics $x_{t+1} = f(x_t, u_t, t)$ .

2. Annealed Sampling Optimization

To solve the highly non-convex and non-continuous optimization landscape inherent in contact-rich manipulation, SPIDER employs a sampling-based optimizer. Unlike standard Reinforcement Learning (RL) which updates a policy network, SPIDER directly optimizes the control sequence using sampled trajectories from a parallelized physics simulator.

Annealed Kernel: The method utilizes an annealed sampling covariance schedule. It begins with a broad search radius to discover feasible contact modes (exploration) and gradually shrinks the radius to refine the solution (exploitation). This is controlled by parameters $\beta_1$ and $\beta_2$ which adjust the sampling variance across iterations and the prediction horizon.
Early Stopping: Optimization halts when improvements become negligible to enhance efficiency.

3. Virtual Contact Guidance

A key innovation to address solution ambiguity (where multiple contact modes achieve the same object motion but differ in naturalness) is Virtual Contact Guidance.

Mechanism: A virtual force is applied between the robot's fingers and the object to "stick" the object to the desired contact point during the initial stages of optimization. This constraint is gradually relaxed in a curriculum-style fashion as optimization progresses.
Robustness: To handle noisy human demonstrations, a contact filter disables virtual constraints for unstable interactions (e.g., contacts with short duration or significant drift), ensuring only reliable contacts guide the optimization.

4. Robustification and Augmentation

Trajectory Robustification: To bridge the sim-to-real gap, the framework employs a pessimistic (min-max) objective over a bounded set of dynamics parameters (e.g., friction, mass, contact margins). This ensures the generated trajectory remains feasible under worst-case parameter variations, similar to domain randomization but without the computational overhead of expectation-based methods.
Data Augmentation: The physics-based nature of SPIDER allows for systematic augmentation. Starting from a single demonstration, the system can generate diverse behaviors by varying object geometries (meshes, scales), environmental conditions (e.g., adding stairs), or applying external forces, all while maintaining physical feasibility.

Key Contributions

Scalable Framework: SPIDER is a flexible, general physics-informed retargeting framework demonstrated across six datasets, nine distinct robot morphologies (including 5 dexterous hands and 4 humanoids), and two task domains (dexterous manipulation and whole-body control).
Contact-Aware Sampling: The introduction of a virtual contact guidance mechanism that preserves manipulation intent and corrects contact sequences. This improves success rates by 18% compared to sampling baselines without guidance.
Efficiency and Scale: The method is 10 $\times$ faster than Reinforcement Learning baselines, enabling the generation of a 2.4 million frame dynamic-feasible robot dataset derived from human data.
Robustness to Noise: The pipeline successfully converts noisy, single-RGB camera reconstructions (with artifacts and penetrations) into executable real-robot trajectories.

Experimental Results

Dexterous Manipulation: Evaluated on the Oakink, GigaHands, and ARCTIC datasets using Allegro, XHand, Inspire, Ability, and Schunk hands. SPIDER achieved success rates up to 1.00 on GigaHands tasks and significantly outperformed kinematic retargeting and vanilla sampling.
Humanoid Control: Applied to Unitree G1, Unitree H1-2, Fourier N1, and Booster T1 robots using LAFAN1, AMASS, and OMOMO datasets. SPIDER reduced joint tracking errors and pelvis orientation errors significantly compared to kinematic-based General Motion Retargeting (GMR), while maintaining competitive generation speeds (approx. 20-23 FPS).
Real-World Deployment: The method was deployed on a Franka Emika Panda with an Allegro hand for tasks like rotating a lightbulb and unplugging a charger, demonstrating direct transferability of simulated trajectories to real hardware without additional policy training.
RL Policy Training: When used as a reference for training RL policies on humanoids, SPIDER-generated trajectories led to faster convergence and better object tracking performance compared to policies trained on raw kinematic references.

Significance and Claims

The paper positions SPIDER as a universal, physics-based retargeting method that addresses the data scarcity problem in robot learning. Its primary significance lies in its ability to:

Leverage Human Data at Scale: Transform abundant, low-cost human motion data into high-quality, physically feasible robot demonstrations without the need for expensive robot-specific data collection or tedious reward engineering.
Bridge the Embodiment Gap: Systematically resolve the morphological and dynamic mismatches between humans and robots through physics-grounded optimization.
Enable Efficient Policy Learning: Provide a "drop-in" replacement for human-to-robot transfer components, generating diverse, high-quality datasets that accelerate downstream Reinforcement Learning and behavior cloning.

The authors note that while SPIDER is highly general, its performance is contingent on the quality of the upstream 3D reconstruction; noisy meshes and motion data can still degrade the resulting trajectories. However, the framework's robustification strategies mitigate many of these issues, making it a practical tool for scaling robot learning from human demonstrations.