💻 computer science

DexTwist: Dexterous Hand Retargeting for Twist Motion via Mixed Reality-based Teleoperation

This paper introduces DexTwist, a mixed reality-based teleoperation framework that overcomes the embodiment gap in dexterous hand retargeting by detecting tripod pinches and applying real-time joint-space refinement to ensure stable object rotation during contact-rich twist motions like cap opening and screwing.

Original authors: Dongmyoung Lee, Chengxi Li, Dongheui Lee

Published 2026-05-13

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongmyoung Lee, Chengxi Li, Dongheui Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot hand how to unscrew a jar lid or turn a key. You put on a high-tech headset (Mixed Reality) that lets you see the robot's workspace and see your own hands as if they were the robot's hands. You make a twisting motion with your fingers, expecting the robot to copy you perfectly.

In the past, this often went wrong. Even if your hand looked exactly like the robot's hand, the robot would fail to turn the object. Instead of rotating the lid, the robot's fingertips would just slide around the edge, like trying to turn a doorknob with greasy fingers. The robot would lose its grip, the "screw axis" (the invisible line the object spins around) would drift, and the task would fail.

The Problem: The "Body Mismatch"
The paper calls this the "embodiment gap." Think of it like trying to wear a pair of shoes that are a different size and shape than your feet. If you try to walk exactly the same way, your feet might slip inside the shoes. Similarly, human hands and robot hands have different bone lengths, joint limits, and finger shapes. Simply copying the shape of your hand doesn't guarantee the robot will perform the function of turning the object.

The Solution: DexTwist
The authors created a new system called DexTwist. Instead of just telling the robot, "Copy my hand shape," DexTwist asks, "What are you trying to do?"

Here is how it works, using a simple analogy:

The "Tripod" Grip: When you want to twist something, you naturally use a "tripod" grip (thumb, index, and middle finger). DexTwist detects this specific grip.
Reading the Mind (The Intent): Instead of just looking at where your fingers are, the system calculates the direction you are trying to twist (the "screw axis") and how much you have turned so far. It's like the robot understands you are trying to unscrew a bottle, not just move your fingers in a circle.
The "Residual" Fix: This is the magic part. The system first gives the robot a basic instruction based on your hand shape. Then, it runs a quick, real-time "correction" (like a spell-checker for movement). It checks:
- "Did we turn the right amount?"
- "Is the robot still holding the object tightly?"
- "Is the robot's fingers sliding off the object?"
- "Is the robot's grip staying stable?"

If the robot is about to slip or drift, DexTwist subtly adjusts the robot's joint angles to keep the grip stable and the turning motion smooth, even if the robot's hand looks slightly different from yours.

The Results
The team tested this in computer simulations and with a real robot arm. They compared DexTwist to a standard method that just tries to copy hand shapes (called "Vector Retargeting").

Standard Method: Often resulted in the robot's fingers sliding around the object, losing the turning progress, or drifting off the correct axis.
DexTwist: Successfully tracked the turning angle much more accurately. The robot kept a stable grip and turned the object along a consistent line, just like a human would.

In Summary
DexTwist is like a smart translator for robot hands. It doesn't just translate your movements; it translates your intent. It ensures that when you twist a virtual knob, the robot actually turns the real knob, rather than just mimicking the shape of your hand and failing the task. This makes remote control of robots much more reliable for delicate jobs like opening caps, turning keys, or screwing bolts.

Technical Summary: DexTwist

Problem Statement

Dexterous teleoperation via Mixed Reality (MR) interfaces offers a scalable method for transferring human manipulation skills to robotic hands. However, conventional retargeting approaches, which typically minimize kinematic dissimilarity (e.g., joint angle or fingertip position errors), often fail in contact-rich rotational tasks such as cap opening, key turning, and bolt screwing.

This failure arises from the embodiment gap: mismatches in link lengths, joint axes, limits, and fingertip geometry between humans and robots mean that direct pose imitation does not guarantee correct contact behavior. Instead of generating stable rotation about a physical axis, direct kinematic mapping often induces tangential fingertip sliding along the object surface. This leads to screw axis drift, contact slip, and grasp instability, rendering gesture imitation insufficient for tasks where success depends on torque transmission and stable rotation rather than hand-shape similarity alone.

Methodology: DexTwist Framework

The authors propose DexTwist, an MR-based teleoperation system designed for functional hand retargeting during in-hand twist manipulation. The system integrates an MR headset for hand tracking and visual feedback with a specialized retargeting module that shifts focus from pose imitation to function preservation.

1. Teleoperation Interface

The system utilizes an MR headset (e.g., Apple Vision Pro) to capture human hand joint configurations ( $q_h$ ) and wrist poses. A local palm frame $\{P\}$ is constructed based on wrist and knuckle keypoints to serve as the reference for control. The interface streams real-time visual feedback from external workspace cameras to the headset, allowing operators to monitor the manipulation scene while issuing commands.

2. Functional Twist Retargeting

Unlike standard retargeting, DexTwist activates a specific mode when a tripod pinch (thumb–index–middle) is detected. The core logic involves:

Intent Estimation: The system estimates the operator's intended screw axis ( $a_h$ ) as the normal of the fingertip tripod and tracks the accumulated twist angle ( $\theta_{task}$ ) in the palm frame. This allows for "ratcheting" behavior, where turning progress persists across re-grasps.
Residual Joint-Space Refinement: The robot hand configuration ( $q$ $q$ ) is determined by minimizing a virtual-object objective ( $J(q)$ $J (q)$ ) defined in the robot's palm frame. This objective is a weighted sum of four terms:
1. Turning Progress: Minimizes the error between the robot's accumulated turning angle ( $\theta_r$ ) and the operator's intent ( $\theta_{task}$ ).
2. Fingertip Closure: Penalizes changes in pairwise fingertip distances ( $e_r$ ) to prevent grip opening.
3. Screw Axis Consistency: Penalizes deviations of the instantaneous robot tripod normal ( $a_r$ ) from the reference axis ( $a_{r,ref}$ ) to prevent axis drift.
4. Tripod Stability: Penalizes translation of the tripod centroid ( $c_r$ ) to maintain geometric stability.

This optimization is solved in real-time as a lightweight residual refinement (using finite-difference gradient descent) initialized from a baseline vector-retargeted configuration.

Key Contributions

The paper outlines three primary contributions:

MR-Based Teleoperation Pipeline: A robust system featuring tripod activation detection and an episode-level turning state that supports ratcheting (continuous turning across re-grasps).
Functional Twist Retargeting Module: A residual joint-space refinement strategy that minimizes a virtual-object objective. This approach tracks turning progress while regularizing tripod stability, specifically addressing the embodiment gap in rotational tasks.
Comprehensive Evaluation: A protocol comparing DexTwist against a vector-based retargeting baseline in both simulation and real-world hardware, demonstrating improvements in turning angle tracking and screw axis stability.

Experimental Results

The authors evaluated DexTwist using a Franka Emika FR3 arm with an Allegro Hand, controlled by an operator wearing an MR headset.

Simulation: Compared to a Vector Retargeting baseline, DexTwist significantly reduced tracking error to human intent (RMSE: 18.8° vs. 27.7°) and improved screw axis stability (Axis Deviation: 6.1° vs. 13.6°).
Real-World Experiments: Using an ArUco marker attached to the manipulated object as ground truth, DexTwist demonstrated superior performance in turning-angle tracking (RMSE: 14.3° vs. 23.2°) and higher temporal correlation with the object's rotation (0.96 vs. 0.83) compared to the baseline.
Qualitative Performance: The system successfully executed contact-rich tasks like cap screwing and key unlocking, maintaining stable fingertip contacts and consistent rotation axes even during re-grasping transitions.

Significance and Limitations

The authors position DexTwist as a solution for remote maintenance, inspection, disaster response, and industrial servicing, where operators must manipulate fasteners or valves without physical presence. By shifting from gesture imitation to function preservation, the framework enables reliable dexterous teleoperation despite embodiment mismatches.

The paper acknowledges specific limitations:

Scope: The current formulation is restricted to tripod-pinch twisting. It does not yet cover other rotational strategies like two/four-finger pinching, cylindrical grasps, or power grasps, which may be required for different object sizes or torque requirements.
Sensing: The system relies on visual hand tracking; severe self-occlusion can bias the inferred twist intent. The authors suggest that incorporating tactile or contact-state sensing could improve robustness against slip and unreliable visual updates.

The work concludes that while DexTwist improves reliability for specific twist primitives, extending it to broader manipulation strategies requires grasp-dependent functional representations derived from contact geometry or tactile feedback.