💻 computer science

Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies

The paper introduces SoftAct, a two-stage, force-aware retargeting framework that leverages immersive VR demonstrations to enable non-anthropomorphic soft robot hands to successfully reproduce human manipulation skills by explicitly modeling contact forces and geometry, thereby significantly outperforming kinematic-only baselines in both simulation and real-world zero-shot deployment.

Original authors: Uksang Yoo, Mengjia Zhu, Evan Pezent, Jom Preechayasomboon, Jean Oh, Jeffrey Ichnowski, Amir Memar, Ben Abbatematteo, Homanga Bharadhwaj, Ashish Deshpande, Harsha Prahlad

Published 2026-04-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Uksang Yoo, Mengjia Zhu, Evan Pezent, Jom Preechayasomboon, Jean Oh, Jeffrey Ichnowski, Amir Memar, Ben Abbatematteo, Homanga Bharadhwaj, Ashish Deshpande, Harsha Prahlad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very clumsy, squishy, balloon-like robot hand how to do delicate tasks, like screwing in a lightbulb or pouring water from a cup. The problem? The robot hand looks nothing like a human hand. It has no bones, no joints, and it bends in weird, unpredictable ways.

If you tried to teach it by simply copying the human's finger movements (like telling the robot, "bend your thumb exactly 30 degrees"), the robot would fail miserably. It's like trying to teach a octopus to play the piano by telling it to move its tentacles exactly like human fingers; the anatomy is just too different.

Enter "SoftAct": The Force-Following Teacher.

This paper introduces a new system called SoftAct. Instead of trying to copy how a human moves their fingers, SoftAct focuses on what the fingers are doing to the object: the force and the touch.

Here is how it works, broken down into simple steps with some analogies:

1. The "VR Dance Class" (Data Collection)

First, the researchers put humans in a Virtual Reality (VR) headset. The humans perform tasks like grabbing a bottle or twisting a cap.

The Magic: While the humans do this, the VR system doesn't just record their hand positions. It acts like a super-sensitive force sensor, recording exactly where the fingers touch the object and how hard they push.
Analogy: Imagine a dance instructor recording a student. Instead of just filming their steps, they are also measuring exactly how much pressure the student's feet apply to the floor at every moment.

2. Stage 1: The "Fair Share" Assignment

The robot hand has a different number of fingers or a different shape than the human. So, the system has to decide: "Which robot finger should do the job of the human thumb?"

The Solution: SoftAct looks at the "force map." If the human used their thumb and index finger to squeeze a cup with 50% of the total force, the system assigns the robot's strongest fingers to handle that 50% of the load.
Analogy: Imagine a group of people carrying a heavy piano. One person is huge, and the others are small. Instead of telling the small person to lift the same amount of piano as the big person, you tell them to lift a proportion of the weight that matches their strength. SoftAct does this with robot fingers.

3. Stage 2: The "Rubber Band" Adjustment

Once the robot starts moving, it doesn't just blindly follow a pre-set path. It constantly checks the "touch map."

The Solution: If the human's finger was pressing hard against the side of a cup, the robot's soft finger will automatically bend and adjust its target to press against that same spot, even if the robot's finger is shaped differently. It uses a mathematical concept called "geodesics" (think of it as the shortest path along a curved surface, like an ant walking on a grape) to understand how the touch travels across the soft, squishy surface.
Analogy: Imagine you are walking on a trampoline with a friend. If your friend steps on a specific spot, the trampoline dips. You don't need to know exactly where your friend's foot is; you just feel the dip and step into that same dip to stay balanced. SoftAct feels the "dip" in the contact forces and steps into it.

4. The "Smart Muscle" (Low-Level Control)

Finally, the robot has to actually move its pneumatic (air-powered) fingers. Since soft fingers are squishy and hard to control (like trying to steer a water balloon), the system uses a special AI "muscle controller."

The Solution: This controller learns exactly how much air pressure is needed to bend the finger to a specific spot. It constantly recalculates to ensure the finger hits the target precisely, despite the squishiness.
Analogy: It's like a driver who has learned exactly how much to press the gas pedal in a slippery car to turn a corner without spinning out. They don't just guess; they calculate the exact pressure needed for the conditions.

The Results: Why It Matters

The researchers tested this on tasks like pouring water, screwing in lightbulbs, and reorienting boxes.

Old Way (Copying Movements): The robot would often drop the object, slip, or fail because it tried to mimic human joint angles that didn't make sense for a soft hand.
SoftAct Way: The robot succeeded much more often (up to 70% better in some cases). It figured out its own unique way to hold the object that worked for its soft body, as long as the force and touch felt right.

The Big Picture

The main takeaway is this: Don't teach a robot to look like a human; teach it to feel like a human.

By focusing on the interaction (the push and the touch) rather than the anatomy (the joints and bones), SoftAct allows a weird, non-human robot hand to learn complex human skills. It's the difference between telling a dog to "walk on two legs" (which looks silly and fails) versus teaching a dog to "fetch the ball" (which uses its natural abilities to achieve the same goal). SoftAct teaches the robot to "fetch the ball" by understanding the physics of the touch.

1. Problem Statement

The core challenge addressed is the difficulty of transferring dexterous manipulation skills from human demonstrations to non-anthropomorphic soft robotic hands.

Embodiment Mismatch: Soft robot hands differ fundamentally from human hands in morphology, degrees of freedom (DoF), actuation mechanisms (pneumatic vs. skeletal), and workspace geometry.
Failure of Kinematic Imitation: Traditional retargeting methods rely on mapping human joint trajectories or fingertip positions to robot configurations. These methods fail for soft robots because:
- Direct kinematic correspondence is ill-defined.
- Soft robots exhibit nonlinear, deformation-driven motion rather than rigid joint kinematics.
- Geometric similarity alone does not guarantee functional success (e.g., stable grasping or force application).
Data Scarcity: Collecting high-quality, force-labeled contact data from real-world soft robot interactions is difficult due to sensing limitations and safety concerns.

2. Methodology: SoftAct Framework

SoftAct is a two-stage, force-aware retargeting framework that prioritizes contact geometry and force distribution over kinematic pose matching. It leverages immersive Virtual Reality (VR) for data collection and a learned low-level controller for execution.

A. Data Collection (Virtual Reality)

Environment: An immersive VR environment (Unreal Engine) mirroring the real-world workspace.
Sensors: Participants wear motion capture suits (OptiTrack) to track hand kinematics.
Data Captured: The system records not just hand poses, but also dense contact patches and contact force magnitudes/directions between the hand and objects. This provides a noise-free, high-fidelity dataset of "how" forces are applied, which is difficult to capture in the real world.

B. Two-Stage Force-Aware Retargeting

The framework converts human demonstrations into robot-executable trajectories through two distinct stages:

Stage 1: Force-Balanced Finger Assignment (Offline)
- Goal: Map human fingers to robot fingers without assuming a 1-to-1 correspondence.
- Mechanism:
  - Contact forces are diffused over the human hand surface using a geodesic heat kernel (measuring distance along the surface mesh rather than Euclidean space) to respect surface curvature.
  - The total contact load attributed to each human finger is calculated.
  - Robot fingers are allocated proportionally to balance the contact load (solving a minimization problem to minimize the max load per assigned robot finger).
- Outcome: A many-to-one or many-to-many mapping that ensures the robot's contact distribution matches the functional intent of the human demo.
Stage 2: Contact-Informed Refinement (Online)
- Goal: Refine the target trajectory for each robot fingertip in real-time.
- Mechanism:
  - The robot tracks a baseline end-effector pose.
  - For each robot fingertip, the system calculates a geodesic-weighted influence based on nearby demonstrated contact points and their force magnitudes.
  - A weighted adjustment ( $\delta$ ) is applied to the fingertip target to align the robot's contact surface with the human's demonstrated contact geometry.
- Outcome: Trajectories that dynamically adapt to maintain stable contact, even if the robot's morphology forces a different physical configuration than the human.

C. Policy Learning and Control

High-Level Policy: A Diffusion Policy is trained to imitate the retargeted trajectories. The input includes robot proprioception and object poses; the output is a composite action (end-effector displacement/orientation and soft-finger torque commands).
Low-Level Controller: A learned inverse kinematic model (MLP-based) maps desired fingertip displacements to pneumatic pressure commands.
- Since soft fingers are underactuated and nonlinear, the controller solves an optimization problem at runtime to find the pressure vector that minimizes the error between the predicted and desired fingertip position.
- This allows precise tracking of the complex, deforming trajectories generated by the retargeting stage.

3. Key Contributions

Contact-Centric Formulation: A novel retargeting approach that treats contact geometry and force distribution as the primary transferable representations, bypassing the need for kinematic similarity.
Force-Aware Algorithm: A two-stage pipeline combining offline force-balanced finger assignment with online geodesic-weighted contact refinement.
State-of-the-Art Low-Level Control: A learned controller specifically designed for highly nonlinear pneumatic soft fingers, significantly outperforming linear and direct prediction baselines.
End-to-End System: A complete pipeline from VR data collection to zero-shot real-world deployment on a custom non-anthropomorphic soft robot hand.

4. Experimental Results

The system was evaluated on six contact-rich tasks (e.g., light bulb insertion, cup pouring, bottle unscrewing) using a custom pneumatic soft hand.

Low-Level Tracking:
- SoftAct's learned controller reduced fingertip trajectory tracking RMSE by 55% and variance by 69% compared to kinematic and learning-based baselines (e.g., Direct MLP, Linear Jacobian).
Object Trajectory Tracking (Simulation):
- SoftAct reduced translational error by 55–75% and rotational error by up to 80% compared to kinematic-only baselines.
- Stage 1 (force assignment) alone improved performance, but the full Stage 2 (contact refinement) was necessary for optimal results.
Task Success Rates:
- Simulation: SoftAct achieved 30–70% higher success rates than kinematic baselines across tasks. Notably, it achieved 100% success on light bulb screwing vs. 77% for the baseline.
- Real-World (Zero-Shot): SoftAct demonstrated superior robustness. For example, in "Paper Cup Pouring," it achieved 85% success vs. 35% for the kinematic baseline. In "Box Reorienting," it achieved 70% vs. 10%.

5. Significance and Conclusion

Bridging the Sim-to-Real Gap: By explicitly modeling contact forces and geometry, SoftAct enables soft robots to reproduce the functional intent of human manipulation rather than just mimicking the shape of the motion.
Handling Extreme Mismatch: The framework proves that successful imitation learning for soft robots does not require anthropomorphic structures; instead, it requires reasoning about how forces are distributed to achieve a task.
Future Impact: This work suggests a new paradigm for learning from demonstration in compliant robotics, moving away from rigid kinematic constraints toward force-centric optimization.

Limitations: The current approach relies on ground-truth contact forces available only in simulation (or VR with specific sensors) and requires motion capture markers for object pose in the real world. Future work aims to extend this to visually estimated forces and marker-free operation.