Technical Summary: Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation

Problem Statement

The primary bottleneck in contact-rich dexterous manipulation is the difficulty of collecting real-world data. While imitation learning from human demonstrations shows promise, it is limited by the high cost and effort required for data collection. Sim-to-real reinforcement learning (RL) offers a scalable alternative, but its success has been largely confined to simple tasks where policies rely on low-dimensional features.

A critical challenge lies in tactile sensing. Accurately simulating tactile responses is notoriously difficult due to complex, high-dimensional physical processes and a lack of standardized sensor designs. Consequently, existing sim-to-real methods often simplify tactile data into coarse, low-dimensional features (e.g., binary contact or discretized signals) to ensure robust transfer. However, this simplification sacrifices the rich contact information necessary for complex, fine-grained manipulation. Conversely, approaches using raw tactile signals often suffer from poor sim-to-real alignment due to sensor-specific mismatches and the difficulty of modeling raw sensor behaviors. The paper identifies a fundamental trade-off: simplistic features enable robust transfer but lack expressiveness, while raw signals contain necessary information but are difficult to align across simulation and reality.

Methodology

1. Center-of-Pressure (CoP) Representation

To resolve the trade-off between robustness and expressiveness, the authors introduce Center-of-Pressure (CoP), a physics-grounded tactile representation. CoP summarizes local tactile information into two compact 3D vectors expressed in the sensor frame:

• Contact Force Vector ($f_{cop}$): The total contact force acting on the robot link.
• Contact Location ($p_{cop}$): The 3D Cartesian position of the contact point.

This representation serves as a "middle ground": it is compact enough to align reliably between simulation and hardware (unlike raw taxel arrays) but expressive enough to preserve the force and location data required for dexterous control.

2. Taxel-to-CoP Mapping via Stress Distribution

The paper proposes a parametric stress-distribution model to map raw tactile sensor readings (from arrays like XELA uSkin) to the CoP representation.

• Modeling: Instead of simple averaging, the model accounts for force spreading through the compliant silicone layer. It decomposes the CoP force into normal and shear components.
• Bidirectional Mapping: The model uses a Gaussian radial weight and inverse distance weighting to approximate how stress propagates from the contact point to individual taxels. This allows for a differentiable, bidirectional mapping between raw taxel forces and the CoP state.
• Solving: The CoP force is estimated by solving a regularized least-squares problem aggregating the individual taxel equations, ensuring computational efficiency for policy learning.

3. Sensor Calibration via Differentiable Dynamics

A significant hurdle in tactile sim-to-real transfer is calibrating the orientation of individual taxels ($R_i$) on curved fingertip surfaces, which is difficult to measure manually. The authors propose a calibration scheme based on differentiable dynamics:

• Principle: The method estimates unknown taxel orientations by matching the external wrench inferred from tactile readings to the joint torques required to maintain static equilibrium.
• Process:
  1. Data Collection: A stiff PD controller maintains fixed finger positions while random contacts are applied. Joint torques ($\tau$), angles ($q$), and raw taxel forces are recorded.
  2. Optimization: Using the differentiable taxel-CoP mapping and forward kinematics, the system computes the expected joint torques required to balance the estimated CoP force.
  3. Loss Minimization: The difference between estimated and recorded torques is minimized via gradient descent to learn the optimal taxel rotation parameters ($R_i$).
• Advantage: This approach eliminates the need for ground-truth force measurements or external high-precision sensors for calibration.

4. Sim-Real Alignment Strategy

To ensure robust transfer, the authors employ several alignment techniques:

• Contact Representation: Due to unreliable shear force estimates in simulation (IsaacLab), the method uses only the surface-normal component of CoP for both simulation and hardware, sacrificing shear information for robustness.
• Actuator Dynamics: A Bayesian optimization-based system identification approach aligns simulated actuator dynamics (friction, stiffness) with hardware.
• Sensor Delay: Measured sensor delays are explicitly introduced into the simulation during training to account for latency.

Experiments and Results

Tasks

The approach was evaluated on two challenging, "blind" (no visual input) contact-rich tasks using a 16-DOF Allegro hand with XELA uSkin sensors:

1. Peg-in-Hole Insertion: Requires inferring secondary contacts (peg-hole interaction) through primary tactile feedback.
2. Ball Balancing: Requires balancing a ball on a plate supported by fingertips, dealing with varying ball masses, friction, and surface textures.

Baselines

The CoP representation was compared against:

• Proprioception only (base).
• Binary contact signals (bin).
• CoP force magnitude only (mag).
• Force vector only (vec).
• Contact position only (pos).
• Raw taxel forces (taxel).
• Human expert performance.

Key Results

• Zero-Shot Sim-to-Real Transfer: Policies conditioned on CoP achieved successful zero-shot transfer to the real world without task-specific fine-tuning.
• Performance Superiority:
  • Peg-in-Hole: CoP achieved the highest overall success rate (78%) across various peg shapes, outperforming all baselines. It demonstrated robustness to out-of-distribution (OOD) initializations where simplified baselines failed.
  • Ball Balancing: CoP (along with force-vector baselines) successfully learned the task, whereas policies without explicit force information (base, bin, pos) failed completely. CoP achieved a time-to-fall of 4.60s on average, significantly outperforming simplified representations.
• Ablation Insights:
  • Raw Taxels: Performed worse than nearly all baselines, likely due to imperfect tactile simulation and high dimensionality.
  • Complementarity: Both force and position components of CoP were found to be complementary; using only one component degraded performance.
• Emergent Capabilities: Analysis of the policy's latent space revealed that CoP-conditioned policies implicitly captured physical properties such as object mass, organizing latent embeddings into distinct clusters corresponding to different masses without explicit supervision.

Significance and Claims

The paper claims that Center-of-Pressure (CoP) provides a promising path toward scalable sim-to-real learning for contact-rich dexterous manipulation. Its significance lies in:

1. Bridging the Representation Gap: CoP successfully balances the trade-off between robustness and expressiveness, enabling the use of dense contact information without the fragility of raw sensor data.
2. Calibration without Ground Truth: The differentiable dynamics-based calibration method allows for the estimation of complex sensor parameters (taxel orientations) without requiring expensive ground-truth force measurements, making the approach more practical.
3. Emergent Physical Reasoning: The results suggest that physically grounded intermediate representations enable policies to implicitly learn and encode task-relevant physical properties (like mass), moving beyond simple reactive control.
4. Robustness: The approach demonstrates that aligning simulation and reality through physics-grounded intermediate states (CoP) is more effective than attempting to align raw sensor data or relying on overly simplified binary features.

The authors acknowledge limitations, noting that the abstraction of raw data may discard sensor-specific details useful for highly complex tasks and that current simulators still struggle with shear force estimation. However, the work establishes that physically grounded tactile representations are a viable and effective strategy for advancing dexterous manipulation in the real world.