Technical Summary: Correspondence-Free, Function-Based Sim-to-Real Learning for Deformable Surface Control

1. Problem Statement

Soft robotic systems with free-form surfaces possess high degrees of freedom (DoFs) due to their elastic bodies, making shape control via inverse kinematics (IK) highly complex. While numerical simulations can predict deformation, significant geometric discrepancies ("reality gaps") exist between simulated shapes and physical deformations due to hardware uncertainties and modeling simplifications.

Existing sim-to-real transfer methods face three primary limitations:

1. Limited Spatial Coverage: They often rely on sparse marker points (e.g., only correcting the end-effector), leaving the rest of the manipulator uncorrected.
2. Dependence on Correspondences: Traditional methods require full point-to-point correspondences between simulated and captured data, rendering them incompatible with unstructured 3D scanner point clouds.
3. Lack of Robustness to Occlusion: Marker-based systems fail when markers are occluded by large local deformations, making affected frames unusable for training.

The paper addresses the need for a correspondence-free sim-to-real learning method that can correct shape errors under large deformations using either 3D scanner data (point clouds without correspondences) or motion capture (MoCap) data (tolerating missed markers).

2. Methodology

The proposed approach establishes a function-based mapping between simulated and real deformation responses, integrated into a neural network (NN) pipeline for efficient IK computation.

A. Compact Shape Representation

To avoid the redundancy of processing full point clouds, the free-form surface $S$ is represented by a B-spline surface.

• The surface is parameterized, and control points ($S_c$) are extracted to serve as a compact shape descriptor.
• Any point $p$ on the surface is reconstructed via a B-spline decoder: $p = B(u, v, S_c)$.
• This reduces the dimensionality of the input for the learning network.

B. Function-Based Spatial Warping

The core of the sim-to-real transfer is a Radial Basis Function (RBF) based spatial warping function $\Phi(p)$ that maps a simulated point $p$ to a corrected point $p^*$:
$$p^* = \Phi(p) = \alpha_0 + Ap + \sum_{i=1}^{N} \beta_i e^{-c\|p - q_i\|^2}$$

• Kernel Centers ($q_i$): Fixed in the UV parametric domain but mapped to 3D space via the B-spline control points.
• Function Prediction Network ($N_{rbf}$): Instead of learning the warping directly for each shape, a neural network predicts the RBF coefficients $\gamma = [\alpha_0, \dots, \beta_i]$ based on the input control points $S_c$. This allows the system to generate a continuous warping function for any simulated shape.

C. Differentiable Alignment and Joint Training

To train the system without explicit point correspondences, the authors introduce a confidence-weighted Chamfer distance as the primary loss metric.

• Confidence Map ($N_{conf}$): A secondary network predicts a scalar confidence value $w_c(p) \in [0, 1]$ for each point, representing the likelihood of finding a reliable correspondence in the captured data.
• Loss Function: The training minimizes a weighted Chamfer distance between the predicted corrected shape $S^*$ and the captured physical shape $S_p$. The confidence map down-weights regions with missing data or low reliability.
• Regularization: Three additional loss terms ensure robustness:
  1. Confidence Regularization: Prevents the confidence map from collapsing to zero.
  2. Normal Compatibility: Ensures surface normals of the predicted shape align with the captured shape.
  3. Geometric Regularization: Enforces compatibility conditions on the RBF coefficients (e.g., $\sum \beta_i = 0$) to prevent "weird" shapes when kernel counts are limited or discrepancies are large.

D. Inverse Kinematics (IK) Pipeline

The trained networks form an end-to-end, analytically differentiable pipeline:

1. Forward Kinematics: A network $N_{fk}$ predicts B-spline control points $S_c$ from actuation parameters $a$.
2. Sim-to-Real Transfer: $N_{rbf}$ predicts RBF coefficients, and $\Phi$ warps the shape to the physical domain.
3. Optimization: The IK problem is solved via gradient descent, minimizing the distance between the predicted physical shape and a target shape $S_t$. The differentiability of the entire chain allows for efficient gradient computation.

3. Key Contributions

1. Resilient Sim-to-Real Pipeline: A framework bridging the reality gap using an RBF-based spatial warping function space, capable of handling unorganized point clouds and partial occlusions.
2. Correspondence-Free Training: A differentiable alignment method using a learned confidence map and weighted Chamfer distance, eliminating the need for pre-defined point correspondences.
3. Efficient NN-Based IK: A computational pipeline that solves the inverse kinematics problem for soft deformable robots efficiently by leveraging analytical gradients through the sim-to-real transfer.

4. Experimental Results

The method was validated on four pneumatically actuated soft robots with varying complexities:

• Deformable Membrane: 9 chambers, large inflation deformations.
• Soft Manipulator: Two segments, 6 chambers, pure bending.
• Robotic Mannequin: 15 chambers, 9 DoFs, complex free-form surface.
• Compressible Manipulator: 3x3 bellows, capable of compression, inflation, and bending.

Performance Metrics:

• Accuracy: Compared to simulation-only or direct learning (mapping actuation to shape), the proposed sim-to-real method reduced average shape errors by 63.3% to 75.5% on the deformable membrane and significantly improved results on the mannequin and manipulators.
• Data Efficiency: The method achieved low error levels with only 40 training frames, whereas direct learning required 400 frames to reach similar performance.
• Robustness: The system successfully handled datasets with up to 21% missing sample points (3D scanner) and 9.8% missing markers (MoCap).
• Applications:
  • 3D Printing: Used as a reconfigurable mold; sim-to-real correction reduced gaps between the printer head and the mold, enabling accurate material deposition.
  • Motion Planning: Enabled collision-free trajectory planning for a soft manipulator navigating obstacles, where simulation-only predictions led to collisions.
  • Mannequin Control: Achieved fast convergence (approx. 15 iterations) to target human body shapes from the CAESAR dataset.

5. Significance and Claims

The paper claims to advance the state of the art in shape control for deformable free-form surfaces by:

• Removing the dependency on full correspondences, thereby enabling the use of dense 3D scanners and robust MoCap systems even with occlusions.
• Providing a unified, differentiable framework that integrates sim-to-real correction directly into the IK solver, allowing for real-time, gradient-based control.
• Demonstrating generalizability across diverse actuation principles (pure inflation, bending, compression) and structural complexities.

Limitations and Future Work:
The authors acknowledge that the method relies heavily on the quality and coverage of the training dataset. Additionally, due to the inherent smoothness of RBFs, the method is less effective on surfaces with sharp features. Future work includes using the deformable mannequin as a reusable mold for sustainable garment fabrication and testing soft manipulators in pick-and-place tasks.