Technical Summary: DeformGen

Problem Statement

While imitation learning and visuomotor policy learning have achieved significant success in rigid-body robot manipulation, scaling these methods to deformable object manipulation remains a bottleneck due to the high cost and difficulty of collecting diverse demonstration data. Existing data augmentation paradigms, which rely on the equivariant assumption (that Euclidean distances between material points are invariant under rigid transformations), fail fundamentally in deformable settings.

The authors identify two core challenges that break the "rigid-object recipe":

1. State-Space Challenge: Deformable objects possess high degrees of freedom (DoF) and dynamic internal constraints. Unlike rigid bodies where a 6-DoF pose suffices, deformable objects exhibit rich shape and topology variations. Naive geometric perturbations often result in physically implausible states (e.g., disconnected particles or impossible shapes) because valid configurations lie on a constrained manifold within the high-dimensional state space.
2. Trajectory-Transfer Challenge: Material points in deformable objects are not equivariant; they do not move rigidly together. Applying a global rigid transformation to a demonstration trajectory causes the end-effector to misalign with the object's local geometry (e.g., failing to grasp a specific point on a folded cloth) and cannot compensate for local deformations, leading to contact mismatches.

Methodology: DeformGen

DeformGen is a dynamics-based topology augmentation framework designed to synthesize physically valid deformable states and transfer manipulation trajectories in a deformation-aware manner. It consists of two primary components:

1. Dynamics-Based Topological State Augmentation

To address the state-space challenge, DeformGen avoids naive geometric perturbations that leave the valid physical manifold. Instead, it employs a forward-simulation approach:

• Process: Starting from a known valid state ($s_0 \in S_{real}$), the system applies localized external forces ($f$) to the object.
• Simulation: The physics simulator ($\Phi_{sim}$) forward-simulates the dynamics over a time step ($\Delta t$) to generate a new state: $s_{aug} = \Phi_{sim}(s_0, f, \Delta t)$.
• Rationale: Because the state evolves through the simulator's own dynamics, the resulting state remains within the physically plausible subspace ($S_{real}$) without requiring post-hoc repair. This method generates diverse topological variations (bending, twisting, folding) that rigid transformations cannot capture.

2. Deformation-Field Warping for Trajectory Augmentation

To address the trajectory-transfer challenge, DeformGen synthesizes valid manipulation trajectories for the augmented states using a continuous deformation field rather than rigid transforms.

• Displacement Calculation: The system computes per-particle displacements ($\delta_i$) between the source object state and the target (augmented) state.
• Field Construction: These displacements are lifted into a continuous spatial function (deformation field $D(x)$) over the workspace using K-nearest-neighbor (KNN) inverse-distance interpolation.
• Trajectory Warping:
  • Position: For each waypoint $x_t$ in the source trajectory, the position offset is calculated by aggregating the displacements of the $K$ nearest particles. A time-dependent decay factor ($\alpha_t$) allows the trajectory to initially follow local deformations and gradually revert to the original path.
  • Orientation: A local Jacobian matrix ($J_t$) is estimated via least squares fitting to map original local vectors to deformed ones. The induced rotation is projected onto the $SO(3)$ manifold, and the final orientation is computed via Spherical Linear Interpolation (SLERP) between the original and induced rotations.
• Adaptation: The method uses a small $K$ for grasp poses (focusing on local geometry) and a large $K$ (total points) for manipulation phases (capturing global deformation).

Key Contributions

1. Problem Formulation: The work formally identifies that effective augmentation for deformable manipulation requires physically valid state synthesis coupled with deformation-aware trajectory transfer, moving beyond the limitations of rigid-style augmentation.
2. Dynamics-Consistent Pipeline: The authors propose a novel pipeline that generates topology-coherent deformable assets through localized perturbation, physics rollout, and stabilization. It simultaneously synthesizes manipulation trajectories via deformation-field warping, ensuring the end-effector remains aligned with the deformed geometry.
3. Empirical Validation: Extensive experiments on high-fidelity benchmarks demonstrate that DeformGen significantly improves policy learning compared to training on original demonstrations alone or using rigid-style augmentation baselines.

Experimental Results

The framework was evaluated on three deformable manipulation tasks: rope routing, toy packing, and cloth folding, using four policy architectures (ACT, Diffusion Policy, SmolVLA, and $\pi_0$).

• Performance Gains: DeformGen achieved the highest average success rates across most policy architectures. For instance, in the rope routing task, DeformGen achieved success rates up to 99.0% with $\pi_0$, compared to 0% for single-source training and 56.0% for rigid-style augmentation (SoftMimicGen*).
• Ablation Insights:
  • State Diversity: Comparing DeformGen (topological state aug) with SoftMimicGen* (rigid state aug) showed that broader coverage of deformable configurations is critical for generalization.
  • Trajectory Transfer: Comparing DeformGen with a variant using local rigid transfer (DeformGen*) demonstrated that deformation-field warping provides complementary gains by better aligning trajectories with local geometry.
• Generalization: Policies trained with DeformGen successfully generalized to unseen test states, including configurations where the trajectory synthesis itself had failed to produce a valid demonstration, suggesting the learning of transferable manipulation strategies rather than mere memorization.
• Rigid Scenarios: In rigid-only test scenarios, DeformGen remained competitive, indicating that training on topologically diverse data does not substantially compromise performance on simpler tasks.

Significance and Claims

The paper claims that DeformGen provides a cost-efficient solution for data acquisition in deformable manipulation by augmenting a single demonstration into a diverse set of physically valid states and corresponding trajectories.

The authors assert that their results suggest a fundamental shift in how augmentation should be approached for deformable objects: effective augmentation must be dynamics-consistent (generating states that respect physical constraints) and deformation-aware (transferring trajectories that adapt to local geometry). This approach overcomes the limitations of rigid-style augmentation, which fails to capture the high-dimensional state space and non-equivariant nature of deformable objects. The work demonstrates that synthesizing such data leads to policies with higher success rates and better generalization to unseen deformable configurations.