Technical Summary: DynaMimicGen (D-MG)

Problem Statement

Learning robust robotic manipulation policies via Imitation Learning (IL) typically requires large, diverse datasets. Collecting such data is time-consuming, labor-intensive, and often impractical for dynamic environments. Traditional IL methods struggle to generalize across different scene configurations, object instances, and robotic platforms without extensive human demonstrations. Furthermore, existing data generation frameworks (e.g., MimicGen, DexMimicGen) largely rely on static assumptions, using rigid SE(3) transformations and linear interpolation. These approaches fail to generalize to dynamic task settings where object poses change during execution or where the environment is non-stationary. The core challenge addressed is how to synthesize large-scale robotic datasets that enable deep learning policies to generalize effectively in dynamic, evolving environments using minimal human supervision.

Methodology: DynaMimicGen (D-MG)

DynaMimicGen (D-MG) is a scalable framework that transforms a minimal set of human demonstrations (potentially as few as one) into diverse, realistic robot trajectories expressed in absolute Cartesian end-effector space. The framework operates through the following pipeline:

1. Task Segmentation: The system assumes tasks consist of a known sequence of object-centric subtasks (e.g., grasping an object relative to its frame, placing it relative to a target). Source demonstrations are parsed into contiguous segments corresponding to these subtasks using automated metrics or manual annotation.
2. Dynamic Movement Primitives (DMPs): For each subtask segment, D-MG trains a dedicated Dynamic Movement Primitive. DMPs encode demonstrations as nonlinear second-order dynamical systems with a learnable forcing term. This formulation ensures convergence to a goal while allowing flexible trajectory shapes.
3. Adaptation and Generalization:
   • Static Adaptation: To generate data for a new scene, D-MG calculates the relative transformation between the demonstrated object pose and the new object pose. It adjusts the DMP's start and goal configurations accordingly, ensuring the trajectory remains consistent with the new scene geometry.
   • Dynamic Adaptation: Unlike prior methods, D-MG continuously monitors object poses during trajectory synthesis and execution. If an object's position or orientation changes (e.g., due to perturbation), the DMP formulation allows for online modulation of the trajectory in real-time to accommodate these changes.
4. Data Synthesis: The framework generates a potentially unbounded set of trajectories by applying structured environmental perturbations (e.g., randomized displacements of target objects) during execution. Successful trajectories are retained to form the synthetic dataset, while failed rollouts are discarded.

The action space is defined as a combination of absolute end-effector pose commands and binary gripper open/close commands. The generated datasets are used to train policies using Diffusion Policy and Behavior Cloning (RNN-based).

Key Contributions

The paper identifies three primary contributions:

• Scalable Data Generation from Minimal Input: D-MG introduces a framework capable of generating large-scale robot manipulation datasets from extremely sparse human input (often a single demonstration). By leveraging DMPs, it generalizes demonstrated behaviors to novel scenes, drastically reducing the need for extensive human data collection.
• Data Generation in Dynamic Task Settings: In contrast to prior methods that assume quasi-static environments, D-MG enables data generation in dynamic and evolving environments. It leverages the adaptability of DMPs to adjust motion plans in real-time to accommodate changes in object position or orientation during task execution. Notably, this robustness is achieved even when human demonstrations are collected in static scenes.
• High-Quality Data for Imitation Learning: The authors demonstrate that D-MG generates high-quality data that trains proficient agents across diverse scene configurations unseen in the original demos. The method produces smooth, realistic, and task-consistent Cartesian trajectories that adapt to changes in object poses, robot states, or scene geometry.

Experimental Results

The authors evaluated D-MG across three task categories: Basic manipulation (Stack), Contact-rich interaction (Square), and Long-horizon sequential manipulation (MugCleanup).

• Data Generation Rates (DGR): In quasi-static settings, D-MG achieved comparable or superior data generation rates to MimicGen (MG) while using only 1–2 demonstrations compared to MG's requirement of 10. For instance, in the Stack task, D-MG achieved ~95% DGR with one demonstration, whereas MG required ten. In dynamic settings, where MG cannot be applied due to its lack of online correction mechanisms, D-MG maintained high success rates (e.g., 68.7% for Stack under perturbation).
• Policy Performance: Agents trained on D-MG-generated datasets consistently outperformed those trained on MG-generated data or the original source demonstrations.
  • In the Stack task, success rates for Diffusion Policy agents trained on D-MG data reached 83.33% (D0 distribution), compared to 74.00% for MG and 1.33% for the source demonstration alone.
  • In the Square task (contact-rich), D-MG improved Diffusion Policy success rates from 75.33% (MG) to 86.67% (D-MG) in the D0 distribution.
• Ablation Studies: Removing the dynamic perturbation mechanism significantly reduced policy performance, confirming that the structured variability introduced by D-MG's online adaptation is critical for robust learning. The benefits were most pronounced in image-based (high-dimensional) settings, though positive effects were also observed in low-dimensional state observations.
• Real-World Validation: The framework was validated on a physical Franka Emika Panda robot, successfully executing tasks like lifting, stacking, and cleanup, demonstrating its practical applicability.

Significance and Claims

The paper claims that D-MG represents a step toward more scalable and efficient data-driven robot learning. By eliminating the need for extensive human demonstrations and enabling generalization in dynamic settings, D-MG offers a powerful alternative to manual data collection. The authors assert that their method bridges the gap between human demonstrations and effective downstream policy learning, allowing robots to autonomously generalize task execution from minimal input. The work highlights that scalability in imitation learning can emerge from synthesizing massive, heterogeneous data from sparse expert inputs, provided the synthesis method can handle dynamic environmental variations.