Technical Summary: One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies

Problem Statement

Visuomotor policies trained via imitation learning often exhibit brittleness when deployed in real-world settings. Minor deviations in the robot's initial configuration or the presence of unseen obstacles can lead to out-of-distribution (OOD) observations. These OOD states often cascade into execution errors and task failures. While collecting extensive human demonstrations under diverse spatial configurations can mitigate this, the process is tedious, costly, and often impractical. Existing data augmentation techniques in computer vision (e.g., color jittering, cropping) fail to address the unique requirements of robotics, specifically the need to maintain action-view consistency and respect physical constraints (3D collisions and contact dynamics) simultaneously. Furthermore, standard state-based augmentation often defers visual invariance to state estimation, while prior visual-action augmentation methods are limited to static, third-person, pinhole camera views or single-step modifications that cannot generate smooth, trajectory-level obstacle avoidance.

Methodology: The 1001 DEMOS Framework

The authors propose 1001 DEMOS, an offline data augmentation framework that generates thousands of visually realistic and physically feasible action-view trajectories from a single real-world human demonstration. The system utilizes a portable parallel gripper equipped with a single wide-field-of-view (FoV) fisheye camera. The framework operates through three primary modules:

1. 3D Scene Reconstruction from Eye-in-Hand Fisheye Video

The system processes fisheye image sequences captured during scene scanning and task demonstration to create two representations:

• Scene Point Cloud: Reconstructed using COLMAP for motion planning and collision checking.
• Fisheye 3D Gaussian Splatting (Fisheye-3DGS): A novel adaptation of 3D Gaussian Splatting (3DGS) to handle wide FoV fisheye cameras.
  • Fisheye Ray Sampler: The authors replace the standard pinhole ray sampler with a KB8-based fisheye ray sampler. For each fisheye pixel $(u, v)$, the ray direction $r_d$ is computed via the KB8 model, then projected through camera intrinsics $K$ to obtain pinhole coordinates $(u_p, v_p)$.
  • Optimization: To preserve the efficiency of 3DGS (CUDA block-thread structure), the system redistributes tile assignments from pinhole rays to fisheye rays, partitioning fisheye rays into the original 256-ray-per-tile layout. This allows for fast rasterization while accurately modeling fisheye distortion.

2. Action Generation via Trajectory Optimization

Using the reconstructed 3D point cloud and the original demonstration trajectory, the system generates two types of novel trajectories via trajectory optimization:

• Free-Space Trajectories: Generated with different initial camera poses sampled within a quaternion cone around the original viewpoint.
• Obstacle-Avoidance Trajectories: Generated by introducing obstacle point clouds (sampled from Objaverse) into the scene and planning paths that avoid them.

The optimization problem minimizes a composite loss function:
$$\mathcal{L} = \mathcal{L}_{funnel} + \mathcal{L}_{collision} + \mathcal{L}_{render} + \mathcal{L}_{smooth}$$

• $\mathcal{L}_{funnel}$: A delta funnel loss that ensures trajectories converge to the same pre-contact pose as the original demonstration, preserving contact dynamics.
• $\mathcal{L}_{collision}$: Uses a Truncated Signed Distance Function (TSDF) of the scene (including augmented obstacles) to penalize collisions.
• $\mathcal{L}_{render}$: A view-consistency loss that keeps generated poses close to the distribution of the original scanning and demonstration views to prevent "floater" artifacts and blurry reconstructions common in 3DGS when rendering from novel viewpoints.
• $\mathcal{L}_{smooth}$: Penalizes velocity jerkiness to ensure smooth trajectories.

3. View Generation and Policy Training

• View Generation: The optimized 6D trajectories are rendered using the trained Fisheye-3DGS to produce corresponding visual observations. For obstacle scenarios, the original scene's 3DGS is augmented with trained 3DGS of the added obstacles.
• Data Compilation: The augmented data is compiled by segmenting the gripper using SAM2 and overlaying it onto rendered images. The gripper opening width is retained from the original demo, while the 6D end-effector poses are taken from the optimized trajectory.
• Training: A Diffusion Policy is trained on the union of the original expert demonstrations and the generated augmented dataset.

Key Contributions

1. Action-View Augmentation Framework: A method that jointly augments visual observations and action trajectories, ensuring consistency between the two while respecting physical constraints.
2. Fisheye-3DGS: A novel formulation extending 3D Gaussian Splatting to wide FoV fisheye cameras, enabling fast, high-fidelity novel view rendering for eye-in-hand systems.
3. Trajectory-Level Obstacle Avoidance: The ability to generate smooth, collision-free trajectories that navigate around unseen obstacles, a capability lacking in prior single-step or third-person augmentation methods.
4. Contact-Aware Augmentation: A mechanism that detects contact events and restricts augmentation to pre-contact phases, thereby preserving the critical contact dynamics of the original demonstration.

Experimental Results

The framework was evaluated in both simulation (RoboMimic "square" task) and real-world scenarios (cup serving task).

• Simulation (RoboMimic):

  • Performance Gain: Action-view augmentation improved task success rates by an average of 56% compared to policies trained without augmentation.
  • Comparison to Baselines: The method outperformed "Aug Action Only" (which perturbs actions but keeps visuals fixed) by up to 35% and the single-step NeRF-based method SPARTN by 15% in average success rate.
  • Upper Bound: The performance closely tracked an "oracle" policy trained with ground-truth novel-view rendering, with a gap of only 8-11%.

• Real-World Evaluation:

  • Robustness to OOD Configurations: Free-space augmentation improved success rates by 55% over non-augmented policies when facing unseen initial robot configurations.
  • Obstacle Avoidance: Policies trained with obstacle-aware augmentation achieved a 100% success rate in tasks with unseen obstacles, significantly outperforming free-space augmentation (10%) and no augmentation (5%). This demonstrated the policy's ability to learn collision avoidance behaviors not present in the original human demos.

Significance and Claims

The paper claims that 1001 DEMOS effectively endows robots with skills not explicitly demonstrated, specifically obstacle avoidance and robustness to novel initial configurations. By generating physically feasible, trajectory-level, eye-in-hand demonstrations from a single human demo, the framework addresses the data scarcity and brittleness issues inherent in visuomotor policy learning. The authors emphasize that their approach bridges the gap between visual invariance and physical consistency, allowing policies to generalize to OOD viewpoints and environments without exhaustive manual data collection.

The authors modestly acknowledge limitations, noting that the current pipeline is restricted to static scenes due to the single moving camera's view coverage and that the generated trajectories do not inherently account for the kinematic limits of the downstream deployment robot, though they are bounded to ensure task-space feasibility. Future work is suggested to explore multi-camera setups and embodiment-aware trajectory optimization.