Technical Summary: Guided Discovery of New Behaviors using Diffusion Policies

1. Problem Statement

Diffusion policies have emerged as powerful tools for modeling multimodal action-trajectory distributions in robotics. However, when trained on limited demonstration data, standard diffusion sampling tends to reproduce dominant behaviors while neglecting valid but rare modes. This "mode collapse" limits the discovery of novel solutions.

Existing approaches to address this face significant limitations:

• Guidance Methods: While they increase diversity, they often push samples into low-support regions, resulting in trajectories that are infeasible or non-executable in the physical environment.
• Reinforcement Learning (RL) Hybrids: Combining RL with diffusion samplers often struggles to escape local minima. Even with pre-trained policies, RL typically refines existing behaviors for optimality rather than exploring new modes, unless explicit reward shaping is used.
• Synthetic Data: Methods that synthesize demonstrations often require complex novelty search or reward shaping to encourage exploration.

The core challenge is to systematically discover new, executable behaviors that are underrepresented in the initial dataset without generating infeasible samples or getting stuck in local optima.

2. Methodology: Guided Discovery of New Behaviors (GDNB)

The authors propose GDNB, a bootstrapping framework that integrates Feynman–Kac correctors with sampling-based trajectory optimization to systematically mine and repair novel trajectories. The framework operates in a closed-loop iterative process consisting of three key steps:

A. Rare-Event Sampling via Feynman–Kac Correctors (FKC)

Instead of standard guidance which can introduce bias, GDNB uses FKC to steer the diffusion process toward a "tilted" target distribution that favors low-probability (rare) regions while maintaining theoretical correctness.

• Biasing Potential: The method defines a guiding potential $B_k$ based on the denoiser-score energy. Samples with large score norms (indicating the model drifts) are identified as lying in low-probability regions.
• Lennard-Jones Potential: A novel guiding potential inspired by the Lennard-Jones potential is used to attract samples to a specific "rare shell" of energy. This prevents samples from drifting too far into unrecoverable regions (out-of-distribution) while avoiding common high-density modes.
• Particle Resampling: A batch of particles is simulated using SDEs that include the biasing potential. Particle log-weights are updated via Euler–Maruyama steps, and particles are resampled based on these weights to focus on the frontier of the distribution.

B. Sampling-Based Trajectory Optimization (SBTO) for Repair

Samples generated by the rare-event sampler are often physically infeasible. GDNB employs a local repair module based on SBTO to refine these candidates.

• Frontier-Preserving Repair: The SBTO algorithm optimizes a candidate action sequence within a local trust region around the sampled draft.
• Objective Function: The repair objective balances task completion (maximizing reward) with draft tracking. It penalizes high-frequency residual edits and large displacements from the original rare draft, ensuring the repaired trajectory retains the novel behavioral characteristics of the rare sample while becoming executable.
• Success Gating: Only trajectories that successfully complete the task after repair are retained.

C. Policy Fine-Tuning (Bootstrapping)

Successful repaired trajectories are added to the training dataset. The diffusion policy is then fine-tuned on this augmented dataset. This creates a targeted curriculum where the model learns to generate previously underrepresented behaviors, which are then used as the base for the next iteration of rare-event sampling.

3. Key Contributions

1. GDNB Framework: A novel bootstrapping loop that combines Feynman–Kac correctors for principled rare-event sampling with SBTO for local trajectory repair.
2. Novel Guiding Potential: The introduction of a Lennard-Jones-like potential based on denoiser-score energy to systematically guide diffusion samples toward promising but underrepresented regions without pushing them into unrecoverable out-of-distribution zones.
3. Systematic Mode Recovery: A method that effectively mines, repairs, and reincorporates novel trajectories, enabling the discovery of diverse behaviors that standard sampling misses.
4. Comprehensive Evaluation: Validation across eight diverse robotic manipulation benchmarks (e.g., Push-T, Franka Kitchen, ToolHang) and real-world experiments, demonstrating the discovery of structured, executable rare behaviors.

4. Experimental Results

The authors evaluated GDNB on eight manipulation benchmarks and a multimodal agent task.

• Mode Recovery: In the Multimodal Agent benchmark, where initial data covered only one of two reward-optimal modes, GDNB successfully recovered the missing mode, whereas baselines like DPPO and standard sampling remained collapsed on the dominant mode.
• Rare Behavior Discovery:
  • Kitchen: Discovered rare subtask sequences, including successful sequences completing at least four subtasks (occurring in ~1% of the original distribution).
  • Can: Found novel gripper-can contact locations and end-effector orientations.
  • Lift: Discovered a "flip-then-grasp" strategy where the end-effector reorients the object before grasping.
  • ToolHang: Identified a rare mid-air release mode where the tool is released toward the hook while in the air.
• Quantitative Metrics:
  • GDNB achieved the highest success rate (0.883) among all compared methods.
  • It demonstrated the strongest Task Feature density–variance score and the largest Task Rareness novel gain, indicating it successfully expanded the policy's support into rare regions while maintaining task performance.
  • Compared to baselines like DPPO, SIME, SOE, MimicGen, and DSRL, GDNB consistently outperformed in discovering diverse, executable behaviors.
• Real-World Validation: The paper reports successful open-loop replays of discovered trajectories (Can, Lift, ToolHang) on a real robot, confirming that the simulated discoveries are physically executable.

5. Significance and Claims

The paper claims that GDNB provides a systematic approach to discovering diverse and executable behaviors in diffusion policies, addressing the limitations of both standard sampling (mode collapse) and existing guidance/RL methods (infeasibility or local minima).

• Systematic Mining: Unlike random exploration, GDNB explicitly targets the "frontier" of the policy's distribution—regions that are rare but not yet unrecoverable.
• Repairability: By integrating SBTO, the framework ensures that the discovered rare behaviors are not just theoretical curiosities but are physically feasible and task-successful.
• Curriculum Learning: The iterative retraining process allows the policy to progressively learn and stabilize these new modes, effectively expanding the policy's capability beyond the initial demonstration set.

The authors note limitations, including the need for careful hyperparameter tuning of the SBTO and biasing potential, and the uncertainty regarding long-term saturation after many iterations. They suggest future work could involve automatic parameter selection and integration with continual learning methods to handle large numbers of iterations.