Technical Summary: DF-ExpEnse

Problem Statement

Improving the sample efficiency of robotic policy improvement is critical for deploying intelligent agents in the real world. A promising approach involves initializing agents with pretrained generative control policies (specifically diffusion policies) derived from offline behavior cloning, and then adapting them using self-collected online experience via reinforcement learning (RL). However, a major bottleneck in this process is effective exploration during the online experience collection phase.

Existing exploration strategies face three primary challenges in this context:

1. Quantifying Exploration Interest: How to principledly measure an agent's interest in an arbitrary action by balancing action quality with uncertainty reduction.
2. Tractable Action Selection: How to identify reasonable actions worth exploring within a continuous action space, where exhaustive search is infeasible.
3. Collaborative Fleet Exploration: How to leverage parallelized robotic fleets to improve the quality of collected experience through collaboration, rather than merely increasing data quantity through independent, potentially redundant, data gathering.

Methodology: DF-ExpEnse

The authors propose DF-ExpEnse (Diffusion Filtered Exploration via Ensembles), a technique designed to enhance exploratory action selection during online inference. It integrates seamlessly with existing RL finetuning frameworks (e.g., DSRL, ResFiT) that update pretrained diffusion policies. The method operates through three core mechanisms:

1. Diffusion-Filtered Candidate Generation

To address the intractability of searching continuous action spaces, DF-ExpEnse leverages the multimodal modeling capabilities of the pretrained diffusion policy ($\pi_{dp}$). Instead of searching the entire space, the agent generates a tractable set of $M$ candidate actions conditioned on the current state $s$:
$$[a_1, \dots, a_M] \sim \pi_{dp}(a | s)$$
This set serves as a constrained search space that offers broad coverage while ensuring candidates possess a base level of admissible quality.

2. Ensemble-Based Exploration Interest (UCB-style)

To select the best candidate from the set, DF-ExpEnse utilizes a critic ensemble ($Q^{[1\dots K]}$), typically used for policy gradient updates, to estimate both value and uncertainty. For each candidate action $a_m$, an exploration interest score $e_m$ is calculated as a linear combination of:

• Value Estimation: The minimum prediction across the ensemble to reduce overestimation bias.
• Uncertainty Estimation: The standard deviation (disagreement) across the ensemble predictions.

The score is computed as:
$$e_m = \min(Q^{[1\dots K]}(a_m, s)) + \alpha \cdot \text{std}(Q^{[1\dots K]}(a_m, s))$$
The agent selects the action with the maximum $e_m$. This approach is inspired by Upper Confidence Bound (UCB) algorithms, balancing exploitation (high value) and exploration (high uncertainty).

3. Behavior Cloning Sampling Regularization (BC-SR)

To prevent the agent from converging prematurely to a single action mode during finetuning (which would degrade exploration), the authors introduce BC-SR. A subset of the candidate action set is overridden with samples drawn from the initial offline-pretrained diffusion policy ($\pi_{dp}^{init}$). This ensures that the candidate set retains the multimodal priors of the original behavior cloning model, preventing the collapse of action diversity.

4. Fleet Normalization for Collaborative Exploration

In fleet settings (multiple agents deployed in parallel), DF-ExpEnse enables real-time cross-agent communication. Before selecting an action, each agent normalizes its value and uncertainty estimates using statistics (mean and standard deviation) aggregated from the candidate sets of all agents in the fleet.
$$\bar{e}_{n,m} = \bar{v}_{n,m} + \alpha \cdot \bar{d}_{n,m}$$
This normalization contextualizes an agent's local choices against the global fleet state, encouraging the group to collectively explore diverse, high-value trajectories rather than redundantly exploring the same regions.

Key Contributions

1. DF-ExpEnse Algorithm: A novel exploration technique that combines diffusion policy multimodality for candidate filtering with critic ensembles for uncertainty-aware action selection.
2. BC-SR Regularization: A method to maintain multimodal diversity in the action candidate set by injecting samples from the initial pretrained policy, preventing mode collapse during online finetuning.
3. Fleet Normalization: A mechanism for collaborative exploration in robotic fleets that normalizes local exploration interests against global fleet statistics to improve collective data quality.
4. General Integration: Demonstration that DF-ExpEnse can be seamlessly integrated with various RL finetuning strategies (specifically DSRL and ResFiT) without requiring additional model components like dynamics models.

Experimental Results

The authors evaluated DF-ExpEnse across manipulation (RoboMimic, DexMimicGen) and locomotion (OpenAI Gym) tasks, comparing it against:

• Vanilla Finetuning: Standard DSRL and ResFiT without enhanced exploration.
• Max-Q Selection: A greedy baseline selecting the highest value candidate (equivalent to DF-ExpEnse with $\alpha=0$).
• Alternative Exploration: E3B (Elliptical Episodic Bonuses) and Plan2Explore, which rely on learned dynamics models.

Key Findings:

• Sample Efficiency: DF-ExpEnse consistently achieved higher success rates and reward scores than all baselines given the same amount of online experience timesteps.
• Ablation Studies:
  • BC-SR: Removing BC-SR from DF-ExpEnse significantly reduced performance, while adding it to Max-Q improved performance, confirming its role in preventing mode collapse.
  • Fleet Normalization: Removing fleet normalization resulted in lower sample efficiency compared to the full collaborative approach, validating the benefit of group-aware exploration.
  • Component Sizes: Performance improved with larger critic ensembles (up to a saturation point) and larger fleet sizes, while increasing the action candidate sample size showed diminishing returns.
• Robustness: The method outperformed baselines that required additional components (like dynamics models), demonstrating that reusing existing critic ensembles is sufficient for principled exploration.

Significance and Claims

The paper claims that DF-ExpEnse provides a general and versatile solution for improving the sample efficiency of finetuning pretrained generative control policies. Its significance lies in:

• Principled Exploration: It offers a tractable way to balance quality and uncertainty in continuous action spaces without needing complex auxiliary models.
• Scalability: It effectively leverages robotic fleets not just for data volume, but for data quality through collaborative, normalized decision-making.
• Practicality: It integrates directly into existing RL pipelines (like DSRL) with minimal architectural changes, making it a practical tool for real-world robotic deployment where sample efficiency is paramount.

The authors conclude that by improving the quality of online experience collection, DF-ExpEnse enables faster and more effective skill learning and refinement for robots, particularly as parallelization capabilities in simulators and real-world deployments continue to grow.