Technical Summary: Drifting Field Policy (DFP)

Problem Statement

The paper addresses the challenges of offline-to-online reinforcement learning (RL) for continuous control. While generative policies (e.g., diffusion and flow models) have successfully captured multimodal action distributions in offline behavior cloning, they face structural limitations when fine-tuned with online reward signals.

Specifically, ODE-based generative policies (including one-step variants) parameterize the policy as a time-indexed velocity field or a stochastic process. When a reward signal is defined at the action level, it must propagate back through the entire ODE trajectory. This creates a non-trivial output-to-trajectory credit assignment problem, where the reward signal is spread across the integration path rather than acting directly on the action output. Even "one-step" variants of these models retain this ODE dependency during training, requiring global re-fitting of the velocity field to adapt to shifting target distributions.

Methodology

The authors propose Drifting Field Policy (DFP), a non-ODE, one-step generative policy built on the drifting model paradigm.

Core Architecture

DFP defines the policy $\pi_\theta(\cdot|s)$ as a single-pass pushforward map $f_\theta(\cdot, s)$ from a prior distribution $p_\epsilon$ to the action space. Unlike diffusion or flow models, DFP has no time variable; the network directly outputs the action.

Theoretical Framework: Wasserstein Gradient Flow

The policy update is framed as a reverse-KL Wasserstein-2 gradient flow (WGF) toward a soft target policy $\pi^+$.

1. Target Policy: The ideal soft policy improvement target is $\pi^+(a|s) \propto \pi_{old}(a|s) \exp(Q_\phi(s, a)/\alpha)$.
2. Drifting Field: The update utilizes a drifting field $V_{p,q} = V^+_p - V^-_q$, which acts as the particle velocity of a WGF.
   • $V^+$ attracts samples toward the target distribution.
   • $V^-$ repels samples from the current model distribution.
3. Gradient Decomposition: The authors derive that the ideal drift field decomposes into two components:
   • $\nabla_a Q$ Ascent: A direction toward higher action-value regions, scaled by temperature $\alpha$.
   • Score Matching: A trust region term that matches the score of the anchor policy $\pi_{old}$, preventing the policy from drifting too far from the behavior distribution.

Tractable Surrogate: Top-K Supervision

Since the ideal target $\pi^+$ involves an intractable normalization constant and cannot be sampled directly, the authors propose a tractable surrogate loss:

• Mechanism: Instead of sampling from $\pi^+$, the method draws $N$ candidate actions from the anchor policy $\pi_{old}$ and selects the top-$K$ actions based on the critic's $Q$-values.
• Loss Function: The training objective becomes a behavior cloning task on these $K$ self-generated candidates. The loss is a combination of:
  1. $L_{BC}$: Drift loss against the replay buffer (anchoring to data).
  2. $L_{top-K}$: Drift loss against the top-$K$ critic-selected actions (policy improvement).
• Theoretical Guarantee: The paper proves that this surrogate has a bounded approximation error relative to the ideal policy improvement loss, provided the kernel is Lipschitz and the $Q$-distribution has a strictly positive density at the quantile.

Key Contributions

1. First Application of Drifting Models to RL Fine-tuning: The paper introduces DFP, framing the drifting field update as a Wasserstein-2 gradient flow direction on probability space. This allows for direct descent toward the soft policy improvement target without ODE integration.
2. Tractable Top-K Surrogate: The authors derive a simple, effective surrogate for the intractable soft policy update. They prove this surrogate has bounded approximation error and demonstrate that this supervision mechanism is uniquely effective for the drifting backbone compared to ODE-based backbones.
3. Structural Advantage over ODE Policies: The paper highlights that DFP performs gradient descent directly on the action output space. In contrast, ODE-based policies must globally re-fit a velocity field, spreading the reward signal across the trajectory. This distinction allows DFP to adapt more efficiently to shifting targets.
4. State-of-the-Art Performance: Extensive experiments show DFP outperforms existing ODE-based generative policies (e.g., QC-FQL, MVP) and multi-step baselines.

Experimental Results

The method was evaluated on 12 manipulation tasks across the Robomimic and OGBench benchmarks.

• Performance: DFP achieved a 95.8% average success rate, ranking first on 9 of 12 tasks and second-best on the remaining 3.
• Comparison: It outperformed the strongest baseline, QC-FQL (86.8%), by +9.0 percentage points. It also surpassed the best multi-step policy, QC-BFN (83.8%), by +12.0 percentage points.
• Ablation Studies:
  • Backbone vs. Loss: When applied to a MeanFlow (ODE-based) backbone, the top-K supervision yielded only marginal gains (+2.4 pp). However, on the drifting backbone, the same supervision yielded substantial gains (+7.4 pp), confirming that the synergy between top-K supervision and the non-ODE parameterization is unique to DFP.
  • Robustness: The method was found to be robust to variations in the top-K loss weight ($\lambda$) and the number of selected candidates ($K$).
• Efficiency: DFP maintains the low-latency benefits of one-step inference, requiring only a single forward pass, with negligible training overhead compared to other one-step baselines.

Significance and Claims

The paper claims that DFP represents a significant step forward in offline-to-online RL by resolving the credit assignment burden inherent in ODE-based generative policies. By bypassing trajectory integration entirely, DFP allows reward signals to act directly at the action level, enabling more efficient and stable fine-tuning.

The authors emphasize that their approach is particularly beneficial for long-horizon, multimodal tasks (e.g., cube-triple and cube-quadruple tasks), where multi-step BC alone is insufficient and ODE-based credit assignment struggles. The work establishes a new paradigm where the "drifting" mechanism, originally proposed for image generation, can be rigorously interpreted as a Wasserstein gradient flow for policy optimization, offering a structurally superior alternative to diffusion and flow policies for continuous control.

Limitations noted by the authors:

• Current evaluation is limited to simulated continuous-action manipulation tasks.
• Extensions to high-dimensional observations and sim-to-real deployment are future work.
• Performance depends on the quality of the learned critic, a shared limitation with actor-critic methods.
• A deeper theoretical understanding of the distinctions between non-ODE and ODE parameterizations remains an open question.