Technical Summary: Agentic-DPO

1. Problem Statement

Large Language Model (LLM) agents are frequently trained via Supervised Fine-Tuning (SFT) on expert trajectories. While SFT is simple and low-cost, it treats multi-turn agent behavior as ordinary text imitation. This approach inherits the limitations of behavior cloning: the model learns to imitate the sequence of expert tokens but fails to learn how to choose the correct action when facing plausible mistakes at a specific state.

Existing methods to address this, such as preference learning or Reinforcement Learning (RL), often require high-cost environment rollouts, reward models, or full-trajectory student exploration. Furthermore, naively applying preference learning to agent actions is problematic because action strings mix the latent policy decision with schema-specific rendering (e.g., JSON vs. ReAct format). If not controlled, the model may optimize for the rendering style rather than the underlying decision.

The core challenge is to convert expert trajectories into state-conditioned action preferences that teach the agent to avoid its own likely mistakes, without the computational overhead of online RL or the need for external reward models.

2. Methodology: Agentic-DPO

The authors propose Agentic-DPO, a lightweight, offline agent policy optimization method. It transforms expert trajectories into preference supervision by contrasting the expert action with a "plausible wrong action" sampled from the current student policy at the same state.

Core Mechanism

1. State-Conditioned Preference Construction:

   • For each expert step $(s_t, u^+_t)$ in a trajectory, the method treats the expert action as the "preferred" choice.
   • It samples a one-step action from the current student policy ($\pi_{\theta_{student}}$) at the same state $s_t$.
   • The student's sampled action is parsed into a latent decision. If it is equivalent to the expert action (same tool/arguments), it is discarded. If it is a different valid decision, it serves as the "rejected" (negative) sample ($u^-_t$).
   • This creates a preference triple $(s_t, u^+_t, u^-_t)$ where the model is trained to prefer the expert action over the specific mistake the student is likely to make.

2. Loss Function:

   • The method uses a Direct Preference Optimization (DPO) style loss, optimized over the generated action tokens only.
   • To handle varying action lengths across different tools or GUI interactions, a length-scaled preference coefficient $\beta_{eff}$ is used to prevent long actions from dominating the gradient simply due to token count.

Stabilization Strategies

To ensure stability and prevent the model from optimizing for surface-level formatting rather than decision logic, Agentic-DPO introduces two key components:

• SFT Anchor:

  • The training begins with an SFT warm-up to ensure the student understands valid action formats.
  • During DPO training, a frozen reference policy ($\pi_{ref}$) from the SFT stage is used.
  • The total loss combines the DPO contrastive term with an SFT term ($L_{train} = L_{ADPO} + \lambda L_{SFT}$). This anchor prevents the preference loss from suppressing the probability of the expert action too aggressively and keeps the policy close to valid syntax.

• Policy-Preserving Augmentation (PPA):

  • PPA renders the same latent trajectory under multiple schema views (e.g., changing tool names, switching between ReAct and JSON formats, or injecting recovery contexts).
  • By sampling different views $\phi$ for the same latent preference pair during training, the method averages out gradients related to schema-specific forms ($H_\theta$), ensuring the model optimizes for the latent decision margin ($G_\theta$).

Training Procedure

The training occurs in "refresh rounds." In each round, the current student policy is used to sample new negative actions for a subset of the expert data. This allows the negative samples to evolve as the student improves, providing a dynamic "hard negative" signal without requiring environment interaction.

3. Key Contributions

1. Agentic-DPO: A lightweight offline policy optimization method that constructs state-conditioned action preferences by contrasting expert actions with one-step student negatives, eliminating the need for online environment rollouts or reward models.
2. Policy-Preserving Augmentation (PPA): A technique that renders latent trajectories under multiple schemas to reduce schema-dominated preference gradients, stabilizing action-level DPO.
3. Empirical Validation: Demonstration that Agentic-DPO improves agent performance across diverse benchmarks (tool use, long-horizon user interaction, and web-GUI grounding) and model scales, outperforming standard SFT and matching online RL methods at a fraction of the cost.

4. Experimental Results

The authors evaluated Agentic-DPO on three benchmarks: StableToolBench (tool use), $\tau$-bench retail (long-horizon user-interleaved interaction), and Mind2Web (web-GUI grounding), using Qwen3.5 models (2B, 4B, 9B) and Gemma3-4B.

• Performance Gains: Agentic-DPO consistently outperformed SFT and other baselines (PPA+SFT, DFT, ETO, GRPO) across all model scales.
  • On $\tau$-bench retail, Agentic-DPO raised the accuracy of a 9B model from 21.7% (SFT) to 41.4%, matching the performance of online GRPO (40.0%) but without online RL rollouts.
  • On StableToolBench, it improved a 2B model from 57.1% (SFT) to 90.9%, surpassing strong baselines.
  • On Mind2Web, it achieved a 64.4% average step success rate on held-out splits.
• Efficiency: Agentic-DPO is significantly cheaper than online RL. On $\tau$-bench, it costs 1.6x the wall time per step compared to SFT, whereas GRPO costs 13.6x due to multi-turn rollouts.
• Ablation Studies:
  • Removing the SFT warm-up caused performance to collapse to 0.0% (parser rejection).
  • Removing the SFT anchor during training led to higher variance and lower accuracy.
  • Removing PPA maintained high canonical accuracy but significantly widened the gap between canonical and perturbed (out-of-distribution) performance, indicating PPA is crucial for robustness.
• Scaling: The method shows diminishing returns on the number of sampled negatives ($K$) beyond 2–4, suggesting the primary benefit comes from finding one plausible local mistake rather than approximating a full rollout distribution. It is also highly data-efficient, performing well with only 25% of the expert data.

5. Significance and Claims

The paper claims that expert trajectories can support low-cost agentic policy optimization when their state-action structure is utilized directly. By converting demonstrations into state-level action preferences, Agentic-DPO bridges the gap between simple imitation (SFT) and expensive online reinforcement learning.

The authors emphasize that Agentic-DPO provides a "student-aware" training signal—teaching the model to avoid its own likely mistakes—without the dominant costs of RL (repeated full-trajectory environment rollouts and reward model training). While it is an offline method limited to states present in the expert data (unlike online RL which can explore new states), the results suggest it is a highly effective and practical approach for improving agent decision-making across various interaction paradigms.