Technical Summary: Driving Intents Amplify Planning-Oriented Reinforcement Learning (DIAL)

1. Problem Statement

The paper identifies mode collapse as the primary bottleneck for applying preference-based Reinforcement Learning (RL) to continuous-action driving policies trained on single demonstrated trajectories per scene.

• The Limitation of Single-Demonstration SFT: When policies are trained via Supervised Fine-Tuning (SFT) on a single logged trajectory, the resulting sampling distribution clusters tightly around that specific maneuver. Even with expressive action heads (e.g., flow-matching or diffusion), the policy fails to represent semantically distinct alternatives (e.g., braking earlier vs. later, changing lanes vs. keeping lane).
• The Failure of Preference RL: Preference-based evaluation (e.g., Best-of-$N$) relies on the policy generating diverse candidates to select the highest-rated one. However, if the sampling distribution is collapsed around a single mode, "oracle selection" cannot recover high-quality trajectories that were never sampled. Consequently, Best-of-$N$ performance plateaus below the quality of the original human demonstration, regardless of the sample budget ($N$).
• The Core Insight: The bottleneck is not the policy update mechanism itself, but the sampling distribution. Standard Group Relative Policy Optimization (GRPO) fails because rollouts drawn from a single intent (or noise seeds) re-collapse the group, providing uninformative group-relative advantages (low preference contrast).

2. Methodology: DIAL Framework

The authors propose DIAL (Driving-Intent-Amplified Learning), a two-stage framework designed to first expand the sampling distribution and then preserve that expansion during preference RL.

Stage 1: Intent-Conditioned Classifier-Free Guidance (CFG)

• Goal: Break the single-demonstration mode collapse by expanding the sampling distribution along semantically distinct maneuver modes.
• Mechanism:
  • The flow-matching action head is conditioned on a discrete intent label $c$.
  • The intent set consists of eight rule-derived categories: cruise, lane-change (L/R), turn (L/R), U-turn, accelerate, decelerate.
  • Classifier-Free Guidance (CFG): During training, intent labels are randomly dropped (replaced with an unconditional placeholder) with probability $p_{drop}$. This teaches the model to generate both conditional (intent-specific) and unconditional action distributions.
  • Inference: A lightweight prefix classifier predicts the intent from the visual context. The generator then decodes the trajectory using the predicted intent, combining conditional and unconditional velocity predictions.
• Outcome: This stage allows the policy to generate diverse trajectories (e.g., a left turn vs. a lane keep) for the same scene, rather than just coordinate-level perturbations of a single path. Intent-conditioned sampling alone raises the Best-of-$N$ ceiling significantly.

Stage 2: Multi-Intent Group Relative Policy Optimization (GRPO)

• Goal: Preserve the expanded diversity during RL fine-tuning and optimize the policy toward human preferences.
• Mechanism:
  • Intent-Balanced Rollouts: For each scene, the system samples $S$ trajectories for each of the $C=8$ intents, creating a group of $K = C \times S$ proposals (e.g., $K=16$ with $S=2$).
  • Group Relative Advantage: The Reward Feedback Score (RFS) is computed for all $K$ trajectories. Advantages are normalized across the entire pooled group. Because the group spans all intent classes, the RL update receives a meaningful preference signal across different maneuver modes, preventing the group from re-collapsing around a single preferred mode.
  • Reward-Hacking Awareness: The training reward modifies the standard RFS to prevent overfitting:
    • Replaces the hard maximum over rater trajectories with a label-softmax aggregation.
    • Densifies scoring anchors from $\{3s, 5s\}$ to $\{1s, 2s, 3s, 4s, 5s\}$ to penalize implausible intermediate motion.
• Policy Update: Uses a clipped GRPO objective with a reference policy penalty (KL divergence) against the Stage 1 SFT checkpoint.

3. Key Contributions

1. Identification of Mode Collapse: The paper establishes that for continuous-action policies, the ceiling of preference RL is limited by the sampling distribution. Competitive SFT baselines saturate below the human demonstration (Best-of-128 RFS < 8.13) because they cannot sample distinct maneuver modes.
2. Intent-Conditioned CFG: Demonstrates that conditioning the action head on discrete intents breaks the sampling ceiling. Intent-conditioned sampling achieves a Best-of-128 RFS of 9.14, surpassing both the prior best planner (RAP, 8.5) and the human demonstration (8.13) for the first time.
3. DIAL Framework: Introduces a two-stage pipeline that combines intent-CFG expansion with multi-intent GRPO. This preserves the diversity established in Stage 1, improving held-out single-trajectory RFS from 7.681 to 8.211.
4. Empirical Validation of Diversity: Shows that single-intent variants (even with optimal intent selection) fail to match DIAL's performance and eventually degrade, proving that spanning the full intent set within the preference group is necessary for stable RL.

4. Experimental Results

Evaluated on the WOD-E2E dataset using the RFS benchmark (Waymo Open Dataset End-to-End):

• Baseline Performance: Four competitive SFT baselines (WAM-Flow, Curious-VLA, AutoVLA, ReCogDrive) showed minimal gains from RL on the held-out split (e.g., ReCogDrive +0.315, WAM-Flow +0.087). Their Best-of-$N$ performance remained below the human demonstration.
• DIAL Performance:
  • Held-out RFS: Improved from 7.696 (SFT init) to 8.211 (+0.515), the highest gain among all RL-trained systems.
  • Full-Split RFS: Improved from 7.369 to 8.631.
  • Best-of-128: Intent-conditioned sampling reached 9.14, exceeding the human demonstration (8.13) and the prior state-of-the-art planner RAP (8.5).
• Ablation Findings:
  • Multi-Intent vs. Single-Intent: Single-intent GRPO variants (using $K=16$ noise seeds for one intent) peaked lower (max 7.992) and degraded by the end of training.
  • Samples per Intent ($S$): $S=2$ (total $K=16$) was identified as the "sweet spot" for peak performance.
  • Reward Shaping: Using softmax aggregation and dense anchors prevented reward hacking and improved stability.

5. Significance and Claims

The paper claims that the fundamental challenge in preference-aligned continuous-action driving is not merely how to update the policy, but how to expand and preserve the sampling distribution being optimized.

• Shift in Paradigm: The work argues that preference RL is ineffective if the policy cannot generate semantically distinct alternatives. The "bottleneck" is the proposal mechanism, not the optimizer.
• Practical Impact: By explicitly modeling driving intents and preserving this diversity through multi-intent GRPO, DIAL successfully escapes mode collapse, allowing the policy to reach and exceed the quality of human demonstrations in a Best-of-$N$ setting, and significantly improving single-trajectory deployment performance.
• Modesty: The authors acknowledge limitations, noting that the eight intents are derived from hand-coded geometric rules (potentially missing long-tail behaviors) and that the RL training data (438 sequences) is small relative to the full Waymo split.

In summary, DIAL demonstrates that intent-diverse proposals are a prerequisite for successful preference optimization in end-to-end driving, enabling policies to explore the full space of admissible maneuvers rather than collapsing into a single geometric imitation.