Technical Summary: CoIRL-AD

1. Problem Statement

End-to-end (E2E) autonomous driving models predominantly rely on Imitation Learning (IL), trained to mimic expert demonstrations. While effective in common scenarios, these models suffer from poor generalization in long-tail situations where expert data is sparse. Small prediction errors can compound over time, shifting the agent into unseen states (distributional shift).

Reinforcement Learning (RL) offers a potential solution by learning from reward signals rather than just demonstrations. However, applying RL to real-world autonomous driving in an offline setting (without interactive simulators) presents significant challenges:

• Data Bias: Real-world datasets are dominated by near-optimal expert trajectories, providing limited coverage of suboptimal or alternative behaviors, which makes learning value differences difficult.
• Objective Conflict: Jointly optimizing IL (behavior cloning) and RL (reward maximization) within a single policy often leads to gradient conflicts.
• Model Bias: In offline settings, world models trained on expert data may produce biased predictions for out-of-distribution actions, leading to over-optimistic value estimates and unstable training if used for RL exploration.

Existing approaches either rely on external simulators (converting the problem to online RL) or use two-stage strategies (pretrain with IL, fine-tune with RL) that are prone to instability in offline real-world data.

2. Methodology: CoIRL-AD

The authors propose CoIRL-AD (Collaborative-Competitive Imitation-Reinforcement Learning), a framework that integrates IL and RL under a unified offline training regime using a shared Latent World Model.

Core Architecture

The framework employs a dual-policy architecture consisting of two separate actors:

1. IL Actor: Optimized via behavior cloning (imitation loss) to mimic expert trajectories.
2. RL Actor: Optimized via reinforcement learning to maximize cumulative rewards.
   Both actors share a perception module and a latent world model but are updated separately to isolate imitation learning from RL exploration noise.

Key Technical Components

A. Latent World Model & Backward Planning

• Latent World Model: Instead of pixel-level generation, the model operates in latent space to predict future states ($\hat{s}'$) given the current state ($s$) and an action sequence ($\tau_a$). This serves as a "reactive simulator" for the RL actor to imagine future rollouts.
• Backward Planning (Inverse Causality): Unlike standard forward-causal planning, the authors propose a backward planning strategy where the $i$-th action is conditioned on current and future waypoint features ($\pi_i(a_i | s_{w,i}, ..., s_{w,n})$). This "goal-to-action" reasoning provides early actions with richer contextual information, which the authors found beneficial for the RL actor's objective of maximizing cumulative rewards.

B. RL Formulation for Offline Data

• Group Sampling (Inspired by GRPO): To address the lack of behavioral diversity in expert-dominated datasets, the RL actor samples $G$ action trajectories from its stochastic policy.
• Reward Function: Rewards are computed using rule-based signals derived from dataset annotations (maps and bounding boxes), including:
  • Imitation Reward: Proximity to expert actions.
  • Collision Reward: Avoidance of static and dynamic obstacles.
• Advantage Estimation: The framework uses the latent world model to predict future states for the sampled trajectories. A critic network estimates the value of current and imagined future states. The advantage is calculated using a Z-score normalization of the total reward (including discounted future value) within the group.
• Stochastic Policy: The RL actor outputs a Gaussian distribution for actions to enable exploration, with a "step-aware" mechanism ensuring trajectory smoothness by making only one action stochastic per sequence while others follow the mode.

C. Competitive Dual-Policy Mechanism

To resolve the instability of offline RL and the gradient conflicts of joint optimization, the two actors engage in a competitive learning mechanism:

• Periodic Comparison: Every $k$ iterations, the performance (accumulative reward) of the IL and RL actors is compared.
• Adaptive Transfer:
  • If performance is similar, both remain unchanged.
  • If the gap is moderate, a "soft merge" transfers knowledge from the winner to the loser.
  • If the gap is large, the loser's parameters are directly replaced by the winner's.
• Role: This mechanism anchors the RL actor to expert-like driving (preventing catastrophic divergence due to world model bias) while allowing the RL actor to selectively transfer beneficial exploratory behaviors back to the IL actor.

3. Key Contributions

1. Offline RL Framework for E2E Driving: A novel approach to integrate RL into E2E driving without external simulators, utilizing a latent world model trained on real-world data for imagined rollouts.
2. Decoupled Competitive Training: A dual-policy framework that separates IL and RL into distinct actors to avoid gradient conflicts, connected via a competitive mechanism that stabilizes training and facilitates knowledge exchange.
3. Backward Planning Strategy: The introduction of inverse causal masking in the planning head, which improves RL performance by aligning the policy with goal-oriented reasoning.
4. Group Sampling for Offline Data: An adaptation of group sampling (GRPO) to generate behavioral diversity from expert-dominated datasets, enabling effective value estimation.

4. Experimental Results

Experiments were conducted on the nuScenes dataset.

• Main Benchmarks: CoIRL-AD consistently outperforms strong IL-based baselines (e.g., LAW) in both L2 displacement error and collision rates. It achieves the best combined metric ($L2 \cdot Col$) and maintains low inference latency.
• Cross-City Generalization: When trained on nuScenes-Singapore and tested on nuScenes-Boston, CoIRL-AD demonstrated significantly higher robustness (77% reduction in the $L2 \cdot Col$ metric compared to the baseline), attributed to the RL actor's exploration capabilities.
• Long-Tail Scenarios: On subsets of data with high collision rates or high L2 errors (rare/challenging cases), CoIRL-AD showed substantial improvements over the baseline, whereas the baseline struggled.
• Ablation Studies:
  • Decoupling & Competition: Only the decoupled actors with the competitive mechanism improved both accuracy and safety. Two-stage and loss-merging strategies failed or performed worse.
  • World Model: Removing the latent world model significantly degraded performance, particularly in long-horizon collision avoidance.
  • Inverse Causality: The backward planning strategy specifically benefited the RL actor; applying it to pure IL degraded performance, highlighting its specific utility for reward maximization.

5. Significance and Claims

The paper claims that CoIRL-AD demonstrates that reinforcement learning can effectively enhance offline end-to-end autonomous driving systems when properly structured.

• Stability over Simulators: By using a learned latent world model instead of an external simulator, the method avoids sim-to-real gaps and simulator-specific assumptions.
• Solving the Offline Challenge: The competitive dual-policy framework addresses the specific instability of offline RL (caused by world model bias and lack of diverse data) by using the IL actor as a stabilizing anchor.
• Targeted Improvement: The authors emphasize that RL provides the most significant benefits in challenging, under-represented, and long-tail scenarios where expert demonstrations are sparse, while maintaining performance in standard scenarios.
• Generalizability: The approach is presented as a general-purpose framework for combining imitation and reinforcement learning in embodied AI, offering a promising direction for improving generalization without relying on interactive simulation.