CarPLAN: Context-Adaptive and Robust Planning with Dynamic Scene Awareness for Autonomous Driving

This paper proposes CarPLAN, a novel imitation learning framework for autonomous driving that enhances robustness and context adaptability through Displacement-Aware Predictive Encoding for improved spatial awareness and a Context-Adaptive Multi-Expert Decoder for dynamic scenario handling, achieving state-of-the-art performance on the nuPlan and Waymax benchmarks.

The Gist

Imagine you are teaching a robot to drive a car. The old way of doing this was like giving the robot a giant, rigid rulebook: "If you see a red light, stop. If you see a stop sign, stop. If a car is 5 meters away, slow down." The problem is, real life is messy. Sometimes you need to speed up to merge, sometimes you need to creep forward in a tight spot, and sometimes you need to be extra cautious because a kid is chasing a ball nearby. Rules can't cover every single scenario.

So, researchers tried Imitation Learning. Instead of rules, they showed the robot thousands of hours of videos of human drivers and said, "Just copy what they do."

But here's the catch: Copying isn't the same as understanding.

If you just tell a student to "copy the teacher's handwriting," they might copy the shape of the letters but miss the intent. In driving, a robot might copy a human's path perfectly but fail to realize why the human stopped, leading to a crash.

Enter CarPLAN. Think of CarPLAN as a super-smart driving student who doesn't just copy the teacher's hand movements; they actually understand the traffic flow and can adapt their driving style on the fly.

Here is how it works, broken down into two main superpowers:

1. The "Future Distance" Crystal Ball (Displacement-Aware Predictive Encoding)

Most driving AI looks at the world and says, "There is a car 10 meters ahead."
CarPLAN asks a deeper question: "Where will that car be relative to me in 3 seconds, and where will I be relative to it?"

• The Analogy: Imagine playing a game of tag. A normal player just sees where the runner is right now. CarPLAN is like a player who instinctively predicts, "If I run left, the runner will be 2 steps to my right in a second."
• How it helps: During training, CarPLAN practices predicting these future "gap distances" between the car and everything around it (other cars, pedestrians, road lines). Even though it stops doing this math once it's actually driving on the road, the practice teaches the AI a deep sense of spatial awareness. It learns to respect the "personal space" of every object, ensuring it doesn't just follow a path, but maintains a safe, comfortable bubble around itself.

2. The "Swiss Army Knife" Brain (Context-Adaptive Multi-Expert Decoder)

This is the coolest part. Imagine a single driver trying to handle every situation: parking in a tight garage, merging onto a highway at 70 mph, and navigating a chaotic school zone. One "brain" trying to do all three often gets confused or picks the wrong strategy.

CarPLAN uses a Mixture of Experts (MoE).

• The Analogy: Instead of one generalist driver, CarPLAN has a team of specialist drivers inside its brain.
  • Expert A is a pro at highway merging.
  • Expert B is a pro at tight city parking.
  • Expert C is a pro at avoiding pedestrians.
  • The "Router": There is a smart manager (the Router) who looks at the current scene. If the car is on a highway, the Router says, "Wake up Expert A!" If the car is in a school zone, it says, "Wake up Expert C!"
• How it helps: The AI doesn't use a "one-size-fits-all" strategy. It dynamically switches to the specific "expert" best suited for the current chaos. This makes the car much more flexible and robust when things get weird or dangerous.

The Results: Why Does This Matter?

The researchers tested CarPLAN on some of the toughest driving simulations in the world (like nuPlan and Waymax).

• The Competition: Other AI planners often crash in complex scenarios or get stuck because they are too rigid.
• CarPLAN's Performance: It beat almost every other system. It handled "Hard" scenarios (like aggressive drivers and bad weather) better than anyone else.
• Real-world feel: In video tests, when a pedestrian stepped out, CarPLAN didn't just brake hard; it smoothly adjusted speed and position, just like a cautious human would. When merging, it found the perfect gap without being aggressive.

The Bottom Line

CarPLAN is a major step forward because it stops treating driving as a simple math problem and starts treating it as a dynamic conversation with the environment.

1. It learns to feel the distance to everything around it (not just where things are, but where they are going).
2. It has a team of specialists that switch roles depending on the situation, rather than relying on one tired brain to do everything.

It's the difference between a robot that blindly follows a map and a robot that actually drives.

Technical Summary

1. Problem Statement

Autonomous driving (AD) motion planning faces two critical challenges when using Imitation Learning (IL):

• Insufficient Context Understanding: Traditional IL models often minimize trajectory imitation loss (distance to ground truth) without explicitly modeling the relational dynamics between the Autonomous Vehicle (AV) and the surrounding environment (other agents, road boundaries, map structures). This leads to unsafe behaviors where the model mimics human trajectories but fails to maintain safe spacing or avoid collisions in complex scenarios.
• Lack of Context-Adaptability: Most existing planners rely on a single shared policy network. This "one-size-fits-all" approach biases the model toward common scenarios and struggles to adapt to diverse, rare, or complex traffic situations (e.g., dense urban intersections vs. open highways) where different decision-making criteria are required.

2. Methodology: CarPLAN Framework

The authors propose CarPLAN, a novel IL-based framework designed to enhance spatial awareness and enable adaptive decision-making. The architecture consists of two primary components:

A. Displacement-Aware Predictive Encoder (DPE)

To improve the model's understanding of spatial relationships, CarPLAN introduces a self-supervised learning objective.

• Mechanism: The DPE predicts future displacement vectors between the AV and all surrounding scene elements (dynamic agents and static map features) over a prediction horizon ($T_f$).
• Training: It employs a Displacement-Aware Predictive Loss that minimizes the error between predicted and ground-truth displacement vectors. This forces the encoder to learn feature representations that explicitly encode relative spacing and relational dynamics.
• Inference: Crucially, this predictive head is used only during training as a supervisory signal. It is deactivated during inference, meaning it adds no computational overhead to the runtime system.

B. Context-Adaptive Multi-Expert Decoder (CMD)

To handle diverse driving contexts, CarPLAN replaces the standard decoder with a Mixture of Experts (MoE) framework.

• Scene-Aware Router: A router module analyzes the current scene structure (using Displacement-Aware Features and Trajectory Queries) to dynamically select the most suitable experts for the current situation.
• Expert Architecture:
  • Shared Experts: Always activated to capture general, context-invariant driving patterns.
  • Routed Experts: A pool of specialized decoders (e.g., 16 experts) from which the router selects the top-$K$ (e.g., top-2) based on the specific scene context.
• Process: The selected experts refine the trajectory query, allowing the model to adapt its planning strategy (e.g., aggressive overtaking vs. conservative following) based on the specific environment.

C. Loss Function

The total training loss combines three terms:

1. Planning Loss ($L_{plan}$): Standard smooth L1 loss and cross-entropy for trajectory and score prediction.
2. Displacement-Aware Predictive Loss ($L_{disp}$): The self-supervised loss for predicting future displacements.
3. Expert Balancing Loss ($L_{bal}$): Ensures a balanced utilization of the routed experts to prevent collapse.

3. Key Contributions

1. Displacement-Aware Predictive Encoding (DPE): A novel approach to enhance spatial awareness by predicting future relative displacements. This improves safety by ensuring the planner accounts for relational spacing, not just absolute trajectory matching.
2. Context-Adaptive Multi-Expert Decoder (CMD): The first application of a dynamic MoE framework in AD motion planning, enabling the model to switch between specialized policies based on scene complexity and structure.
3. State-of-the-Art Performance: The framework achieves superior results on standard benchmarks without requiring complex post-processing or rule-based overrides.

4. Experimental Results

The authors evaluated CarPLAN on the nuPlan and Waymax benchmarks.

• nuPlan Benchmark (Val14, Test14-Hard, Test14-Random):

  • CarPLAN achieved State-of-the-Art (SOTA) performance across all Closed-Loop Simulation (CLS) metrics.
  • Val14: Achieved a CLS-NR score of 91.4 and CLS-R of 84.6, outperforming previous bests (Diffusion-Planner, BeTopNet) by significant margins.
  • Test14-Hard: Demonstrated robustness in complex scenarios with a CLS-NR of 78.9 and CLS-R of 72.5.
  • Hybrid Settings: When combined with post-processing, CarPLAN reached a CLS-NR of 95.0 on Val14.

• Waymax Benchmark:

  • Demonstrated strong generalization capabilities, achieving the highest Arrival Rate (AR) and lowest Collision Rate (CR) compared to baselines.

• Ablation Studies:

  • Removing DPE reduced CLS-NR by 1.5 points.
  • Removing the MoE structure (CMD) reduced performance by 1.9 points.
  • Combining both components yielded a total improvement of 3.1 points over the baseline.
  • Efficiency: Despite the MoE architecture, the system maintains real-time inference (~15 FPS) because the DPE is excluded at inference time, and the MoE overhead is manageable.

5. Significance and Impact

• Safety and Robustness: By explicitly modeling relative displacement, CarPLAN addresses the "safety gap" often found in pure imitation learning, where models might mimic unsafe human behaviors if they don't understand the underlying spatial constraints.
• Adaptability: The MoE approach solves the limitation of single-policy networks, allowing the system to handle the "long tail" of rare and complex driving scenarios more effectively.
• Practicality: The design ensures that the enhanced training objectives (DPE) do not compromise inference speed, making it viable for real-world deployment.
• Future Direction: The paper suggests extending this framework to an end-to-end system where perception and planning are jointly trained, and moving toward action-conditioned world models.

In summary, CarPLAN represents a significant advancement in autonomous driving planning by bridging the gap between data-driven imitation and robust, context-aware decision-making through innovative architectural designs.