🤖 machine learning

One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation

This paper demonstrates that a single pretrained diffusion traffic model can simultaneously serve as a high-performance motion planner, utilizing a novel SSDS decoder and DAPSE guidance, and as a controllable generator of realistic safety-critical scenarios to systematically evaluate planner robustness in closed-loop simulations.

Original authors: Arka Pal, Rajesh Kumar, Hannes Eriksson, Rémi Lacombe, Arvid Laveno Ling, Ankit Gupta, Maciej Wozniak

Published 2026-09-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Arka Pal, Rajesh Kumar, Hannes Eriksson, Rémi Lacombe, Arvid Laveno Ling, Ankit Gupta, Maciej Wozniak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Autonomous vehicles do not drive in a vacuum; they navigate a world teeming with other cars, pedestrians, and unpredictable human choices. To move safely, a self-driving car must do two difficult things at once. First, it must predict what everyone else will do next, understanding that a driver might suddenly change lanes or a pedestrian might step off a curb. Second, it must decide how to react to those predictions, choosing a path that avoids danger while staying comfortable and efficient. For years, engineers have struggled to teach computers this kind of flexible, multi-layered thinking. Traditional methods often rely on rigid rules or simple averages, which fail when the road presents rare, chaotic, or dangerous situations that have never been seen before. The core challenge is creating a system that can imagine many possible futures simultaneously and then steer itself toward the safest one, all while being tested against the very worst scenarios nature can offer.

A team of researchers at KTH Royal Institute of Technology and Zenseact has tackled this problem by developing a single artificial intelligence model that plays two distinct roles in the development of self-driving cars. Instead of building separate systems for planning a route and for testing its safety, they created one unified model based on a type of technology known as a diffusion model. Originally designed to generate images by gradually removing noise from a static picture, this technology has been adapted to understand the complex, multi-layered flow of traffic. The researchers trained this model on thousands of hours of real human driving data, teaching it to predict how a car and its neighbors will move together over time. They then demonstrated that this single trained brain could be used in two complementary ways: first, as a motion planner that guides the car safely through traffic, and second, as a scenario generator that creates realistic, dangerous situations to stress-test that same planner.

In its first role, the model acts as the driver. The researchers improved the model's ability to understand the road by changing how it processes information. Instead of looking at the car's path and the surrounding traffic as separate streams of data that are only combined at the very end, they designed the system to fuse these two streams early on. This allows the model to see the connection between a neighbor's movement and its own potential path much more clearly. When tested in a closed-loop simulation, where the car's decisions immediately affect the environment and the environment reacts back, this improved model performed significantly better than previous versions, especially in complex, reactive situations where other cars are moving unpredictably. To make the car even safer, the team added a method to guide the model's decisions without needing to retrain it. By applying a simple set of rules—such as maintaining a specific speed or staying within lane markings—the system can nudge its predictions toward safer outcomes in real time, correcting its course as it generates a path.

In its second role, the same model becomes a challenger. The researchers repurposed the AI to act as an adversary, generating traffic scenarios designed to expose weaknesses in the car's planning system. By applying different guidance rules to the "enemy" vehicles in the simulation, they could coax the model into creating specific, high-risk interactions. They taught the adversary to perform aggressive maneuvers, such as cutting in front of the self-driving car at the last moment, or to brake suddenly and unexpectedly. Crucially, these dangerous behaviors were not pre-programmed scripts; they emerged naturally from the model's understanding of traffic, guided only by the researchers' desire to create a conflict. This allowed them to generate long, realistic sequences of events where the self-driving car had to react to a car that was actively trying to create a dangerous situation, all while the rest of the traffic behaved normally.

When the researchers put their new planner to the test against these generated scenarios, they discovered a revealing truth about current self-driving technology. The planner that performed best under normal, standard conditions was actually the one that struggled the most when faced with these difficult, adversarial situations. While it handled everyday driving with high scores, it failed to react quickly enough when an aggressive car cut in or braked hard, often relying on simple braking rather than taking proactive evasive action. In contrast, a slightly less dominant planner in standard tests showed more resilience in these extreme cases. The study suggests that performing well on standard benchmarks does not guarantee that a system is robust enough for the real world. By using a single model to both drive the car and generate the tests, the researchers created a closed loop where the system is constantly challenged by the very logic it uses to navigate, revealing failure modes that would otherwise remain hidden.

The work demonstrates that a single learned understanding of traffic can serve as both a driver and a rigorous examiner. By generating realistic, safety-critical interactions like sudden lane changes and braking events, the researchers were able to show that current planners often rely too heavily on reactive braking rather than proactive maneuvering. The results indicate that while the new architecture improves performance in complex traffic, true safety requires testing against the kinds of rare, difficult interactions that standard data rarely captures. This dual-use approach offers a powerful new way to develop self-driving cars, ensuring that the systems we build are not just good at following the rules, but are also robust enough to survive the unexpected.

Technical Summary: Guided Diffusion for Planning and Scenario Generation

Problem Statement

Autonomous vehicle (AV) development faces two distinct but critical challenges that are often studied in isolation. First, learning-based motion planners must generate safe, human-like trajectories in interactive traffic environments. Standard imitation learning approaches often suffer from "mode averaging," where regression objectives fail to capture the multi-modal nature of human driving behaviors, leading to deterministic outputs that lack diversity. Second, these planners must be rigorously evaluated against rare, safety-critical interactions (long-tail scenarios) that are underrepresented in logged driving data. Standard benchmarks, dominated by nominal driving, often fail to reveal robustness issues, meaning high benchmark scores do not guarantee safety in complex, adversarial situations.

Existing methods struggle to address both simultaneously: safety-steering techniques often require costly retraining or rely on trajectory proposals that may lack realism, while scenario generators often operate in open-loop settings or fail to produce realistic multi-agent interactions.

Methodology

The authors propose a unified framework leveraging a single pretrained Diffusion Probabilistic Model (DPM) to serve two complementary roles: an ego motion planner and a controllable generator of safety-critical scenarios. The system is built upon a Diffusion Transformer (DiT) architecture trained on the nuPlan dataset.

1. Architecture: Single-Stream Dual-Stream (SSDS) Decoder

To improve the fusion of scene context and trajectory generation, the authors replace the standard DiT decoder with a Single-Stream Dual-Stream (SSDS) architecture, adapted from video generation models.

Dual-Stream Blocks: Trajectory tokens (ego and neighbors) and context tokens (scene features) are processed in separate streams. These streams interact via joint attention (concatenating queries, keys, and values) rather than the asymmetric, late cross-attention used in standard DiTs. This allows for early, symmetric interaction between modalities.
Single-Stream Stage: A subsequent stage fuses the unified representation, performing deep fusion of the context and trajectory information.
Conditioning: All layers are modulated by navigation routes and diffusion time-step embeddings via adaptive layer normalization (adaLN).

2. Planning: DAPSE Guidance

For motion planning, the authors introduce Decoupled Annealing Posterior Sampling with Energy (DAPSE). This is a training-free guidance scheme designed to steer the diffusion process toward trajectories that minimize arbitrary energy functions (e.g., target speed, safety constraints) without retraining the model.

Mechanism: DAPSE generalizes the Decoupled Annealing Posterior Sampling (DAPS) framework to arbitrary, non-analytical energy functions. It utilizes a non-Markovian forward process (DDIM) and decoupled sampling to approximate the posterior distribution.
Sampling: The method employs Langevin dynamics over inner MCMC iterations. The update step combines a reconstruction term (correcting errors from the reverse ODE trajectory), an energy guidance term (gradient of the energy function), and Langevin noise.
Advantage: Unlike Diffusion Posterior Sampling (DPS), which relies on test-time approximations that are typically reliable only near the clean sample ( $t \approx 0$ ), DAPSE avoids these specific approximation errors by utilizing a Gaussian approximation of the unguided posterior centered around the current reverse ODE trajectory estimate. This approach requires no auxiliary networks, making it suitable for real-time planning constraints.

3. Scenario Generation: Adversarial Closed-Loop Simulation

The same pretrained diffusion model is repurposed as a behavior generator for an "adversary" agent in a closed-loop nuPlan simulation.

Pipeline: The simulation includes the ego vehicle (controlled by an independent black-box planner), a guided adversary, background vehicles (Intelligent Driver Model), and static agents. The adversary's trajectory is generated via guided sampling.
Guidance Design: To ensure realistic safety-critical interactions, the authors design specific energy functions applied during the low-noise stage of the reverse diffusion process (using DPS for this specific task to allow composability):
- Time-to-Collision (TTC) with Temporal Lead: Encourages the adversary to intercept the ego's future position rather than its current one, creating causally meaningful conflicts.
- Lane Change (Cut-ins): Minimizes lateral deviation from the ego's lane centerline.
- Lead-Vehicle Braking: Penalizes longitudinal progress beyond a desired deceleration profile.
- Drivable-Area Compliance: Uses Euclidean Signed Distance Fields (ESDF) to ensure trajectories remain on the road.
Composition: These objectives can be composed sequentially (e.g., braking followed by a cut-in) to generate complex, long-tail interactions.

Key Contributions

SSDS-DiT Decoder: A novel decoder architecture that fuses scene context and trajectories via joint attention in dual-stream blocks, followed by a single-stream fusion stage, improving closed-loop planning performance.
DAPSE: A training-free guidance method for zero-shot trajectory steering that enables the enforcement of arbitrary energy constraints on pre-trained diffusion planners without auxiliary networks, utilizing a Gaussian approximation of the unguided posterior to avoid the first-order approximation errors of Diffusion Posterior Sampling (DPS) near the clean sample.
Controllable Safety-Critical Scenario Generation: A framework that repurposes the unified diffusion model to generate realistic, adversarial long-tail scenarios (aggressive cut-ins, sudden braking) in a closed-loop setting, exposing planner failure modes that standard benchmarks miss.

Experimental Results

The models were trained on nuPlan with varying dataset sizes (250K, 650K, 1M) and evaluated on Val14, Test14, and Test14-hard splits.

Planning Performance: The SSDS-DP variant demonstrated superior performance in reactive, high-interaction scenarios (e.g., +14.4 points on Test14-hard at 650K data) compared to the baseline Diffusion Planner (DP). It showed greater data efficiency, consistently outperforming the baseline on smaller datasets. However, at the 1M data scale, the performance gap narrowed, with the baseline occasionally outperforming SSDS-DP on specific splits.
Adversarial Evaluation: When tested against generated adversarial scenarios (specifically combined cut-in and braking), both planners suffered significant performance degradation.
- Robustness Insight: The SSDS-DP, which achieved higher nominal scores, experienced a larger performance drop under adversarial conditions compared to the baseline DP.
- Behavioral Analysis: Qualitative analysis revealed that both planners relied heavily on reactive braking rather than proactive evasive maneuvers (e.g., lane changes). In complex scenarios like intersections, braking alone was insufficient to avoid collisions.
- Realism: The generated adversarial scenarios maintained low "realism deviation," matching nominal driving behaviors in terms of speed and acceleration profiles while still inducing planner failures.

Significance and Claims

The paper claims that a single learned traffic prior can simultaneously serve as an effective motion planner and a rigorous evaluator within the autonomous driving development loop.

Dual-Use Capability: The unified framework demonstrates that the same model architecture can be steered toward low-energy (safe) trajectories for planning and high-risk (adversarial) interactions for stress-testing.
Limitations of Benchmarks: A central finding is that superior performance on standard benchmarks does not necessarily translate to robustness against long-tail, multi-agent interactions. The fact that the stronger nominal planner (SSDS-DP) degraded more under adversarial conditions highlights the need for systematic robustness evaluation beyond standard metrics.
Methodological Advancement: The work advances the field by providing a training-free mechanism (DAPSE) for safety steering and a method for generating realistic, controllable safety-critical scenarios in closed-loop simulation, moving beyond open-loop or trajectory-proposal-based generation.

The authors conclude that while their approach improves planning and evaluation, future work is needed to explore additional energy functions (e.g., explicit collision avoidance) and to quantitatively evaluate their effectiveness in terms of collision rates and TTC.