🤖 AI

An Open-Source Modular Benchmark for Diffusion-Based Motion Planning in Closed-Loop Autonomous Driving

This paper introduces an open-source, modular benchmark that integrates a C++-optimized diffusion-based motion planner into the ROS 2 Autoware stack, enabling real-time, runtime-configurable closed-loop evaluation and demonstrating significant latency and accuracy improvements through systematic comparisons of solver algorithms and step counts.

Original authors: Yun Li, Simon Thompson, Yidu Zhang, Ehsan Javanmardi, Manabu Tsukada

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yun Li, Simon Thompson, Yidu Zhang, Ehsan Javanmardi, Manabu Tsukada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: From "Black Box" to "Lego Set"

Imagine you have a self-driving car. To drive safely, the car needs a "brain" that plans its path forward, deciding where to steer and how fast to go. Recently, scientists have built a very smart type of brain using something called Diffusion Models. Think of these models like an artist who starts with a canvas full of static noise (like TV snow) and slowly cleans it up, step-by-step, until a perfect picture of a driving path emerges.

These new "artist brains" are incredibly good at planning paths. However, there's a big problem: they are currently built like sealed, monolithic concrete blocks.

The "Concrete Block" Problem: Once you build this block, you can't change anything inside it without breaking the whole thing and starting over. If you want to see how the car thinks during the process, or if you want to speed it up by changing a few settings, you can't. You have to rebuild the whole engine.
The "Fake Drive" Problem: Most tests of these brains happen in a video game simulator where the car doesn't actually talk to the real car's computer. It's like testing a race car engine on a treadmill in a garage. It looks fast, but we don't know if it can handle the heat, the noise, and the traffic of a real highway.

This paper introduces a new way to build these brains. Instead of a concrete block, they turned the system into a modular Lego set. They broke the giant brain into three separate, interchangeable parts and built a custom "controller" (written in C++) to run them.

The Three Main Upgrades

1. The "Context" vs. The "Thinking" (Encoder Caching)

The Analogy: Imagine you are writing a story.

The Old Way (Monolithic): Every time you write a new sentence, you re-read the entire history of the story, re-analyze the characters, and re-check the setting from scratch. It's incredibly slow because you keep doing the same homework over and over.
The New Way (Modular): You read the story and set the scene once at the beginning. You write that "scene summary" on a sticky note. Then, for every new sentence, you just look at the sticky note and do the creative writing. You don't re-read the whole book.

The Result: The researchers found that the "scene setting" part of the AI takes up most of the time. By doing it once and saving the result (caching), they made the system 3.2 times faster. This is the difference between a car that can't keep up with traffic and one that can.

2. The "Smart Solver" (Second-Order vs. First-Order)

The Analogy: Imagine you are trying to guess the temperature outside based on a few clues.

First-Order (Dumb Guess): You guess, "It's probably 20 degrees." Then you guess again, "Maybe 21?" You take small, slow steps to get the answer.
Second-Order (Smart Guess): You guess 20, then you look at how fast the temperature is changing and the direction it's going. You make a "correction" to your guess. You jump straight to the right answer much faster.

The Result: By using a "smarter" math method (called a second-order solver), the car can plan its path with 41% more accuracy using the same number of steps. Or, it can use fewer steps to get the same accuracy.

3. The "Real-World" Test (Closed-Loop)

The Analogy:

Old Tests: Like watching a movie of a car driving. The car follows a script, but the movie doesn't react if the car makes a mistake.
New Test: Like playing a video game where the car actually drives itself. If the car hesitates, the other cars in the game react. If the car is too slow, the traffic jams.

The Result: The researchers put their new "Lego brain" into a real self-driving software stack (Autoware) and tested it in a realistic simulator. They proved it works in real-time, reacting to traffic lights and other cars just like a real car would.

Why This Matters

Before this paper, if a researcher wanted to test a new way to make the AI think faster, they had to:

Break the giant concrete block.
Rebuild the whole thing.
Re-train the AI (which takes days).
Hope it still works.

Now, thanks to this paper:

Researchers can swap out the "thinking" part like changing a battery in a remote control.
They can change settings (like "how many steps to think") while the car is running.
They can see exactly what the AI is thinking at every single step, turning a "black box" into a clear window.

The Bottom Line

The authors took a powerful but rigid AI planner, took it apart, and rebuilt it so it's faster, smarter, and easier to tinker with. They proved that by simply organizing the code better (modular design) and using a smarter math trick (second-order solving), they can make self-driving cars plan their routes in real-time without crashing, all while keeping the system open for anyone to improve.

In short: They turned a rigid, slow, mysterious machine into a fast, flexible, and transparent tool that actually works in the real world.

1. Problem Statement

Despite diffusion-based motion planners achieving state-of-the-art results on benchmarks like nuPlan, their evaluation and deployment in real-world, closed-loop autonomous driving (AD) stacks remain largely unexplored. The paper identifies two critical gaps in current research:

The "Pseudo-Closed-Loop" Gap: Existing evaluations often rely on Python-based simulators or dataset replay (e.g., nuPlan reactive agents). These setups abstract away critical production constraints, such as ROS 2 communication latency, sensor preprocessing pipelines, and real-time scheduling on shared vehicle compute. Consequently, the performance gap between "benchmark metrics" and "vehicle-deployable performance" is unmeasured.
The "Monolithic Deployment" Gap: Current deployment pipelines export the entire diffusion process (including the noise schedule, solver logic, and iterative DiT blocks) into a single monolithic ONNX graph.
- Rigid Parameters: Solver parameters (step count $N$ , solver order, noise schedule) are frozen at export time. Changing them requires re-exporting and recompiling the model.
- Redundant Computation: In a monolithic graph, the context encoder is re-executed at every denoising step, leading to massive computational waste.
- Black Box: Researchers cannot observe intermediate denoising steps or dynamically adjust the solver during runtime.

2. Methodology

The authors propose an open-source modular benchmark that decomposes a monolithic diffusion planner into independently executable modules, orchestrated by a native C++ solver within the Autoware (ROS 2) stack.

A. Graph Decomposition (ONNX GraphSurgeon)

Using ONNX GraphSurgeon, the authors decompose the original 18,398-node monolithic graph (based on the Diffusion Planner architecture) into three distinct modules:

Context Encoder (3,417 nodes): Processes scene context (ego pose, neighbors, map, route, signals). It runs once per planning cycle, and its output is cached.
DiT Core (1,237 nodes): The diffusion transformer backbone. It is invoked $N$ times (where $N$ is the number of denoising steps) with the cached context embedding.
Turn Indicator Classifier (7 nodes): A lightweight linear layer for post-hoc signal prediction.

B. Native C++ Solver Implementation

The iterative denoising loop is reimplemented in native C++ using the DPM-Solver++ algorithm.

Runtime Configurability: Parameters such as step count ( $N \in [1, 50]$ ), solver order (1st or 2nd), and noise schedule ( $\beta_0, \beta_1$ ) are exposed as ROS 2 parameters. They can be changed at runtime without model recompilation.
Encoder Caching: The context embedding is computed once and reused for all $N$ DiT invocations, eliminating redundant computation.
Integration: The system is deployed as a drop-in ROS 2 node within the Autoware stack, validated using AWSIM (a closed-loop simulator integrated with Autoware).

C. Solver Logic

The system supports:

DPM-Solver++ (1st and 2nd order): High-order ODE solvers for efficient sampling.
DDIM: For deterministic sampling comparison.
Timestep Scheduling: Uses logSNR-uniform spacing to concentrate steps in high-noise regions where trajectory changes are most rapid.

3. Key Contributions

Open-Source Modular Benchmark: A framework that decomposes a monolithic ONNX diffusion planner into executable modules, enabling plug-and-play solver evaluation within a production-grade ROS 2 stack (Autoware + AWSIM).
Runtime-Configurable Inference: A native C++ implementation allowing researchers to study solver behavior (step count, order, noise schedule) without retraining or recompiling the neural network.
Systematic Solver Analysis: The first evaluation of diffusion planners under true closed-loop conditions with per-step observability, moving beyond "black box" monolithic deployments.
Codebase Release: The complete codebase is released to facilitate reproducible research from simulation to real-vehicle deployment.

4. Key Results

Experiments were conducted on an Intel Core i9 CPU (isolating architectural benefits from GPU acceleration) and validated in AWSIM urban driving scenarios.

Latency Reduction via Encoder Caching:
- The monolithic architecture redundantly re-executes the encoder $N+1$ times.
- The modular approach runs the encoder once.
- Result: A 3.2× latency reduction (e.g., from 119 ms to 37 ms for $N=3$ ) while maintaining identical prediction accuracy. This brings inference times within the typical 100 ms planning budget of Autoware nodes.
Solver Order Impact:
- Second-order DPM-Solver++ significantly outperforms first-order solvers.
- Result: At $N=3$ , second-order solving reduces the Final Displacement Error (FDE) by 41% (from 0.61 m to 0.36 m) compared to first-order. It achieves the accuracy of a 1st-order solver with $N=5$ using only $N=3$ steps.
Timestep Scheduling:
- Using a dedicated timestep schedule for low step counts is crucial. Truncating a 10-step schedule to 3 steps yields poor results (2.53 m FDE), whereas a dedicated 3-step schedule yields 0.36 m FDE (7× improvement).
Numerical Equivalence:
- The modular pipeline maintains numerical equivalence with the monolithic model (max error $< 10^{-5}$ ), ensuring the decomposition does not degrade model semantics.
Closed-Loop Validation:
- Successfully deployed in AWSIM for 600+ planning cycles in an urban environment, maintaining a ~10 Hz output rate without collisions.

5. Significance

This work bridges the critical gap between academic diffusion model research and industrial autonomous driving deployment.

Realism: It moves evaluation from "pseudo-closed-loop" (Python simulators) to "true closed-loop" (ROS 2 stacks with communication and scheduling constraints).
Flexibility: By decoupling the solver from the model, it enables Anytime Planning, where the system can dynamically adjust the number of denoising steps based on available compute time (e.g., falling back to $N=3$ if the system is overloaded) while maintaining safety.
Reproducibility: It provides a standardized, open-source platform for comparing different solvers and scheduling strategies under realistic conditions, addressing the lack of transparency in monolithic TensorRT deployments.

In summary, the paper demonstrates that modular decomposition and encoder caching are essential for making diffusion-based planners viable for real-time, safety-critical autonomous driving, offering a clear path from simulation benchmarks to on-vehicle deployment.