💻 computer science

Do Open-Loop Metrics Predict Closed-Loop Driving? A Cross-Benchmark Correlation Study of NAVSIM and Bench2Drive

This study analyzes the correlation between NAVSIM open-loop metrics and Bench2Drive closed-loop performance across state-of-the-art methods, revealing that while aggregate NAVSIM scores show strong but non-monotonic correlation with real-world driving success, Ego Progress is the most predictive single metric and a simplified three-metric formula can effectively replace the full five-metric score for ranking purposes.

Original authors: Yiru Wang, Anqing Jiang, Shuo Wang, Yuwen Heng, Hai Yang, Yang Chen, Hao Sun

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiru Wang, Anqing Jiang, Shuo Wang, Yuwen Heng, Hai Yang, Yang Chen, Hao Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to drive a car. To see if the robot is good, you have two ways to test it:

The "Paper Test" (Open-Loop): You give the robot a map and a set of instructions, and you ask it to draw a line on a piece of paper showing where it would go. You then compare that line to the perfect path. This is fast, cheap, and you can do it a thousand times in a second.
The "Real Drive" (Closed-Loop): You actually put the robot in a car (or a super-realistic video game) and let it drive. The car reacts to traffic lights, other cars, and pedestrians. If the robot gets stuck or drives too slowly, it fails. This is the "gold standard," but it takes hours, costs a lot of money, and is hard to repeat exactly the same way.

The Big Question: Can the "Paper Test" reliably predict how the robot will do in the "Real Drive"?

For a long time, the answer was no. Old tests just measured how far off the robot's drawing was from the perfect line (like measuring the distance between two dots). The paper shows that even if a robot draws a nearly perfect line, it might still crash or get stuck in real life.

What This Paper Did

The authors looked at 8 different top-tier robot drivers. They checked how these robots did on the "Paper Test" (using a new, smarter test called NAVSIM) and compared it to how they actually did on the "Real Drive" (using a test called Bench2Drive).

Here is what they found, explained simply:

1. The "Safety vs. Speed" Trap

The new "Paper Test" (NAVSIM) is much better than the old ones. It checks for safety (did you hit anything?) and progress (did you move forward?).

The Problem: Some robots are too safe. They drive like a turtle, stopping at every little shadow to make sure they don't hit anything.
The Result: On the "Paper Test," these turtle-robots get high scores because they never hit anything. But in the "Real Drive," they get penalized for driving too slowly or getting stuck in traffic. They fail the real test because they are too cautious.
The Analogy: Imagine a student who answers every question on a test by saying "I don't know" to avoid getting a single point wrong. On a test that only counts wrong answers, they get an A. But on a test that requires you to actually finish the exam, they get an F because they didn't finish.

2. The Secret Ingredient: "Ego Progress"

The researchers broke down the "Paper Test" into its parts to see which one actually predicted real-world success.

They found that the most important number wasn't "Did you crash?" (Safety). It was "Did you move forward?" (Progress).
The Analogy: Think of a marathon. If you run perfectly without tripping (Safety) but you stop to tie your shoe every 10 seconds, you won't win the race. The robot needs to keep moving forward. The "Progress" score was the single best predictor of whether the robot would actually finish the drive successfully.

3. The "Snowball Effect"

Why do small mistakes in the "Paper Test" become huge failures in the "Real Drive"?

The Analogy: Imagine you are walking down a hallway. If you take one tiny step to the left, it doesn't matter. But if you keep taking tiny steps to the left for 10 minutes, you will eventually walk into a wall.
In the "Paper Test," the robot only plans 4 seconds ahead. A tiny hesitation might look like a small error. But in the "Real Drive," that tiny hesitation causes the robot to miss a green light, wait for a red light, fall behind schedule, and eventually get timed out. Small errors "snowball" into total failure.

4. A Simpler Formula

The "Paper Test" uses a complex formula with 5 different numbers to calculate a final score. The authors realized that two of those numbers (how comfortable the ride is and how close you are to a crash) were basically the same for all the top robots—they were all perfect at those things.

The Discovery: You can throw away the complex formula and just use 3 simple numbers: "Did you crash?", "Did you stay in your lane?", and "Did you move forward?"
The Result: This super-simple formula predicted the real-world results just as well as the complex one. It's like realizing you don't need a 50-page report to know if a car is good; you just need to know if it moves and doesn't crash.

The Bottom Line

The paper concludes that while we can't perfectly predict real driving just by looking at a drawing, we are getting much closer.

Don't just look at safety: A robot that is safe but doesn't move is a bad driver.
Watch the "Progress": The ability to keep moving forward is the best sign of a good driver.
Simplify: We don't need overly complex scoring systems to tell good drivers from bad ones; a simple focus on safety and movement works best right now.

The authors warn that as robots get better, we might need to tweak these rules again, but for now, this "Progress" metric is the key to knowing who will actually succeed on the road.

1. Problem Statement

Evaluating autonomous driving (AD) planners involves a trade-off between open-loop and closed-loop evaluation:

Closed-loop evaluation (e.g., Bench2Drive on CARLA) is the "gold standard" for assessing real-world competence but is computationally expensive, suffers from high variance, and requires complex infrastructure.
Open-loop evaluation (e.g., NAVSIM) is fast and reproducible but traditionally relies on geometric metrics like Average Displacement Error (ADE) and Final Displacement Error (FDE).
The Core Question: Can modern, safety-aware open-loop metrics (specifically those from NAVSIM v2) reliably predict closed-loop driving performance? Prior work suggests traditional metrics fail at this, but it is unclear if the new paradigm of NAVSIM bridges the gap.

2. Methodology

The authors conducted a systematic cross-benchmark correlation study using a paired dataset of state-of-the-art (SOTA) methods.

Data Collection:
- The team cross-referenced published results from 15 methods that reported on both NAVSIM (open-loop) and Bench2Drive (closed-loop).
- They filtered for methods using consistent model configurations across both benchmarks to avoid confounding variables (e.g., different backbones or training data).
- Final Dataset: 8 methods with complete paired data points (NAVSIM sub-metrics vs. Bench2Drive Driving Score).
Metrics Analyzed:
- Open-Loop (NAVSIM): PDM Score (PDMS) and its sub-metrics: No at-fault Collision (NC), Drivable Area Compliance (DAC), Time-to-Collision (TTC), Ego Progress (EP), and Comfort (C).
- Closed-Loop (Bench2Drive): Driving Score (DS), which combines route completion with penalties for infractions (collisions, slow driving, timeouts).
Statistical Analysis:
- Computed Spearman rank correlation ( $\rho$ ) and Kendall's $\tau$ to measure the relationship between open-loop scores and closed-loop performance.
- Investigated specific "ranking inversions" where a method's open-loop rank differed significantly from its closed-loop rank.
- Proposed and tested a simplified proxy metric (CL-Proxy) to determine if fewer sub-metrics could achieve the same predictive power.

3. Key Findings & Results

A. Traditional Metrics Fail

Finding: Traditional displacement metrics (L2 error, ADE, FDE) show no significant correlation with closed-loop Driving Score ( $\rho = -0.36, p = 0.43$ ).
Implication: Minimizing geometric trajectory error does not guarantee safe or successful driving in interactive environments.

B. NAVSIM PDMS Shows Strong but Non-Monotonic Correlation

Finding: The aggregate NAVSIM PDM Score (PDMS) exhibits a strong positive correlation with Bench2Drive DS ( $\rho = 0.90, p = 0.002$ ).
Nuance: The correlation is not perfectly monotonic. There are notable ranking inversions.
- Example: SafeDrive ranks 3rd in PDMS (91.6) but drops to 5th in DS (66.77).
- Cause: SafeDrive optimizes heavily for safety (NC, DAC, TTC), achieving near-perfect scores, but sacrifices Ego Progress (EP). In closed-loop, this conservatism triggers "Too Slow" and "Timeout" penalties, drastically reducing the Driving Score.

C. Ego Progress (EP) is the Strongest Predictor

Finding: Among individual NAVSIM sub-metrics, Ego Progress (EP) is the single best predictor of closed-loop success ( $\rho = 0.83$ ).
Surprise: The most safety-critical metric, No at-fault Collision (NC), shows only weak-to-moderate correlation ( $\rho = 0.45$ ).
Reasoning: Current SOTA methods already achieve very high NC scores (ceiling effect, e.g., >98%). Therefore, marginal improvements in NC do not translate to better closed-loop performance, whereas the ability to make progress (EP) is the primary differentiator.

D. The "Snowball Effect"

The paper identifies a mechanism for the residual gap between open and closed-loop: The Snowball Effect.
Small deviations in a 4-second open-loop trajectory (e.g., driving slightly slow) may incur minor penalties in NAVSIM. However, in a multi-minute closed-loop route, these small delays compound: missing a green light $\rightarrow$ waiting for a full red cycle $\rightarrow$ falling behind schedule $\rightarrow$ triggering severe "Too Slow" or timeout penalties. This explains why EP (a measure of progress tendency) is more predictive than instantaneous safety metrics.

E. A Simpler Proxy: CL-Proxy

The authors propose a simplified formula: $CL\text{-}Proxy = NC \times DAC \times EP$ .
Result: This 3-metric formula achieves the exact same rank correlation ( $\rho = 0.90$ ) as the full 5-metric PDMS on the dataset.
Implication: For current top-tier methods, TTC and Comfort add negligible marginal information for ranking purposes because they are either narrowly distributed or saturated (near 100%).

4. Key Contributions

Cross-Benchmark Dataset: Compiled the first paired dataset of 8 SOTA methods evaluated on both NAVSIM and Bench2Drive.
Quantitative Gap Analysis: Demonstrated that while NAVSIM PDMS correlates strongly with closed-loop performance, it suffers from ranking inversions due to the safety–progress trade-off.
Metric Hierarchy: Identified Ego Progress (EP) as the critical predictor of closed-loop success, surpassing collision metrics in predictive power for current SOTA models.
Simplified Proxy: Proposed CL-Proxy, a lightweight 3-metric formula that matches the predictive power of the complex 5-metric PDMS, suggesting current benchmarks can be streamlined.
Mechanism Identification: Proposed the Snowball Effect as the primary reason small open-loop deviations lead to closed-loop failures.

5. Significance & Recommendations

For Benchmark Design:
- Report Sub-metrics: Aggregates hide the safety–progress trade-off. Leaderboards must display individual sub-metrics (especially EP).
- Introduce Slow-Driving Penalties: NAVSIM should adopt a "Too Slow" penalty similar to Bench2Drive to better align open-loop evaluation with closed-loop outcomes.
- Simplify Aggregation: Current top methods saturate on Comfort and TTC; these metrics may be deprioritized for ranking in favor of EP, NC, and DAC.
For Research:
- Researchers should not rely solely on aggregate open-loop scores. Optimizing for safety (NC) at the expense of progress (EP) leads to poor real-world performance.
- Future work should focus on sub-metric regression across architectures rather than relying on single aggregate scores.

Conclusion

The paper concludes that while open-loop metrics have improved significantly with NAVSIM, they are not perfect proxies. The safety–progress trade-off and the snowball effect create a gap where overly conservative planners fail in closed-loop. By focusing on Ego Progress and using a simplified CL-Proxy metric, the community can achieve more reliable open-loop predictions of closed-loop driving competence.