When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit

Technical Summary: When Shared Rollouts Fail in Defensive Driving

Problem Statement
The paper identifies a structural validity risk in the NAVSIM v2.2 defensive driving evaluation framework, specifically within its "reference-conditioned forgiveness" mechanism. In standard re-simulation benchmarks, an agent's trajectory is rolled forward and scored against a logged human reference. If the human reference fails a compliance channel (e.g., lane keeping), the agent may be "forgiven" on that channel to avoid penalizing it for violations inherent to the reference data.

The authors argue that this mechanism becomes defective when the agent and the reference share an unstable rollout transformation (specifically a velocity refit step). If the numerical backend produces an unstable, unphysical rollout for the reference (e.g., a kilometer-scale off-road deviation), the reference-conditioned forgiveness rule propagates this shared failure into "compliance credit" for the agent. This creates a "shared-failure forgiveness collapse," where agents that ignore surrounding actors (actor-blind probes) can systematically outrank sophisticated, actor-aware planners because the score basis effectively erases the structural constraints that would normally penalize invalid trajectories.

Methodology
The authors conduct a "score basis audit" to isolate the cause of this ranking anomaly. Their methodology involves:

1. Blind Probes: They introduce two diagnostic agents:
   • Ignore-All: A route-blind, fixed-straight policy that receives no surrounding actor input.
   • Route-Aware Actor-Blind: A policy that follows mission roadblocks using static map data but ignores all dynamic actors.
   • Hypothesis: If these blind probes outrank actor-aware comparators (like Human Replay or PDM-Closed) under a score intended to measure actor-sensitive driving, the score basis has failed a necessary sanity check.
2. Controlled Interventions: They test the score basis under various conditions to isolate the trigger:
   • Stock Filter-On/Off: Comparing scores with and without reference-conditioned forgiveness.
   • Solver Replacement: Replacing the default pseudoinverse solver (in the velocity refit step) with a direct solve or Hermitian pseudoinverse.
   • Two-Sided Diagnostic: Applying finite-difference kinematics symmetrically to both the agent proposal and the human reference to bypass the shared unstable refit.
   • Dependency Stack Control: Running the identical scorer source code with updated numerical dependencies (NumPy 2.x, OpenBLAS 0.3.27) to check for dependency-sensitive behavior.
3. Metrics: They analyze Extended Predictive Driver Model Score (EPDMS) rankings, core compliance channel overwrite rates (Drivable Area, Driving Direction, Lane Keeping), and rollout divergence (path deviation).

Key Results

• Anomalous Ranking: Under the affected "stock filter-on" condition, the Ignore-All probe (79.6 EPDMS) and the Route-Aware Actor-Blind probe (79.2 EPDMS) outrank Human Replay (74.0 EPDMS) and PDM-Closed (67.1 EPDMS) on the full 12,146-token navtest split.
• Mechanism Identification: The anomaly is driven by a dependency-sensitive numerical instability in the default pseudoinverse velocity refit. This instability causes the reference trajectory to diverge massively (median path length of 32.11 km vs. 20.41 m in controls), triggering failure on structural channels (DAC, DDC, LK) for the reference.
• Forgiveness Collapse: Because the reference fails these channels, the "filter-on" rule overwrites the agent's corresponding failures with passes. For the Ignore-All probe, this results in a 99.98% pass rate on structural channels despite the rollout being unphysical, effectively removing the constraints that differentiate safe driving from random motion.
• Restoration of Order:
  • Disabling forgiveness ("filter-off") exposes the failures, resulting in near-zero scores for all agents.
  • Replacing the solver (direct solve or Hermitian pseudoinverse) or using the updated dependency stack eliminates the rollout divergence. This restores the correct "blind-last" ordering, where Human Replay and PDM-Closed significantly outperform the blind probes.
  • The two-sided diagnostic confirms that the instability is in the shared transformation, not the scoring logic itself.

Key Contributions

1. Definition of Shared-Failure Forgiveness Collapse: The authors define a specific failure mode where reference-conditioned forgiveness, combined with a shared unstable transformation, maps structurally different failures into a common "pass" state, invalidating the score's ability to distinguish actor-aware policies.
2. Empirical Evidence of Instability: They demonstrate that the documented NAVSIM v2.2 stack, under specific dependency conditions, ranks actor-blind probes above actor-aware comparators, falsifying the necessary condition for defensive driving claims.
3. Audit Protocol: They propose a rigorous audit protocol for reference-conditioned re-simulation metrics, requiring:
   • Disclosure of score basis and dependency stacks.
   • Reporting of reference failure masks and overwrite rates.
   • Inclusion of route-blind and route-aware actor-blind probes.
   • Rollout stability tests to ensure shared transformations do not diverge.

Significance and Claims
The paper claims that the numerical instability in the shared velocity refit is the direct trigger, while reference-conditioned forgiveness acts as the propagation mechanism that converts this instability into a ranking inversion. The authors emphasize that this is a diagnostic finding, not a replacement metric. They argue that a score used to support claims of "actor-sensitive" or "defensive" driving must preserve the distinction between policies that observe actors and those that do not. When a blind probe wins, it indicates a failure of the score basis to capture the intended behavioral distinctions.

The authors remain modest about the scope, noting that their findings are specific to the audited configuration (NAVSIM v2.2, specific backend, and dependency stack) and do not necessarily imply failure across all platforms or branches. However, they assert that the lack of dependency disclosure in published evaluations makes it difficult to determine how often this condition has affected reported results. The paper concludes that before using such scores for defensive driving claims, the community must adopt transparency regarding dependency stacks, overwrite reporting, and rollout stability.