Technical Summary: Neural Token Reconstruction for Scene Token Bottleneck in End-to-End Driving

1. Problem Statement

Recent perception-free end-to-end (E2E) autonomous driving methods bypass explicit perception outputs (e.g., detection, segmentation) by compressing dense image patch tokens into a compact set of "scene tokens." These tokens serve as the sole visual interface between the vision backbone and the trajectory planner.

The core limitation identified in this work is that these scene tokens receive supervision solely from the planning objective (trajectory generation and scoring). This provides only weak constraints on the fine-grained visual information preserved during compression. Consequently, the learned scene tokens tend to develop redundant representations with highly overlapping spatial attention patterns, failing to preserve diverse and informative visual evidence necessary for robust planning. This issue is analogous to known failures in Vision Transformers where aggregation tokens drift toward lazy encoding without explicit supervision.

The fundamental question addressed is: How can compact scene tokens preserve planning-relevant visual information without explicit perception supervision?

2. Methodology: Neural Token Reconstruction (NTR)

The authors propose Neural Token Reconstruction (NTR), a self-distillation framework designed to directly supervise the scene-token bottleneck during training. The framework operates without modifying the planner architecture or introducing explicit perception heads at inference time.

Core Mechanism

NTR introduces a masked latent reconstruction objective that forces the compact scene tokens to act as the sole memory for reconstructing masked patch-level latent features.

1. Self-Distillation Setup:
   • Online Encoder: Processes masked images to generate scene tokens and masked patch features.
   • EMA Teacher: An Exponential Moving Average (EMA) copy of the online encoder processes clean images to provide stop-gradient latent targets ($Z_{v,*}$).
2. Reconstruction Process:
   • A lightweight decoder attempts to reconstruct the teacher's latent features at specific masked locations.
   • Crucially, the decoder only cross-attends to the compact scene tokens ($\tilde{S}_{v,m}$) and has no access to the dense online patch outputs.
   • This architectural constraint ensures that reconstruction gradients must pass exclusively through the scene-token bottleneck, compelling the tokens to encode richer, less redundant visual evidence to satisfy the reconstruction loss.

Semantic-Guided Allocation

To improve reconstruction efficiency and relevance, NTR utilizes semantic priors derived from foundation-model segmentation (e.g., SAM3).

• Instead of uniform random masking, reconstruction targets are allocated toward structured, safety-critical driving regions (agents, drivable areas, traffic control elements).
• These priors act as a weak bias during training but are completely removed at inference time, ensuring zero additional computational overhead for the deployed planner.

Training Objective

The total loss function combines the original planning loss ($L_{plan}$) with the reconstruction loss ($L_{rec}$):
$$L = L_{plan} + \lambda_{rec}L_{rec}$$
Where $L_{rec}$ is computed as the $L_1$ distance between the reconstructed latent features and the stop-gradient teacher targets at the selected patch locations.

3. Key Contributions

1. Identification of the Bottleneck: The paper identifies the scene-token compression interface as a critical representation learning challenge in perception-free E2E driving, where tokens often collapse into redundant features due to lack of direct supervision.
2. NTR Framework: Proposes a self-distillation framework using masked latent reconstruction to directly supervise the compact planning bottleneck, encouraging the retention of informative and complementary visual evidence.
3. Semantic Priors without Overhead: Introduces a method to bias reconstruction targets toward structured driving elements using foundation-model annotations, improving supervision quality without introducing explicit perception heads or inference-time costs.
4. Empirical Validation: Demonstrates that NTR improves planning performance while simultaneously reducing token redundancy and increasing the effective rank of the scene token representation.

4. Experimental Results

The authors evaluated NTR on three public benchmarks and a private large-scale dataset.

• Waymo Open Dataset (E2E Track):
  • Achieved a state-of-the-art RFS (Rater Feedback Score) of 8.0461 with ensembling (7.9982 with a single model).
  • Outperformed existing methods (e.g., RAP, Poutine, IRL-VLA) in both human-rating scores and trajectory accuracy (ADE).
• NavSim V1 & V2:
  • Achieved 94.1 PDMS and 90.9 EPDMS on the navtest split, outperforming previous SOTA methods like DrivoR and RAP.
• Token Diagnostics:
  • Analysis of pairwise cosine similarity and effective rank showed that NTR significantly reduces token redundancy and increases the effective dimensionality of the scene token memory compared to planning-only baselines.
• Real-World Deployment:
  • Validated on a private large-scale driving dataset and integrated into a real-vehicle planning stack, demonstrating practical applicability and scalability.

5. Significance and Claims

The paper claims that NTR addresses a fundamental representation learning gap in perception-free planning. By directly supervising the bottleneck through reconstruction, the method forces the compact scene tokens to learn richer and less redundant visual representations.

Key takeaways emphasized by the authors:

• Training-Time Only: All auxiliary components (reconstruction decoder, EMA teacher, semantic priors) are removed during inference, leaving the deployed planner unchanged.
• Complementary Nature: NTR focuses on the compression bottleneck rather than the visual backbone itself, making it complementary to backbone-level pretraining methods like MAE or JEPA.
• Performance vs. Efficiency: The approach achieves state-of-the-art planning performance without increasing inference-time computation or architectural complexity.

The authors conclude that direct bottleneck supervision via masked latent reconstruction is a viable and effective strategy for improving the quality of compact visual representations in end-to-end autonomous driving.