SimScale: Learning to Drive via Real-World Simulation at Scale

1. Problem Statement

End-to-end (E2E) autonomous driving systems rely heavily on large-scale real-world data to learn rational decision-making. However, existing real-world datasets collected by human experts are dominated by common, safe scenarios. Safety-critical and out-of-distribution (OOD) states (e.g., near-collisions, aggressive cut-ins, off-road drifts) are severely underrepresented.

• The Gap: Planners trained solely on human data struggle to generalize to rare or unseen situations, leading to distribution shifts and causal confusion during deployment.
• The Challenge: Simply scaling real-world data collection is inefficient and costly. While simulation can generate OOD states, current methods fail to provide feasible corresponding demonstrations (expert trajectories) for these new states, making them difficult to use for supervised learning. Furthermore, the scaling laws of simulation data in conjunction with fixed real-world data remain unexplored.

2. Methodology: The SimScale Framework

The authors propose SimScale, a scalable simulation framework that synthesizes massive unseen states from existing driving logs and generates corresponding "pseudo-expert" demonstrations. The system consists of three core components:

A. 3DGS Simulation Data Engine

To bridge the sim-to-real gap, the authors utilize 3D Gaussian Splatting (3DGS) for high-fidelity, photorealistic rendering.

• Reconstruction: They reconstruct static backgrounds and dynamic vehicle assets from real-world datasets (nuPlan) in a block-wise manner.
• Rendering: The engine takes ego and agent states (position, yaw) as input to render multi-view RGB observations from the ego's perspective. This allows for the generation of novel views that are visually consistent with the real world.

B. Pseudo-Expert Scene Simulation Pipeline

This is the core innovation for generating training data. The pipeline operates in two stages within a reactive environment (where surrounding agents interact with the ego):

1. Trajectory Perturbation: The ego trajectory is perturbed to create a new "perturbed state" (e.g., drifting off-center, near-collision). Perturbations are sampled from a clustered human trajectory vocabulary but filtered for physical validity and plausibility.
2. Pseudo-Expert Generation: Once the ego reaches the perturbed state, a "pseudo-expert" policy generates a feasible recovery trajectory. The paper compares two strategies:
   • Recovery-based Expert: Retrieves the closest human trajectory from a large vocabulary that steers the vehicle back to the human manifold. This is conservative and human-like.
   • Planner-based (Privileged) Expert: Uses a rule-based privileged planner (PDM-Closed) with access to ground-truth states to generate an optimal, exploratory trajectory. This offers diverse, non-human-like solutions for challenging OOD states.

C. Scalable Sim-Real Co-training

The framework employs a co-training strategy where models are trained on a mixture of real-world data and the generated simulation data.

• Strategy: Real data is kept constant while simulation data is progressively added (non-overlapping samples).
• Applicability: The method is model-agnostic, supporting:
  • Regression-based (e.g., LTF): Imitation learning on trajectories.
  • Diffusion-based (e.g., DiffusionDrive): Conditional denoising of trajectories.
  • Vocabulary Scoring-based (e.g., GTRS-Dense): Ranking candidate trajectories. For scoring-based planners, the authors also explore reward-only training (using EPDMS rewards without explicit trajectory supervision) in simulation.

3. Key Contributions

1. Scalable Simulation Framework: A novel pipeline that uses 3DGS and reactive simulation to generate high-fidelity OOD states with feasible pseudo-expert demonstrations, effectively bootstrapping E2E systems from limited real data.
2. Sim-Real Co-training Synergy: Demonstrates that combining real and simulated data leads to synergistic improvements in both robustness (handling safety-critical cases) and generalization (handling diverse scenarios).
3. Scaling Laws Discovery:
   • Exploratory Experts: Planner-based (exploratory) experts yield better scaling properties than conservative recovery-based experts as data volume increases.
   • Multimodal Advantage: Planners with multimodal modeling capabilities (e.g., Diffusion models) exhibit superior data scaling trends compared to single-mode regression models, which suffer from mode confusion when faced with diverse simulation supervision.
   • Reward-Only Potential: For scoring-based planners, reward signals alone in simulation can outperform trajectory supervision, provided the model is stabilized by real-world expert data.
4. State-of-the-Art Performance: The approach sets new benchmarks on the NAVSIM-v2 leaderboard.

4. Experimental Results

The authors evaluated SimScale on the NAVSIM-v2 benchmarks (navhard for safety-critical OOD scenarios and navtest for generalization) using three planner families.

• Performance Gains:
  • navhard: Significant improvements across all models. The best result (GTRS-Dense with reward-only scoring) achieved 48.0 EPDMS, a new SOTA, representing a +8.6 point increase over the baseline.
  • navtest: Consistent improvements up to +2.9 EPDMS, demonstrating better generalization across diverse conditions.
• Scaling Trends:
  • As simulation data increased (up to ~237K samples), performance improved predictably.
  • Planner-based experts showed continuous improvement with more data, whereas recovery-based experts plateaued earlier due to limited diversity.
  • Diffusion-based planners maintained linear improvement with data scaling, while regression-based models showed diminishing returns or degradation due to mode confusion.
• Reactive vs. Non-Reactive: Reactive simulation (where agents respond to the ego) significantly outperformed non-reactive simulation, highlighting the importance of interaction dynamics.

5. Significance

• Data Efficiency: SimScale proves that high-quality simulation can amplify the value of existing real-world datasets, reducing the need for massive, costly new data collection to handle long-tail scenarios.
• Architectural Insights: The paper provides crucial insights into how different planner architectures (regression vs. diffusion vs. scoring) interact with diverse simulation data, suggesting that multimodal modeling is essential for scalable autonomous driving.
• Open Source: The authors released the simulation data, code, and framework, fostering further research into sim-real co-training and data scaling laws for autonomous driving.

In conclusion, SimScale offers a systematic recipe for scaling end-to-end autonomous driving by generating diverse, high-fidelity, and supervised simulation data, effectively addressing the data scarcity problem in safety-critical scenarios.