DriveStack-VLA: Render-Teacher Alignment for BEV-Based DeepStack Vision-Language-Action Model

Based on the paper provided, here is a detailed technical summary of DriveStack-VLA.

Problem Statement

Existing Vision-Language-Action (VLA) driving models leverage the world knowledge and reasoning capabilities of large pre-trained Vision-Language Models (VLMs) to predict driving trajectories from multi-view images and language commands. However, these models suffer from a lack of driving-oriented spatial intelligence. Current approaches primarily ground their policies on perspective image tokens and language priors, which are viewpoint-dependent and fail to explicitly encode the metric geometry and top-down scene structure required for precise motion planning.

This limitation manifests in two key areas:

1. Weak Visual Geometry: Policies rely on 2D appearance cues rather than robust geometric representations, making them vulnerable to perceptual coverage gaps in expert demonstrations.
2. Perceptual Misalignment: While synthetic or rasterized images can provide cleaner geometric structures and diverse counterfactuals, directly mixing them with real images introduces a domain gap. Existing methods often fail to ensure that the action decoder attends to the same planning-relevant cues (e.g., lanes, agents) in both real and synthetic domains.

Methodology: DriveStack-VLA

The authors propose DriveStack-VLA, a unified three-stage training framework built upon a large VLM backbone (specifically Qwen3-VL-4B). The framework integrates a multimodal LLM planner with an enhanced visual stack to address geometric, dataset, and trajectory sampling bottlenecks.

1. Dual Visual Modeling Components
To strengthen spatial grounding, the model employs two specific mechanisms:

• BEV DeepStack Injection: The authors inject Bird-Eye-View (BEV) representations directly into the LLM decoder using a "DeepStack-style" connection. A BEV encoder (based on BEVFormer) extracts a top-down feature map, which is projected into tokens and concatenated with perspective camera tokens at specific decoder layers. This provides the LLM with stable, metric geometric priors for long-range contexts.
• Render-Teacher Alignment: To leverage the benefits of rasterized images without introducing domain shift, the authors treat rasterized images as a "teacher" modality. They align the perceptual focus of real images with that of rasterized images through two losses:
  • Masked Camera-Token Alignment: A render-guided soft mask down-weights background tokens (which are often black in rasterized images) during feature alignment, ensuring the model focuses on foreground agents and lanes.
  • Action-to-Vision Attention Distillation: The attention distribution from action tokens to visual tokens in the real image pass is distilled from the rasterized image pass, forcing the model to attend to safety-critical cues in real images just as it does in the cleaner synthetic data.

2. Three-Stage Training Pipeline

• Stage 1: Supervised Fine-Tuning (SFT): The actor is trained using the dual visual modeling components described above. The model learns to generate action tokens from multi-view images, instructions, and ego states.
• Stage 2: Reinforcement Fine-Tuning (RFT): To improve the quality of sampled trajectories, the authors apply Group Relative Policy Optimization (GRPO). The policy is updated based on a joint reward comprising a driving reward (evaluating collision-freedom, safety margins, and drivable area compliance) and a format reward (ensuring strict decodability of action tokens).
• Stage 3: Scoring and Refinement (Self-Critic): A lightweight critic module is trained to rank candidate trajectories and conditionally refine the best one.
  • Scoring Head: Reuses the last-layer hidden states of the LLM to output a quality score for candidate trajectories.
  • Refinement Head: If the best candidate score is below a threshold, this head predicts a bounded residual to refine the trajectory, avoiding the need for slow iterative generate-critique-rewrite loops during inference.

Key Contributions

1. DriveStack-VLA Framework: A unified VLA driving framework that addresses geometric grounding, dataset coverage, and trajectory sampling simultaneously.
2. BEV DeepStack Injection: A novel method to embed top-down geometric priors directly into the LLM decoder, improving long-range planning stability.
3. Render-Teacher Alignment: A technique to align visual-action attention across real and rasterized images, allowing the model to benefit from synthetic geometric augmentation without suffering from domain misalignment.
4. Lightweight Self-Critic: A module that enables trajectory ranking and conditional refinement without redundant text generation or complex inference loops.

Experimental Results

The authors evaluated DriveStack-VLA on both real-log-driven and simulated closed-loop benchmarks:

• NAVSIMv1: Achieved a 91.6 PDMS (Predictive Driver Model Score) with RFT, outperforming all prior VLM-based methods and setting a new state-of-the-art.
• NAVSIMv2: Achieved 91.0 EPDMS (Extended Predictive Driver Model Score) with the human penalty filter enabled, establishing a new state-of-the-art under both evaluation settings.
• Bench2Drive (Closed-Loop): Achieved a driving score of 79.49 and a success rate of 56.36%, demonstrating strong transferability to complex urban simulations without requiring RFT.
• Navhard: Maintained robust performance (34.9 EPDMS) on this challenging benchmark involving out-of-distribution scenarios.

Significance and Claims

The paper claims that DriveStack-VLA successfully bridges the gap between the reasoning capabilities of large language models and the strict spatial requirements of autonomous driving. By injecting BEV features and aligning perceptual focus between real and synthetic data, the model overcomes the "weak visual geometry" limitations of previous VLA approaches. The authors emphasize that their method achieves state-of-the-art quantitative results on both open-loop (NAVSIM) and closed-loop (Bench2Drive) benchmarks, validating that a unified VLA architecture can effectively handle safety-critical planning tasks when equipped with appropriate geometric grounding and alignment strategies.