Senna-2: Aligning VLM and End-to-End Driving Policy for Consistent Decision Making and Planning

1. Problem Statement

Autonomous driving systems increasingly rely on Vision-Language Models (VLMs) to provide high-level semantic reasoning and decision-making, which guide End-to-End (E2E) policies responsible for low-level trajectory planning. However, a critical "consistency gap" exists between these two systems:

• Misalignment: Existing methods often fail to explicitly align the VLM's high-level decisions (e.g., "accelerate," "turn right") with the E2E planner's low-level trajectory generation.
• Consequences: This leads to trajectories that deviate from intended driving decisions, resulting in incorrect directions, mismatched speed changes, and weakened top-down guidance.
• Safety Risk: The lack of consistency reduces the system's interpretability and can cause suboptimal or unsafe driving behaviors, particularly in complex or long-tail scenarios.

2. Methodology: Senna-2

The authors propose Senna-2, a unified VLM-E2E driving policy designed to explicitly align high-level decision-making with low-level planning through a consistency-oriented three-stage training paradigm.

A. Dual-System Architecture

The model consists of three core components (illustrated in Fig. 2):

1. VLM Module: Uses Qwen2.5-VL-3B as the backbone. It processes front-view images, navigation commands, and ego states to output structured meta-actions (speed control: accelerate/decelerate/keep/stop; direction control: straight/turn/lanes).
2. Decision Adapter: A bridge module that transforms VLM outputs into features compatible with the E2E planner. It generates two types of tokens:
   • VLM Tokens: Hidden states from the VLM preserving reasoning context.
   • Decision Tokens: Learnable embeddings for specific speed and direction categories.
   • These are fused to create a VLM condition feature ($F_{vlm}$) that guides the planner.
3. E2E Driving Policy: A diffusion-based planner using a DiT (Diffusion Transformer) architecture. It takes spatio-temporal features from multi-view images and fuses the VLM condition via AdaLN (Adaptive Layer Normalization) to modulate the trajectory generation process.

B. Three-Stage Training Paradigm

The training process is designed to progressively align the two systems:

• Stage 1: Driving Pre-Training

  • VLM Pre-training: Trained via Question-Answering (QA) to predict meta-actions from trajectories.
  • E2E Pre-training: Trained as a diffusion model to predict residual trajectories.
  • Adapter Pre-training: The VLM is frozen; the adapter and E2E policy are optimized using standard planning loss to ensure the adapter can effectively bridge the modules without disrupting VLM reasoning.

• Stage 2: Open-Loop Alignment

  • Goal: Explicitly enforce consistency between VLM decisions and E2E trajectories in an offline setting.
  • Mechanism: A kinematic mapping function ($f_K$) projects the predicted trajectory back into a decision category.
  • Loss Strategy:
    • If the trajectory matches the VLM decision (Consistent): The model treats the prediction as an implicit expert signal (self-reinforcement) and skips external supervision.
    • If they mismatch (Inconsistent): The model applies explicit supervision using ground-truth expert trajectories and decision labels to correct the gap.
  • Loss Function: $L_{stage2} = (1 - C(\tau, d))(L_{E2E} + \gamma L_{VLM})$, where $C$ is a binary consistency indicator.

• Stage 3: Closed-Loop Alignment with Hierarchical Reinforcement Learning (HRL)

  • Goal: Optimize for safety and efficiency in dynamic, out-of-distribution scenarios using 3D Gaussian Splatting (3DGS) environments.
  • Mechanism: A bottom-up hierarchical optimization:
    1. Low-Level Planner: Optimized first using composite rewards based on Time-to-Collision (TTC) for safety and velocity for efficiency.
    2. High-Level Decision: Once the planner is optimized, the VLM is updated to align its decisions with the refined, safe trajectories.
  • Loss: Combines high-level decision alignment loss ($L_{high}$) and low-level planner loss ($L_{low}$).

3. Key Contributions

1. Senna-2 Framework: A unified VLM-E2E policy that explicitly aligns high-level reasoning with low-level planning, ensuring consistent decision-making.
2. Three-Stage Training Paradigm: A novel training recipe involving pre-training, open-loop consistency alignment, and closed-loop HRL, which progressively bridges the gap between semantic reasoning and trajectory execution.
3. Decision Adapter Design: A mechanism that fuses semantic context (VLM tokens) with structured interpretability (decision tokens) to effectively guide the diffusion planner.
4. Comprehensive Evaluation: Extensive validation on large-scale benchmarks demonstrating significant improvements in consistency, planning accuracy, and safety.

4. Experimental Results

The authors evaluated Senna-2 against state-of-the-art methods (e.g., Senna, VAD, TransFuser, RAD) on open-loop and closed-loop benchmarks.

• Decision-Planning Consistency:

  • Achieved a 19.3% improvement in average F1 score compared to the previous Senna model.
  • Visualizations show Senna-2 produces distinct, decision-aligned speed distributions, whereas previous methods showed scattered distributions.

• Open-Loop Performance:

  • Final Displacement Error (FDE): Reduced by 5.7% compared to prior methods.
  • Collision Rate (CR): Reduced to 0.288% (vs. 0.294% for Senna).

• Closed-Loop Performance (3DGS Environment):

  • At-Fault Collision Rate (AF-CR): Reduced by 30.6% compared to Senna.
  • Safety Metrics: Outperformed RL-based baselines (like RAD) in safety and collision avoidance, proving the gains come from the alignment strategy rather than just reinforcement learning.

• NAVSIM v2 Benchmark:

  • Achieved an EPDMS score of 86.6, surpassing the previous best model by 1.1 points, demonstrating robustness in complex driving scenes.

• Ablation Studies:

  • Confirmed that both the dual-system design and the three-stage training are essential. Removing Stage 2 (Open-Loop Alignment) significantly hurt consistency, while removing Stage 3 (Closed-Loop HRL) degraded closed-loop safety.

5. Significance and Limitations

Significance:

• Safety & Reliability: Senna-2 addresses the critical "consistency gap" in autonomous driving, ensuring that high-level intentions are faithfully executed by low-level controllers.
• Interpretability: The system acts as a human-machine interface, providing explainable decisions (e.g., "decelerate due to cut-in") that directly influence the trajectory.
• Generalization: By combining VLM generalization with E2E adaptability, the system handles long-tail scenarios more effectively than existing methods.

Limitations:

• Real-Time Inference: The VLM currently cannot achieve real-time inference (10 Hz) on on-board edge devices.
• Asynchronous Operation: The VLM and E2E policy operate asynchronously, relying on a memory bank to cache VLM features. Fully synchronous cooperation requires further hardware optimization.

In conclusion, Senna-2 represents a significant step forward in creating explainable, safe, and consistent autonomous driving systems by rigorously aligning semantic reasoning with trajectory planning.