OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models

1. Problem Statement

Autonomous driving requires a system capable of multi-task learning involving heterogeneous outputs:

• Perception: Parallel structured outputs (e.g., 3D bounding boxes, lane detection).
• Planning: Continuous trajectory regression.
• Reasoning/Description: Autoregressive text generation.

The Core Challenge: Existing Vision-Language Models (VLMs) excel at autoregressive text generation but struggle to integrate the parallel decoding required for perception and planning without significant architectural fragmentation.

• Current Approaches: Most systems use separate decoders (dual-system) or cascaded decoders (Q-Former style). This leads to:
  • Inability to reuse pretrained weights effectively across tasks.
  • Architectural complexity and limited information flow between tasks.
  • Performance degradation when relying solely on vision (without LiDAR) due to the mismatch between native VLM structures and driving requirements.
• The Question: Can a single pretrained multimodal model handle diverse End-to-End (E2E) driving tasks with different output formats while maintaining coherent text generation and generalizing to generative modeling paradigms?

2. Methodology: OneDrive Framework

OneDrive proposes a unified single-decoder framework built upon a pretrained VLM (specifically InternVL3) that reconciles autoregressive text generation and parallel structured prediction within a single Transformer decoder.

Key Architectural Components:

1. Unified Token Representation:

   • The model concatenates all inputs into a single sequence: [Image Tokens, Detection Queries, Lane Queries, Planning Queries, Text Tokens].
   • This allows all tasks to share the same attention backbone.

2. Mixed Decoder Layers (The Core Innovation):

   • Shared Causal Attention: The model retains the pretrained LLM causal self-attention as the backbone. This allows query tokens (for detection/planning) to naturally condition on visual context (image tokens) via the original attention mechanism, eliminating the need for explicit cross-attention modules.
   • Task-Specific Adaptations:
     • Query Interaction: To support parallel structured prediction (which causal attention is not ideal for), the model adds a dedicated self-attention block only among structured perception queries (detection and lane). This enables bidirectional interaction within the query group without altering the backbone.
     • Task-Specific FFNs: The Feed-Forward Networks (FFNs) for structured queries are replaced with task-specific FFNs. The paper's diagnostic study revealed that while attention weights transfer well from text-to-driving tasks, pretrained FFNs (optimized for text) degrade performance when reused for structured tasks.
     • 3D Positional Embeddings: Added to image and query tokens to support spatial reasoning, while text tokens remain unchanged.

3. Multi-Stage Training Strategy:

   • Stage 1 (Perception-Language Pretraining): Freeze the ViT encoder. Train the mixed decoder with perception and text objectives. Fine-tune causal attention (via LoRA) and initialize perception-specific modules.
   • Stage 2 (Planning Adaptation): Introduce planning tokens. Optimize planning queries and heads while keeping perception modules fixed.
   • Stage 3 (Joint Finetuning): Jointly fine-tune all modules (including ViT) with a combined loss function ($\mathcal{L}_{joint} = \lambda_{perc}\mathcal{L}_{perc} + \lambda_{plan}\mathcal{L}_{plan} + \mathcal{L}_{text}$).

3. Key Contributions

1. Transferability Insight: Demonstrated that pretrained VLM causal attention is highly transferable to modeling relationships between queries and visual features, whereas pretrained FFNs are not suitable for heterogeneous downstream tasks and must be replaced.
2. Unified Architecture (OneDrive): Proposed the first framework to perform autoregressive text generation, parallel perception, and trajectory planning within a single Transformer decoder without isolated branches.
3. Efficiency: The unified design avoids custom attention masks and allows the use of standard hardware optimizations (e.g., FlashAttention), significantly reducing inference latency compared to cascaded or dual-system approaches.

4. Experimental Results

The framework was evaluated on nuScenes (open-loop) and NAVSIM (closed-loop) benchmarks.

• nuScenes (Open-Loop Planning):
  • Achieved 0.28m L2 error and 0.18% collision rate, setting a new state-of-the-art (SOTA) among action-based planners.
  • Outperformed strong baselines like SOLVE-E2E and ColaVLA in both accuracy and safety.
• NAVSIM (Closed-Loop Planning):
  • Achieved 86.8 PDMS (Planning Decision Making Score), outperforming the Query Decoder baseline (85.0) and other VLA models.
  • Demonstrated that the unified causal decoding design improves interaction awareness and stability.
• Inference Efficiency:
  • Reduced latency by ~40% compared to ReCogDrive (156ms vs. 263ms) and ColaVLA (513ms vs. 727ms) on NVIDIA H20 GPUs.
  • This efficiency is achieved by forwarding only the necessary "shadow" LLM layers rather than the full model repeatedly.
• Ablation Studies:
  • Confirmed that text supervision acts as a regularizer, improving collision rates in planning.
  • Showed that joint training significantly outperforms sequential adaptation.
  • Verified that the token sequence order Detection → Lane → Planning yields the best results.

5. Significance

• Bridging the Paradigm Gap: OneDrive successfully bridges the gap between the sequential nature of Large Language Models (LLMs) and the parallel nature of autonomous driving perception/planning, proving that a single architecture can handle both.
• Scalability: By preserving the native VLM structure and leveraging pretrained attention, the approach is more scalable than methods requiring heavy 3D architectural modifications (like BEV modeling).
• Practical Deployment: The significant reduction in inference latency makes VLM-based driving systems more viable for real-time, safety-critical applications.
• Future Direction: The paper suggests that future work should focus on enhancing the visual backbone's detection capabilities (currently limited by VLM downsampling) and exploring larger model scales.

In summary, OneDrive represents a paradigm shift in End-to-End autonomous driving, moving away from fragmented, multi-head architectures toward a unified, token-centric approach that leverages the full power of pretrained Vision-Language-Action models.