Technical Summary: FLARE – Learning Future-Aware Latent Representations from Vision-Language Models for Autonomous Driving

1. Problem Statement

Current end-to-end autonomous driving (AD) systems face two primary limitations when integrating Vision-Language Models (VLMs):

1. Annotation Bottleneck & Semantic Bias: Existing approaches rely heavily on labor-intensive language annotations (e.g., Visual Question Answering, scene descriptions) to align VLMs with driving tasks. This paradigm yields diminishing returns, introduces semantic biases misaligned with actual driving behaviors, and fails to fully utilize the parameter capacity of large VLMs.
2. Discrete-Continuous Mismatch: There is a fundamental incompatibility between the discrete linguistic tokens used by VLMs and the continuous nature of driving trajectories. Tokenizing actions leads to discretization errors, sparse supervision signals, and unstable control policies that struggle with fine-grained motion planning.

The paper argues that relying on explicit text generation or action tokenization prevents VLMs from effectively learning the tacit, non-verbalizable nuances of driving dynamics.

2. Methodology: The FLARE Framework

FLARE (Future-aware LAtent REpresentation) is a novel end-to-end framework designed to activate the visual-semantic capabilities of pre-trained VLMs without requiring language supervision. It bridges the gap between high-level reasoning and low-level control through three core components:

A. Self-Supervised Future Feature Prediction

Instead of aligning with text, FLARE introduces a self-supervised objective to predict future scene dynamics directly in the latent space.

• Mechanism: The model is trained to reconstruct fine-grained DINOv2 feature patches of future frames rather than generating text or pixel-level images.
• Rationale: Predicting semantic patches forces the backbone to capture spatially explicit scene dynamics and object permanence. This provides dense, continuous gradients that leverage the VLM's capacity more effectively than sparse action tokens.
• Mental Simulation: By conditioning the predictor on the planned action intent, the model performs an "action-conditional" future simulation, anticipating how specific decisions influence the environment.

B. Hierarchical Query-Based Fusion Module

To effectively utilize the VLM's output for planning, FLARE employs a two-stage Multimodal Attention Pooling (MAP) mechanism:

1. Visual Compression: Visual tokens are compressed into a compact set of visual latents, distilling scene-level semantic content.
2. State-Conditioned Aggregation: An action MAP block condenses these visual latents into a single decision vector. Crucially, this process uses the vehicle's ego-state embedding (velocity, acceleration, historical trajectory) as the initialization query. This ensures the visual aggregation is conditioned on the vehicle's dynamic status and navigational intent.

C. Dual-Head Prediction & GRPO Refinement

The framework utilizes a dual-head architecture:

1. DiT Planner: A conditional Diffusion Transformer (DiT) generates multi-modal future trajectories, grounded by the fused action query and physical constraints.
2. Future Feature Predictor (FFP): The auxiliary head predicts the DINOv2 features of the target frame.

Training Strategy:

• Stage 1 (Supervised Fine-Tuning - SFT): The model is jointly optimized using a composite loss balancing trajectory generation (denoising score matching) and future feature prediction (magnitude regression and directional alignment).
• Stage 2 (Reinforcement Fine-Tuning - RFT): To address the limitations of imitation learning in handling rare safety violations, FLARE employs Group Relative Policy Optimization (GRPO). This refines the policy using a non-differentiable reward function (PDM-Score) based on safety, progress, and comfort, without requiring a separate value network. A Behavior Cloning (BC) regularization loss is introduced to prevent policy collapse.

3. Key Contributions

1. Future Feature Prediction Objective: A novel self-supervised task that reconstructs spatial DINOv2 patches, enabling VLMs to learn robust motion representations from unlabeled video data and mitigating the sparsity of pure action-token supervision.
2. Query-Based Fusion & GRPO Integration: A two-stage fusion mechanism that aligns visual and state modalities, coupled with GRPO to fine-tune the trajectory policy for safety-critical outcomes rather than simple imitation.
3. State-of-the-Art Performance: Achieving new SOTA results on the NAVSIM benchmark among VLM-based methods under both SFT and RFT settings using only single-camera inputs.

4. Experimental Results

The authors evaluated FLARE on the NAVSIM benchmark (v1 and v2) using the PDM Score (PDMS) and Extended PDM Score (EPDMS).

• SFT Performance: FLARE-4B achieved a PDMS of 86.9, surpassing ReCogDrive-8B (86.8) while using half the parameters (4B vs. 8B). It significantly outperformed the baseline QwenVL3-4B (+6.1 PDMS) and narrowed the gap with multi-sensor end-to-end models like DiffusionDrive (88.1) and WoTE (88.3) despite using only a single camera.
• RFT Performance: With reinforcement fine-tuning, FLARE-4B reached a PDMS of 91.4, outperforming ReCogDrive-2B (90.8) and ReCogDrive-8B (90.5). While other methods (e.g., DriveVLA-W0) reported higher scores, those relied on a "best-of-N" sampling strategy; under fair single-sample evaluation, FLARE achieved the best performance among VLM-based approaches.
• NAVSIM v2: FLARE achieved the highest EPDMS of 86.3, demonstrating superior balance in safety (NC, TTC) and comfort compared to baselines that often sacrificed one for the other.
• Ablation Studies:
  • Prediction Target: Spatial DINO prediction (Ours) yielded the highest performance (86.9), outperforming global feature prediction (85.9) and pixel-level prediction (84.7), confirming the importance of preserving spatial granularity.
  • Fusion Module: Removing the two-stage MAP fusion or ego-state injection resulted in significant performance drops (-1.9 and -2.1 PDMS respectively), validating the necessity of state-conditioned visual aggregation.

5. Significance and Claims

The paper claims that FLARE validates a highly efficient paradigm for autonomous driving: leveraging VLM world knowledge through predictive self-supervision rather than explicit language generation.

• Paradigm Shift: It demonstrates that VLMs can be effectively adapted to driving without the cost and bias of massive language annotation datasets.
• Robustness: By predicting future latent features, the model learns to anticipate scene dynamics and ego-motion directly, bridging the mismatch between discrete tokens and continuous control.
• Scalability: The approach offers a scalable pathway for utilizing large-scale unlabeled trajectory data, potentially overcoming the data coverage limitations of current modular or purely supervised end-to-end systems.

The authors conclude that FLARE successfully unlocks the visual-semantic reasoning of VLMs for continuous control, achieving state-of-the-art safety and planning quality with a single-camera setup.