Technical Summary: CLAP – Contrastive Latent-space Prompt Optimization for End-to-End Autonomous Driving

1. Problem Statement

End-to-end autonomous driving (AD) systems powered by Vision-Language-Action (VLA) models have demonstrated strong performance on common driving scenarios by leveraging large-scale pre-training and fine-tuning. However, these systems remain brittle in long-tail scenarios—rare but safety-critical situations such as active construction zones, complex yielding geometries, and adverse weather combinations.

The dominant approach to addressing these failures involves scaling data collection and fine-tuning models. The paper argues this approach faces diminishing returns due to:

• The sparsity of rare events.
• The combinatorial complexity of within-category variations (layouts, weather, time of day).
• The risk of catastrophic forgetting when frequently fine-tuning on niche data.

Furthermore, the paper identifies a specific limitation in current adaptation strategies: long-tail failures are often tied to specific road regions (roadblocks) rather than being purely category-level generalization issues. Additionally, within a single roadblock, challenging (long-tail) frames and normal frames are heavily intermixed in the model's latent space. Consequently, optimizing a prompt to fix challenging frames often inadvertently degrades performance on normal frames due to this latent intermixing.

2. Methodology: CLAP Framework

The authors propose CLAP (Contrastive Latent-space Prompt Optimization), a framework that augments a frozen VLA driving model with per-roadblock soft prompts. These prompts are optimized offline on crowdsourced data and retrieved on-demand via Vehicle-to-Everything (V2X) communication.

Core Observations

The methodology is grounded in two observations derived from analyzing the latent space of state-of-the-art VLAs:

1. Locality: Scenarios from the same roadblock (a contiguous spatial region with stable visual/geometric identity) cluster tightly in the latent space, distinct from other roadblocks.
2. Intermixing: Within a single roadblock, challenging and normal frames are not separable into distinct clusters; they are heavily intermixed.

The Two-Stage Pipeline

CLAP operates via a two-stage prompt optimization process for each roadblock $R$:

Stage 1: Direction Discovery via Supervised Contrastive Learning

• Goal: Discover a "hard-scene direction" ($d^\star$) that separates challenging frames from normal frames in the latent space.
• Mechanism: An auxiliary soft prompt $P_A$ is optimized using a Supervised Contrastive (SupCon) loss. This loss pushes the hidden states of challenging frames away from normal frames.
• Output: The principal direction of the separated representation is extracted as $d^\star$, a compact descriptor of the challenging-scene direction specific to that roadblock.

Stage 2: Directionally Regularized Prompt Optimization

• Goal: Optimize a deployable performance prompt $P_B$ that improves challenging frames while strictly preserving normal frame performance.
• Mechanism: $P_B$ is trained under a joint loss function:
  $$L_{total} = L_{plan} + \lambda_1 L_{dir} + \lambda_2 L_{reg}$$
  • $L_{plan}$: Standard trajectory regression loss (ADE) computed only on challenging frames.
  • $L_{dir}$ (Directional Regularizer): Encourages $P_B$ to displace challenging frame representations along the pre-discovered axis $d^\star$. This constrains the optimization to the specific subspace where separation is possible.
  • $L_{reg}$ (Normal-Frame Regularizer): Explicitly penalizes any displacement of normal frame representations relative to the Stage 1 baseline. This prevents the prompt from "leaking" into the subspace that affects common driving.

Data Partitioning

To apply this, the system must partition roadblock traces into "challenging" and "normal" subsets. The paper proposes a VLM-based pipeline that uses semantic identification (via a strong VLM) and empirical validation (planning error exceeding a margin $\delta$) to label frames.

3. Key Contributions

1. Per-Roadblock Formulation: The paper demonstrates via latent-space analysis that long-tail problems can be addressed locally per roadblock rather than globally, as roadblocks form distinct manifolds in the latent space.
2. CLAP Framework: A novel two-stage prompt optimization method that uses contrastive learning to identify a separation direction and directional regularization to selectively improve long-tail performance without regressing normal performance.
3. Empirical Validation: Demonstration that CLAP reduces planning errors on challenging scenarios across multiple SOTA VLA backbones without requiring backbone fine-tuning.

4. Experimental Results

The authors evaluated CLAP on the NAVSIM benchmark using three state-of-the-art VLA backbones: ReCogDrive, Alpamayo-R1.5, and DriveVLA-W0.

• Performance on Challenging Frames: CLAP reduced the Average Displacement Error (ADE) on challenging scenarios by an average of 24% across the three models.
• Performance on Normal Frames: Unlike unconstrained soft prompt optimization (which caused regression in some models) or explicit text notices (which often failed or degraded performance), CLAP maintained or slightly improved performance on normal frames.
• Overall Improvement: The combination of reduced long-tail errors and preserved normal performance led to an average 14% improvement in overall ADE.
• Ablation Studies:
  • Removing Stage 1 (contrastive direction discovery) resulted in significant performance drops, confirming that the specific direction $d^\star$ is critical.
  • Replacing $d^\star$ with a random direction performed worse than omitting Stage 1 entirely, highlighting the necessity of the learned contrastive direction.
• Generalization: When tested under shifted conditions (rain and dusk) without augmentation, performance degraded. However, using generative image editing to augment training data with these conditions allowed CLAP to generalize effectively, reducing the ADE gap under shifted conditions significantly.

5. Significance and Claims

The paper positions CLAP as a practically efficient alternative to data scaling and model fine-tuning for improving VLA performance in autonomous driving.

• Modality of Improvement: The authors claim that long-tail failures can be "patched locally" via soft prompts rather than requiring global model retraining.
• Selectivity: The core significance lies in the ability to selectively improve difficult scenarios while structurally preventing collateral damage to common driving performance, a challenge that unconstrained prompt optimization fails to solve due to latent intermixing.
• Deployment Viability: The framework is designed for V2X deployment, where a central server optimizes prompts based on crowdsourced data and pushes them to vehicles entering specific roadblocks, keeping the vehicle-side inference cost minimal (frozen model + prompt injection).

The authors acknowledge limitations, noting that cross-weather generalization beyond tested conditions remains an open question and that the current contrastive objective assumes a single dominant challenge type per roadblock, which may need refinement for heterogeneous scenarios.