Operating Within the Operational Design Domain: Zero-Shot Perception with Vision-Language Models

Technical Summary: Operating Within the Operational Design Domain with Zero-Shot Vision-Language Models

Problem Statement
The widespread adoption of Autonomous Driving Systems (ADS) is constrained not merely by capability, but by the need for verifiable safety and compliance with stringent regulatory standards. A central concept in these regulations is the Operational Design Domain (ODD), which defines the specific environmental and operational conditions under which an ADS is intended to function safely. Ensuring ODD compliance requires perception systems capable of detecting a vast array of static and transient environmental elements (e.g., road markings, weather conditions, signage, and surface states). Traditional approaches relying on manually developed, task-specific perception models are inefficient for the diverse and evolving nature of ODD taxonomies. There is a critical need for a flexible, adaptable perception system that can recognize complex ODD elements without extensive retraining, particularly for rare or evolving conditions that are impractical to pre-map exhaustively.

Methodology
The authors investigate the potential of Vision-Language Models (VLMs) to function as "zero-shot ODD sensors." The study employs a multi-phase experimental design:

1. Dataset Construction: The authors curated a custom dataset, ODD-TAX-232, comprising 2,320 images representing 232 distinct ODD concepts. These concepts are categorized into five groups: Traffic Signage, Road Markings, Scenery Elements, Weather Conditions, and Trigger Conditions. The dataset focuses on fine-grained environmental features relevant to European, UK, and US infrastructure, excluding moving objects to isolate static perception capabilities.
2. Zero-Shot Optimization: Before comparing models, the authors conducted an ablation study on the GPT-4o model to identify optimal prompting strategies. They tested various approaches, including:
   • Baseline: Flat taxonomy lists and label aliasing.
   • Process-Oriented: Self-refinement, two-stage pipelines (e.g., road-dependent classification), and Persona Decomposition (assigning specialized roles like "Weather Expert" or "Sign Expert").
   • Reasoning Enhancements: Chain-of-Thought (CoT) prompting to enforce intermediate reasoning steps.
   • Retrieval-Augmented Generation (RAG): Integrating external knowledge bases to reduce token usage.
   • Chaining: Multi-stage pipelines where outputs from one stage inform the next.
3. Model Evaluation: Using the best-performing prompt strategy (Persona + CoT), the authors evaluated four VLMs:
   • Closed-source General: GPT-4o and Gemini 2.5 Pro.
   • Open-source General: Llama 4 Maverick.
   • Open-source Specialized: Molmo-72B.
4. Tasks: The evaluation covered two tasks:
   • Classification: Determining if an image contains a specific ground-truth ODD element (measured by Recall).
   • Detection: Localizing the element within the image using bounding box coordinates (measured by Precision, F1-score, and spatial accuracy via Average L2 Error). This was tested on both the custom ODD-TAX-232 and the Mapillary Vistas dataset (which provided bounding box annotations).

Key Contributions
The paper contributes three primary artifacts to the field:

1. Empirical Study: A comprehensive analysis of zero-shot ODD classification and detection across four VLMs, including a detailed failure analysis of common error patterns.
2. Optimization Analysis: An ablation of zero-shot optimization strategies, providing a cost-performance overview (token budget vs. recall) and identifying that definition-anchored Chain-of-Thought prompting with persona decomposition yields the best results.
3. Open Resources: A publicly available suite of reusable prompting templates, adaptation guidelines, and the ODD-TAX-232 dataset to facilitate future research and extension.

Results

• Prompting Strategy: The Persona + CoT strategy achieved the highest recall (0.73) on ODD-TAX-232, significantly outperforming the baseline (0.40) and other strategies like RAG or cascaded pipelines (which often reduced performance due to error propagation). However, this high performance came with a substantial token cost (~25,000 tokens per sample).
• Model Performance (Classification): Closed-source models (GPT-4o and Gemini 2.5 Pro) demonstrated superior performance (Recall ~0.72–0.73) compared to open-source alternatives (Llama 4 Maverick at 0.67, Molmo-72B at 0.45). While GPT-4o and Gemini 2.5 Pro were comparable overall, GPT-4o showed an edge in signage and weather recognition, while Gemini excelled in markings and scenery.
• Model Performance (Detection): On the Mapillary Vistas dataset, GPT-4o achieved the best balance of precision (0.82) and recall (0.922), resulting in the highest F1-score (0.87). Gemini 2.5 Pro had the highest recall (0.977) but lower precision. Llama 4 Maverick demonstrated the highest spatial accuracy (lowest L2 error) but lagged in overall F1-score. Molmo-72B performed poorly in precision and F1-score.
• Dataset Discrepancy: All models performed significantly better on Mapillary Vistas than on ODD-TAX-232. The authors attribute this to the coarser, more common categories in Mapillary versus the fine-grained, rare, and safety-critical concepts in ODD-TAX-232, which are less frequent in pre-training data.

Significance and Claims
The authors conclude that while VLMs show promising potential for ODD recognition, a zero-shot recall of approximately 0.73 is insufficient for standalone, on-board production deployment in safety-critical applications where undetected ODD violations could lead to unsafe system activation.

Instead, the paper positions VLMs as highly viable tools for offline applications, such as:

• Pre-route ODD auditing.
• Regulatory compliance reporting.
• Simulation seeding.

In these contexts, the cost of missed detections is lower, and human oversight can compensate for residual errors. The study emphasizes that the design of prompts (specifically CoT with persona decomposition) is as critical as model selection. Furthermore, the authors argue that aggregate metrics on general datasets like Mapillary Vistas can overestimate readiness for ODD perception; taxonomy-aware evaluations focusing on fine-grained, safety-critical elements are essential for accurate assessment. The work paves the way for transparent, effective, and adaptable ODD-based perception systems, provided they are used within appropriate safety boundaries.