Technical Summary: Distance-Annotated Traffic Perception Question Answering (DTPQA)

Problem Statement

While Vision-Language Models (VLMs) have shown remarkable progress in various tasks, their application to automated driving remains limited by a lack of robust perception capabilities. In safety-critical traffic scenarios, models must understand complex scenes with simultaneous events, often involving agents at significant distances (30+ meters). Existing benchmarks for automated driving VQA often fail to isolate perception from reasoning or advanced world knowledge. Furthermore, many existing datasets lack specific distance annotations, making it difficult to evaluate how model performance degrades as object distance increases. There is a critical need for a benchmark that:

1. Focuses exclusively on trivial, perception-only visual questions that do not require complex reasoning.
2. Ensures questions are crucial for driving decisions (e.g., pedestrian direction, signal status) rather than generic queries.
3. Utilizes a multiple-choice format to eliminate biases introduced by language generation metrics.
4. Includes distance annotations to analyze perception as a function of range.

Methodology

The authors propose DTPQA, a Visual Question Answering (VQA) benchmark comprising 19,149 samples split into two distinct parts: DTP-Synthetic and DTP-Real.

1. DTP-Synthetic (Simulator-Based)

• Source: Generated using the CARLA simulator (v0.9.15) across multiple maps (Town01–07, Town10HD, Town12, Town15).
• Procedure: The pipeline involves loading a map, applying random weather/daytime conditions, spawning an ego-vehicle with an RGB camera, and placing target agents at specific distances (5m to 50m). Images are captured, and agents are respawned at new distances to cover the full range.
• Categories: Six categories were defined to test simple visual concepts relevant to driving:
  1. Pedestrian presence: Is a pedestrian crossing the road?
  2. Pedestrian direction: Is the pedestrian moving left or right?
  3. Multiple pedestrians: Counting pedestrians (1, 2, 3, or 4).
  4. Vehicle indicators: Which indicator (left/right/none) is active on a truck ahead?
  5. Traffic lights: What is the color of the traffic light?
  6. Traffic signs: What is the displayed sign?
• Data Balancing: The dataset is strictly balanced. For each distance and category, the number of samples is divisible by the number of possible answers to prevent models from exploiting language biases.
• Validation: All synthetic images underwent manual review to remove samples with incorrect spawn locations, occlusions (e.g., due to road geometry), or simulator rendering glitches.

2. DTP-Real (Real-World)

• Source: Built upon the nuScenes v1.0 trainval split.
• Procedure: Existing images were filtered using nuScenes annotations to identify scenes matching specific criteria (e.g., exactly one visible pedestrian, specific velocity for direction).
• Distance Handling: Unlike the synthetic set, objects could not be placed at exact distances. Instead, samples were assigned to distance bins (5, 10, 20, 30, 40, 50 meters) based on the nearest bin, while retaining the precise distance in annotations. Samples beyond 55 meters were discarded.
• Categories: Four categories were adapted for real-world data:
  1. Person presence: Is there a person in the scene?
  2. Person direction: Is the walking person moving left or right? (Filtered for velocity > 0.5 m/s).
  3. Multiple people: Counting people (2 or 3) in the scene.
  4. Person location: Is the person on a road, sidewalk, or crosswalk?
• Data Balancing: Similar to the synthetic set, the final dataset was balanced by randomly removing samples from over-represented answer categories to ensure equal distribution.

Dataset Structure

The dataset includes:

• Images: .jpg files (Synthetic images are included; Real images must be downloaded from nuScenes).
• Annotations: A single .json file containing metadata for all samples, including image paths, questions, ground truth answers, distance bins, precise distances, and environment settings (e.g., weather, town).

Key Contributions

1. DTPQA Benchmark: The first VQA dataset specifically designed to evaluate VLM perception in traffic scenarios, isolated from reasoning skills, with a focus on distance-dependent performance.
2. Distance Annotations: Explicit inclusion of object distance (both binned and precise) allows for granular analysis of perception degradation over range.
3. Strict Data Balancing: The methodology ensures that answer distributions are uniform across distances, preventing models from achieving high scores through statistical bias rather than visual understanding.
4. Dual-Source Approach: Combines a controlled, fully annotated synthetic environment (CARLA) with a large-scale real-world dataset (nuScenes) to provide a comprehensive evaluation suite.
5. Open Source: The authors provide the dataset, Python scripts for generation, and detailed documentation on GitHub.

Results and Evaluation

The paper reports on the evaluation of State-of-the-Art (SOTA) small VLMs using DTPQA (referenced in [11]).

• Performance Gap: There is a substantial gap between human performance and current VLMs. SOTA models perform significantly worse than humans, particularly on tasks requiring spatial perception (e.g., determining pedestrian direction or blinker status).
• Distance Sensitivity: The benchmark successfully demonstrates that model performance degrades as the distance to the object increases, validating the necessity of distance-annotated evaluation.
• Chance-Corrected Accuracy: Figure 5 in the paper illustrates that even the best-performing models struggle with these "trivial" perception tasks compared to human baselines.

Significance

The authors position DTPQA as a critical tool for the automated driving community. Its significance lies in its ability to:

• Isolate Perception: By using trivial questions that do not require reasoning, DTPQA allows researchers to diagnose specific weaknesses in a model's visual perception capabilities without the confounding variable of language generation or world knowledge.
• Address Safety-Critical Distances: It highlights that current models struggle with long-range perception (30+ meters), a critical requirement for safe autonomous driving.
• Standardize Evaluation: By providing a balanced, multiple-choice benchmark with distance annotations, it offers a standardized metric for comparing different VLMs in traffic scenarios.
• Future Utility: While primarily a benchmark, the authors note the dataset can be used for fine-tuning VLMs or studying the effects of prompt phrasing and environmental variables (e.g., weather in DTP-Synthetic).

The paper concludes that DTPQA effectively challenges current models, serving as a valuable benchmark to drive improvements in the perception capabilities of VLMs for automated driving.