RoadscapesQA: A Multitask, Multimodal Dataset for Visual Question Answering on Indian Roads

1. Problem Statement

Autonomous driving systems rely heavily on high-fidelity data to interpret visual surroundings and make safe decisions. However, existing multimodal driving datasets (e.g., KITTI, BDD100K, NuScenes) predominantly focus on structured environments in the US, Europe, and East Asia. These datasets fail to capture the unique complexities of South Asian driving conditions, specifically in India, which are characterized by:

• Unstructured environments: Unmarked roads, mixed traffic, and heterogeneous vehicle types.
• Unpredictable agent behavior: High density of pedestrians, cyclists, animals, and erratic driving patterns.
• Environmental variability: Frequent nighttime driving, glare, and resource-constrained sensor setups.
• Data Scarcity: A lack of large-scale Visual Question Answering (VQA) datasets tailored to Indian road scenarios, particularly for rural and nighttime settings.

Creating such datasets traditionally requires expensive sensor suites and massive human annotation efforts, hindering scalability and reproducibility.

2. Methodology

The authors propose Roadscapes, a novel dataset and a scalable generation framework designed to bridge these gaps.

A. Data Collection

• Source: 5 hours of driving footage captured in Coimbatore and Kochi (Southern India) and the connecting national highway.
• Hardware: A low-cost, monocular front-facing camera mounted on a vehicle dashboard (1920×1080 resolution, 30 FPS).
• Scope: 35 video sequences covering diverse scenarios: highways, service roads, crowded city streets, and village paths.
• Conditions: Captured during daytime, dusk, and nighttime, including natural sensor artifacts like motion blur, glare, and shadows.
• Preprocessing: Raw data yielded ~17,000 frames; manual filtering removed distorted frames, resulting in 8,983 usable images.
• Privacy: A semi-automated pipeline using YOLOv5 detected and anonymized license plates, followed by manual spot-checks. The dataset is released under a non-commercial license.

B. Annotation and Label Generation

To minimize human labor costs while maintaining quality, the authors employed a hybrid human-AI workflow:

1. Pre-annotation: The YOLOWorld (zero-shot) model generated initial bounding boxes for common classes (cars, trucks, buses, etc.).
2. Human Verification: A team of four annotators manually verified and corrected these bounding boxes.
3. Task Scope: The dataset includes annotations for Object Detection and Drivable Area Segmentation.

C. Visual Question Answering (VQA) Generation

Instead of manual question writing, the authors developed a rule-based heuristic framework to generate QA pairs automatically from the object detection annotations.

• Categories: The dataset covers three main VQA categories:
  1. Object Counting: (e.g., "How many trucks are present?", "Are there more cars than trucks?")
  2. Object Description: (e.g., "What is the color of the truck?", "What class is the object at [x,y]?")
  3. Surrounding Description: (e.g., "What time of day is it?", "What is the traffic density?")
• Scale: Approximately 7 questions per image, totaling roughly 63,000 QA pairs across the dataset.

3. Key Contributions

1. Roadscapes Dataset: A diverse collection of ~9,000 monocular images from Southern India, filling a critical gap in geographic and environmental diversity (rural, urban, day/night) for autonomous driving research.
2. Scalable Generation Framework: A novel pipeline combining foundation models (YOLOWorld), human verification, and rule-based heuristics to generate large-scale VQA data without prohibitive annotation costs.
3. Benchmarking Baselines: The first evaluation of state-of-the-art Vision-Language Models (VLMs) on Indian road scenarios, providing initial performance metrics and hallucination analysis.
4. Privacy-Aware Release: A dataset curated with explicit anonymization protocols, available for non-commercial research.

4. Experimental Results

The authors evaluated four VLMs in a zero-shot setting (no fine-tuning) on the Roadscapes dataset:

• Models Tested: Phi-3.5, GPT-4o, Paligemma, and GPT-4o-mini.
• Metrics:
  • Object Counting: Exact-match accuracy.
  • Object/Surrounding Description: Cosine similarity of sentence embeddings (using all-MiniLM-L6-v2).

Performance Highlights:

• Object Counting: Phi-3.5 achieved the highest accuracy (66.7%), followed by 4o-mini (62.8%). Common failures included undercounting in complex scenes and hallucinating non-existent objects.
• Object Description: Paligemma performed best (0.501 similarity). Models frequently hallucinated incorrect colors or misclassified object classes, indicating difficulty with fine-grained attribute recognition.
• Surrounding Description: GPT-4o excelled (0.701 similarity), showing strong capability in temporal and contextual reasoning (e.g., identifying time of day).
• Hallucination Analysis:
  • Object Description tasks had the highest hallucination rates (50.8% – 61.6%), particularly for 4o-mini (61.6%).
  • Surrounding Description tasks had the lowest rates (e.g., Phi-3.5 at 6.2%), suggesting models handle high-level context better than fine-grained visual details.

5. Significance and Future Work

• Significance: RoadscapesQA demonstrates that robust VQA models for unstructured, developing-world environments are feasible using low-cost data and scalable annotation pipelines. It highlights that while current VLMs can handle high-level scene understanding, they struggle significantly with fine-grained perception (colors, specific counts) in chaotic Indian traffic.
• Limitations: The dataset currently lacks explicit spatial relationship benchmarks (e.g., "left/right" positioning) and is geographically limited to Southern India.
• Future Directions: The authors suggest expanding geographic coverage, incorporating multi-turn dialogues, temporal reasoning, and leveraging advanced embedding models to improve spatial understanding.

In conclusion, RoadscapesQA serves as a critical resource for advancing autonomous driving research in underrepresented regions, pushing the boundaries of how VLMs handle the "messiness" of real-world, unstructured driving environments.