UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics

This paper introduces UDVideoQA, a large-scale, privacy-preserving benchmark dataset derived from real-world urban traffic footage that features 28K question-answer pairs for evaluating multi-object spatio-temporal reasoning, revealing a significant perception-reasoning gap in current models while demonstrating that fine-tuning smaller open-source models can achieve performance comparable to proprietary systems.

The Gist

Imagine you are trying to teach a robot how to drive a car through a busy city. You don't just want the robot to know what a stop sign looks like; you want it to understand why a car stopped, what the pedestrian was doing three seconds ago, and what would have happened if the light had turned green instead of red.

This paper introduces a new, massive "training gym" for these robot brains, called UDVideoQA.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: The "Textbook" vs. The "Real World"

Most AI models today are like students who only study from perfect, clean textbooks. They are trained on short, curated clips where everything is clear, the lighting is perfect, and the actors know their lines.

• The Reality: Real city traffic is messy. It's raining, people are jaywalking, cars are honking, and the lighting changes from bright noon to dark night.
• The Gap: When you put these "textbook-trained" robots on a real street, they get confused. They might hallucinate (imagine things that aren't there) or miss simple details because they've never seen a messy intersection before.

2. The Solution: The "Urban Dynamics" Gym

The researchers built UDVideoQA, a giant dataset of 16 hours of real, unscripted traffic video from city intersections.

• The Volume: It's like watching 1.7 million frames of video. That's a lot of traffic!
• The Privacy Shield: Since they filmed real people, they had to protect their identities. Instead of just blurring faces (which can look weird and block the view), they used a special "motion-blur" technique.
  • Analogy: Imagine a painter who only paints over the parts of the canvas that are moving (like a walking person or a driving car) but leaves the static background (like the road, trees, and signs) perfectly clear. This keeps the scene looking real while protecting privacy.

3. The Test: 5 Levels of "Brain Power"

They didn't just ask simple questions like "Is there a car?" They created a hierarchy of questions to test how smart the AI really is. Think of it like a video game with five levels:

1. Level 1: The Observer (Attribution): "What color is that car?" or "Is it raining?" (Basic facts).
2. Level 2: The Narrator (Basic Understanding): "What is happening in this scene overall?" (Summarizing the vibe).
3. Level 3: The Detective (Event Reasoning): "Why did the silver car brake?" (Connecting cause and effect).
4. Level 4: The Time Traveler (Reverse Reasoning): "The pedestrian is halfway across the street. What was the traffic light doing before they stepped off the curb?" (Working backward).
5. Level 5: The Philosopher (Counterfactual Inference): "If the light had been green, would the motorcycle have crashed?" (Testing "What if" scenarios without hallucinating).

4. The Results: The "Smart but Blind" Paradox

The researchers tested 10 of the world's most advanced AI models (like Gemini, GPT-4, and Qwen) on this new gym. The results were surprising:

• The "Big Brain" Trap: The biggest, most expensive models (like Gemini Pro) were great at the "Philosopher" level. They could guess what might happen in a hypothetical scenario. However, they were terrible at the "Observer" level. They often couldn't tell if a car was silver or grey, or if the road was wet.
  • Analogy: It's like a brilliant professor who can write a thesis on traffic theory but fails to notice that the person standing right in front of them is wearing a red hat. They are smart, but they aren't "seeing" the video.
• The "Small but Trained" Hero: A smaller, open-source model (Qwen 2.5-VL) started out average. But when the researchers gave it a "crash course" (fine-tuning) specifically on this messy traffic data, it became a superstar. It learned to see the details and reason about them, eventually beating the giant models in low-light conditions.

5. The New Challenge: "Ask the Right Question"

The paper also introduced a side challenge called VideoQGen. Instead of answering questions, the AI has to create them.

• The Result: The best models could ask deep, complex questions. But many others just asked boring, repetitive questions like "Is it raining?" over and over. It showed that while AI is getting better at talking, it still struggles to be truly creative and diverse.

The Big Takeaway

This paper tells us that to make AI truly safe and useful for real-world tasks (like self-driving cars or city monitoring), we can't just feed them more data. We need to teach them to look closely (grounding) before they try to think deeply (reasoning).

UDVideoQA is the new standard tool to help developers fix this "blindness" and build AI that doesn't just guess, but actually sees and understands the chaotic, beautiful mess of our cities.

Technical Summary

1. Problem Statement

Understanding complex, multi-agent dynamics in urban traffic remains a significant challenge for Video-Language Models (VideoLMs). Existing benchmarks suffer from several critical limitations:

• Data Scarcity & Quality: Most datasets rely on curated, short clips from autonomous driving platforms or internet sources, lacking the temporal continuity, density, and realism of real-world surveillance footage.
• Reasoning Gaps: Current benchmarks often fail to test robustness against hallucinations (e.g., asking about objects not present) and lack a hierarchy of reasoning skills, moving from basic perception to complex causal and counterfactual inference.
• Privacy vs. Fidelity: Traditional anonymization methods (detector-based blurring) often compromise scene context or fail under low-light/occlusion conditions, hindering the creation of ethical, large-scale public datasets.
• Evaluation Bias: Standard evaluation relies on lexical similarity (BLEU/ROUGE), which penalizes semantically correct but lexically different answers, and lacks metrics for evaluating the generation of diverse, grounded questions.

2. Methodology

A. Dataset Construction (UDVideoQA)

• Data Collection: The dataset comprises 16 hours of raw traffic footage (approx. 1.7 million frames) recorded at 30 fps from multiple metropolitan intersections. It covers diverse conditions: morning/evening rush hours, midday, nighttime, and varying weather (clear, rain, dust storms).
• Privacy-Preserving Anonymization: The authors introduced an event-driven dynamic blurring technique. Unlike semantic segmentation (YOLO+SAM) which can fail on occlusion or low light, this method uses motion-based masks derived from pixel-intensity changes (inspired by event cameras). It effectively anonymizes moving agents (pedestrians, vehicles) while preserving static background details (road markings, signs) essential for reasoning.
• Annotation Pipeline:
  • Segmentation: Videos are segmented into 10-second clips to balance temporal context with modeling complexity.
  • QA Generation: A semi-automated pipeline generates 28,800 QA pairs across 8 hours of densely annotated video (1 QA per second).
  • Taxonomy: Questions are categorized into a hierarchical taxonomy of five reasoning levels:
    1. Attribution (Atr): Basic perception (color, count, text).
    2. Basic Understanding (BU): Global scene context (weather, lighting).
    3. Event Reasoning (ER): Cause-and-effect over time.
    4. Reverse Reasoning (RR): Inferring prior states from observed events.
    5. Counterfactual Inference (CI): Testing hallucination robustness via hypothetical scenarios.
  • Validation: A rigorous human-in-the-loop process ensures clarity, relevance, and answerability, with a focus on removing hallucinations.

B. Benchmarking Framework

• VideoQA Benchmark: Evaluates 10 State-of-the-Art (SOTA) VideoLMs (including Gemini 2.5 Pro, GPT-4o/5, Qwen2.5-VL, InternVL3) in both zero-shot and fine-tuned settings.
• VideoQGen Benchmark: A novel benchmark to evaluate a model's ability to generate diverse, grounded questions. It assesses Relevance, Answerability, and Diversity across the five reasoning categories.
• Evaluation Metrics:
  • Semantic-Semantic Scoring: Uses an LLM Judge (Gemini 2.5 Pro) to evaluate semantic equivalence rather than lexical overlap.
  • Weighted Complexity Scoring: Assigns higher weights to complex reasoning tasks (CI: 1.5x, ER/RR: 1.3x) to prioritize higher-order cognitive skills.

3. Key Contributions

1. UDVideoQA Dataset: A large-scale, open-source benchmark with 1.7M frames and 28.8K QA pairs, featuring high-density multi-agent interactions and diverse environmental conditions.
2. Ethical Anonymization Framework: A novel event-driven dynamic blurring method that preserves scene fidelity while ensuring privacy, outperforming traditional detector-based redaction in low-light and cluttered scenarios.
3. Comprehensive Taxonomy: A hierarchical reasoning structure spanning from basic attribution to counterfactual inference, enabling systematic evaluation of visual grounding and causal reasoning.
4. VideoQGen Benchmark: The first benchmark in the traffic domain to evaluate the linguistic diversity and contextual grounding of generated questions, not just answers.
5. Open Resources: Release of the dataset, annotation tools, and benchmarks for both VideoQA and VideoQGen to foster reproducible research.

4. Experimental Results

VideoQA Performance

• Perception-Reasoning Gap: A critical finding is that models excelling in abstract reasoning (Counterfactual Inference) often fail at basic visual grounding (Attribution). For instance, Gemini 2.5 Pro achieved high CI scores (91%) but very low Attribution scores (22%) in morning conditions, indicating it hallucinates details while reasoning correctly about hypotheticals.
• Fine-Tuning Impact: Fine-tuning smaller, open-source models significantly bridges the gap. Qwen2.5-VL 7B, when fine-tuned on UDVideoQA, achieved performance comparable to proprietary systems (e.g., 61.38% in evening conditions vs. GPT-5's 54.01%), demonstrating that domain-specific adaptation is more effective than sheer parameter scale for low-level visual tasks.
• Generalization: The fine-tuned Qwen2.5-VL 7B showed improved generalization on external datasets (RoadSocial, SUTDTrafficQA), gaining ~7.3% and ~4.3% respectively, proving the dataset's utility for cross-domain transfer.

VideoQGen Performance

• Top Performers: Gemini 2.5 Pro and Qwen3 Max generated the most relevant and answerable questions.
• Diversity Bottleneck: Despite high relevance, all models exhibited low linguistic diversity (Gemini 2.5 Pro scored ~60% on diversity), often resorting to repetitive phrasing. GPT-4o performed poorly, generating generic questions with low grounding.

Failure Modes

• Hallucination: Models frequently hallucinate agents (e.g., predicting a pedestrian collision when none exists) to satisfy causal reasoning scripts.
• Temporal Localization: Models struggle to verify premises within specific time windows (e.g., failing to detect that "no car entered the foreground" in the last 3 seconds).
• Fine-Grained Perception: Significant errors in identifying small visual primitives like road markings (e.g., confusing an 'X' marking with a bicycle symbol).

5. Significance

UDVideoQA addresses a fundamental gap in multimodal AI research by providing a realistic, privacy-compliant, and ethically curated benchmark for urban traffic. Its primary significance lies in:

• Exposing the "Perception-Reasoning Gap": It demonstrates that current SOTA models are often "reasoning engines" that lack "seeing" capabilities, relying on language priors rather than visual evidence.
• Advancing Ethical AI: The event-driven blurring technique sets a new standard for creating public datasets from sensitive surveillance footage without compromising research utility.
• Guiding Future Architecture: The results suggest that future VideoLMs must prioritize fine-grained visual grounding and temporal precision over pure parameter scaling to achieve robust real-world performance.
• Community Resource: By releasing the full suite (dataset, tools, benchmarks), it enables the community to develop the next generation of perception-reasoning aligned models for intelligent transportation and surveillance.