🤖 AI

Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City

This paper introduces the Robusto-2 benchmark to evaluate and compare the performance of human drivers from Lima and New York City against Vision-Language Models (VLMs) on diverse driving scenarios across these two challenging geographies, revealing that while human and VLM responses diverge based on question type, neither group shows significant performance variation attributable to the specific city due to the high out-of-distribution nature of the scenarios.

Original authors: Adrian Cespedes, Marcelo Chincha, Dunant Cusipuma, Victor Flores-Benites, David Ortega, Arturo Deza

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adrian Cespedes, Marcelo Chincha, Dunant Cusipuma, Victor Flores-Benites, David Ortega, Arturo Deza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Do Self-Driving Cars Have a "Culture Shock"?

Imagine you are teaching a robot to drive. You train it in a quiet, orderly town. Then, you drop it into a chaotic, bustling city where traffic rules are more like suggestions than laws. Will the robot panic? Will it understand what's happening?

The authors of this paper wanted to test exactly that. They asked: If we show self-driving car "brains" (called VLMs) videos from two very different, chaotic cities—Lima, Peru, and New York City, USA—will they react differently than human drivers? And will humans from Lima react differently than humans from New York?

The Experiment: A Driving Test for Humans and Robots

Think of this study as a massive, multi-lingual driving test.

The Test Subjects:
- 10 Humans from Lima: Real drivers who know the streets of Peru.
- 10 Humans from New York: Real drivers who know the streets of NYC.
- 10 AI Models: Different "brains" from various tech companies (the VLMs).
- Total: 30 "drivers" taking the test.
The Test Material:
- They didn't use perfect, sunny-day footage. They used dashcam videos from Lima and NYC.
- They specifically picked the "messiest" 5-second clips—scenes where the AI was most confused or disagreed with itself. These are the "edge cases," like a taxi cutting someone off in the rain or a pedestrian jaywalking.
The Test Questions:
Instead of just watching, everyone (humans and AI) had to answer questions about the video, like a quiz. The questions fell into four categories:
- Factual: "What is the car doing?" (Simple observation).
- Ratings: "On a scale of 1 to 10, how messy is this traffic?" (Subjective judgment).
- Counterfactual: "What would have to happen for a crash to occur?" (Imagination).
- Reasoning: "Who has the right of way?" (Logic and rules).

The Surprising Results

The researchers expected the AI to get confused by the "foreign" city (e.g., an AI trained on US data might fail in Lima). They also expected Lima drivers to see things differently than NYC drivers. Here is what they actually found:

1. The "Geography" Myth

The Expectation: "Lima drivers will see Lima differently than NYC drivers see Lima."
The Reality: Nope. Humans from Lima and humans from New York gave almost identical answers, regardless of which city's video they were watching.
The Analogy: Imagine showing a picture of a storm to a fisherman from the Amazon and a fisherman from the Arctic. Even though they live in different places, they both agree: "That's a storm." The chaos of driving in Lima or NYC is so universal that human brains process it the same way.

2. The Human vs. Robot Gap

The Expectation: Maybe the AI is just as good as humans.
The Reality: Nope. Humans and AI often gave very different answers.
The Analogy: If you ask a human and a robot, "Is that car about to crash?", the human might say, "Yes, the driver looks distracted," while the robot might say, "No, the distance is safe." They are looking at the same scene but "thinking" in completely different languages.

3. The "Question" Matters More Than the "Place"

The biggest difference in answers didn't come from where the video was filmed (Lima vs. NYC). It came from what kind of question was asked.
When asked simple facts, everyone agreed. When asked to imagine "what if" scenarios, the AI and humans diverged wildly.

The "Chaotic" Conclusion

The paper concludes with a twist. Usually, we think of "Out-of-Distribution" (OOD) data as something weird and unique to one place. But the authors found that chaos is universal.

Whether it's a chaotic street in Lima or a chaotic street in NYC, the "messiness" looks the same to both humans and AI. The gap isn't between the cities; the gap is between biological brains (humans) and digital brains (AI).

Why This Matters (According to the Paper)

The authors suggest that because these cities are so chaotic and currently no self-driving cars operate there, they are actually perfect "stress tests."

The Analogy: If you want to test if a new car engine is strong, you don't drive it on a smooth highway. You drive it over rocks. Lima and NYC are the "rocks" of the driving world.
By testing AI on this messy data, we can see exactly where the AI's "cognitive backbone" breaks down compared to a human driver.

In short: Humans from different cultures see driving chaos the same way. AI, however, sees it differently than humans, regardless of where the AI was trained. The paper provides a new dataset to help researchers fix this gap.

Technical Summary: Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City

Problem Statement

As self-driving car (SDC) technology evolves from hard-coded heuristics to modular systems powered by Vision-Language-Action (VLA) models with Vision-Language Models (VLMs) as cognitive backbones, a critical open question remains: how well do these systems generalize to new geographies and out-of-distribution (OOD) edge-case scenarios?

Current multi-modal foundation models are predominantly trained on Western Hemisphere data, specifically from the United States. This raises safety concerns regarding their deployment in complex, chaotic environments where training data is scarce or non-existent. While cities like San Francisco and Shenzhen have active SDC deployments, locations such as Lima, Peru, and New York City, USA, remain challenging environments where no SDC fleets currently operate. The paper investigates the cross-cultural generalization gap between human drivers and VLMs when exposed to driving scenarios from these distinct geographies.

Methodology

The authors present Robusto-2, a benchmarking study employing a full factorial design ( $2 \times 3$ ) involving three populations:

Human drivers from Lima, Peru.
Human drivers from New York City, USA.
Ten distinct VLMs from various startups and organizations.

Data Collection and Selection

Stimuli: The study utilizes dashcam footage recorded from the same camera setup in both Lima and NYC. From raw recordings, 200 five-second clips were extracted (100 per city).
OOD Selection Pipeline: To ensure the selection of challenging, out-of-distribution scenarios, the authors fed 200 clips to multiple VLMs with 20 VQA questions. They computed embeddings of the answers and selected the 10 clips from each city that exhibited the highest variance in model responses. These high-dispersion clips were interpreted as OOD scenes and used as the final stimuli for the survey.
Participants: The study recruited 10 human participants from Lima and 10 from NYC. All humans were verified university students (aged 18–30) with valid driver's licenses. All 30 systems (20 humans + 10 VLMs) were prompted with the same 20 questions in English.

Visual Question Answering (VQA) Framework

The evaluation utilized a four-block question structure:

Factual (Q1–Q5): Questions regarding weather, time, traffic signals, and flow (unique per video).
Ratings (Q6–Q10): Subjective quantitative assessments (1–10 scale) of clutter, hazard, likelihood of encountering similar scenarios, and expected performance of autonomous vehicles vs. human drivers.
Counterfactual & Hypothetical (Q11–Q15): Creative questions exploring crash scenarios without a ground truth.
Reasoning (Q16–Q20): Complex reasoning questions regarding right-of-way and pedestrian safety, designed to probe Chain-of-Thought capabilities.

Analysis Techniques

Embedding Analysis: Answers were projected into 2D space using Principal Component Analysis (PCA) and analyzed via Representational Similarity Analysis (RSA) to visualize convergence/divergence patterns.
Systematic Bias Analysis: Multi-Dimensional Scaling (MDS) was applied to the L1 distances of rating scores. VLMs were prompted 20 times per video-question pair to construct distributional violin plots.
Semantic Similarity: An "LLM-as-a-Judge" framework (using Qwen3-4B in thinking mode) performed pairwise comparisons of answers, scoring alignment on a scale from -2 (strong disagreement) to 2 (strong agreement).

Key Results

1. Geography vs. Question Type

The analysis revealed that response patterns were modulated primarily by the type of questions asked (e.g., Factual vs. Reasoning) rather than the geography of the video data.

Geographic Invariance: There was no strong difference in answers (from either Humans or VLMs) based on whether the video was from Lima or NYC. The authors attribute this to the high OOD nature of the selected clips, which likely overwhelmed geographic nuances.
Human Consistency: Humans from Lima and NYC demonstrated very similar answering patterns regardless of the video source.

2. Human vs. VLM Divergence

Convergence/Divergence: Humans and VLMs diverged in their responses, but this divergence was dependent on the question category.
Geographic Modulation of Divergence: While human patterns were consistent across geographies, the gap between Humans and VLMs was slightly more pronounced in the Lima context compared to NYC. Specifically, VLMs appeared further from the human cluster in Lima than in NYC.
VLM Consistency: Different VLMs shared similar answering patterns regardless of their specific training data or procedures, supporting the "Platonic Representation hypothesis."

3. Specific Findings on Ratings

Hazard Perception (Q8): VLMs showed high similarity in rating hazard levels across both geographies.
Self-Awareness (Q9): When asked to rate how well an autonomous vehicle would handle the scene, VLMs showed the highest distance (dissimilarity) across geographies, suggesting a sensitivity to geographic context when evaluating self-driving capabilities.

Key Contributions

Robusto-2 Dataset: The release of a fully factorial dataset containing dashcam footage from Lima and NYC, annotated with VQA responses from 20 humans and 10 VLMs.
Cross-Cultural Benchmark: A novel study design that controls for geography by including human subjects from both the "training" region (US/NYC) and a "non-native" region (Lima), addressing biases present in previous studies (e.g., Robusto-1) that compared non-native speakers to US-trained models.
Methodological Framework: A pipeline for selecting OOD video clips based on model disagreement variance, ensuring the benchmark focuses on edge cases rather than average driving scenarios.

Significance and Claims

The paper claims that while a gap exists between human and machine cognition, there is no significant cross-cultural generalization gap between geographies (Lima vs. NYC) for either humans or VLMs when facing highly chaotic, OOD driving scenarios.

The authors emphasize that their findings suggest both humans and VLMs navigate these chaotic environments in "similarly chaotic ways." This implies that the lack of geographic specificity in model responses may be due to the inherent difficulty of the OOD scenarios rather than a failure of cultural adaptation.

Limitations and Modesty:
The authors explicitly note the study's modest scale (20 clips, 20 questions, 30 subjects, totaling ~12,000 data points). They acknowledge that while the data volume is low compared to aggregate crowd-sourced studies, the design allows for a unique dissection of inter-system differences that is often obscured in "Mega-Human" aggregate datasets. The primary bottleneck identified was the collection of high-quality human driver data, not the computation of VLM responses. The authors plan to expand this work to include larger populations and data from five continents in future iterations.