🤖 AI

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

This paper argues that benchmarks for vision-language models in urban perception must evolve to treat human disagreement and abstention as valid measurement outcomes, report inter-annotator reliability alongside model alignment, and frame label spaces as negotiable artifacts to better support urban governance applications.

Original authors: Rashid Mushkani

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rashid Mushkani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to describe a city street, not just by counting cars or trees, but by telling you how the street feels. Is it safe? Is it welcoming? Is it comfortable?

This paper argues that when we test these "Vision-Language Models" (robots that see and talk), we are making a big mistake if we treat their answers like a simple math test with one right answer. Instead, we need to treat these tests more like a town hall meeting where people have different opinions.

Here is the breakdown of the paper's main ideas using simple analogies:

1. The Problem: The "One Right Answer" Trap

Imagine you show a photo of a park to 12 different people and ask, "Is this park safe?"

Person A says, "Yes, it's very safe."
Person B says, "No, it feels dangerous at night."
Person C says, "I can't tell from this picture."

In traditional robot testing, we would force all these answers into a single "Ground Truth" (e.g., "Safe"). If the robot says "Unsafe," we mark it as wrong. If it says "Safe," we mark it as right.

The Paper's Argument: This is unfair. The "truth" here isn't a single fact; it's a messy mix of opinions. If we ignore the disagreement, we might think the robot is failing when it's actually just reflecting a valid human opinion.

2. The Solution: The "Reliability-Aware" Scorecard

The authors suggest we need a new kind of report card for these robots. Instead of just giving a score (like 85%), the report card should also show:

How much humans disagreed: If humans can't agree on whether a street is "safe," the robot shouldn't be penalized for guessing.
Who said "I don't know": Sometimes, the best answer is "I can't judge." If a robot guesses when it should have stayed silent, that's a problem.

The Analogy: Think of it like a weather forecast. If 12 meteorologists look at a storm and 6 say "Rain" and 6 say "Snow," a good forecast shouldn't just pick one and say "It's Rain." It should say, "There is a 50/50 split, and here is the uncertainty."

3. The Experiment: The Montreal Street Test

To prove this, the researchers created a specific test:

The Scene: They took 100 pictures of streets in Montreal (some real photos, some computer-generated).
The Judges: They asked 12 real people from local community groups to describe these streets on 30 different topics (like "Is it clean?" or "Does it feel inclusive?").
The Robots: They asked 7 different AI models to describe the same streets using the exact same instructions.

What They Found:

The "Easy" Stuff: When the question was about something obvious (like "Are there trees?"), humans agreed a lot, and the robots did well.
The "Hard" Stuff: When the question was about feelings (like "Is it comfortable?"), humans disagreed a lot. The robots also struggled, but interestingly, they sometimes disagreed with the pattern of human disagreement.
The "Silent" Issue: Humans often said, "I can't judge this." The robots, however, often tried to guess anyway. This is a mismatch in behavior.

4. The "Negotiated" Approach

The paper argues that these tests shouldn't be set in stone. They should be negotiated.

The Analogy: Imagine a recipe for a city. If the people living in the city say, "This ingredient (the definition of 'safety') doesn't make sense to us," the recipe shouldn't just be ignored. The people and the scientists should sit down and rewrite the recipe together.

The authors say that when we use these robots to make decisions about cities (like where to build a park or how to police a street), we must:

Show the disagreement: Don't hide the fact that people argue about what "safe" means.
Keep the rules flexible: If the community says the rules are wrong, we should be able to change the test and re-run it.
Be honest about uncertainty: If the robot isn't sure, or if humans aren't sure, that uncertainty should be visible in the final report.

Summary

This paper tells us that when we use AI to understand how people feel about their cities, we can't just look for a "correct" answer. We have to accept that people disagree. A good test for these robots must show us how much humans disagreed and how often the robot refused to guess, rather than just giving a single number that pretends the world is simple and clear-cut.

If we don't do this, we might build robots that think they are "smart" because they agree with a made-up average, while actually missing the real, messy, and important conversations happening in our communities.

Technical Summary: Reliability-Aware and Negotiated Benchmarking for Urban Perception VLMs

Problem Statement
Vision–language models (VLMs) are increasingly deployed to generate structured descriptions of street-level imagery for urban governance tasks, such as streetscape auditing and public consultation. These tasks often require combining observable physical attributes with subjective appraisal categories (e.g., comfort, safety, inclusivity). A critical evaluation problem arises because the "ground truth" for these appraisals is not a stable label but a distribution of human judgments characterized by disagreement, abstention, and plurality.

Current benchmarking practices often collapse these pluralistic judgments into a single label, treating disagreement as annotation error. This approach obscures systematic uncertainty and makes model accuracy contingent on the specific labeling population and the semantics of abstention. The paper argues that for domains where model outputs inform institutional decisions, benchmarking must treat disagreement and abstention as informative signals rather than noise, and must treat label spaces as negotiable artifacts subject to revision by affected communities.

Methodology
The paper grounds its argument in a new empirical benchmark and a deterministic zero-shot evaluation protocol.

Dataset: The benchmark consists of 100 street-level scenes from Montreal, organized into ten panels. The dataset is balanced between 50 photorealistic synthetic renders and 50 photographs.
Annotation: Twelve participants from seven community organizations annotated the images across 30 dimensions. These dimensions span objective settings (e.g., vegetation, built form) and subjective appraisals (e.g., overall impression, safety).
- Protocol: Annotations were collected in French. The study explicitly distinguishes between "observed absence" (e.g., "No trees") and "abstention" (e.g., "Not applicable" or "Cannot judge").
- Consensus: A deterministic French-to-English mapping was applied. For multi-label dimensions, a hard consensus set was formed by retaining options selected by at least 50% of annotators. For single-choice dimensions, majority vote was used, with exact ties flagged as "unsure" and excluded from accuracy calculations.
Models: Seven VLMs were evaluated in a zero-shot setting without task-specific fine-tuning: claude-sonnet, openai-o4-mini, gpt-4.1, gemini-2.5-pro, grok-2-vision, qwen2.5-vl, and llama-4-maverick.
Evaluation Metrics:
- Model Alignment: Accuracy for single-choice items and Jaccard index for multi-label items against the human consensus.
- Human Reliability: Inter-annotator reliability was computed per dimension using Krippendorff's $\alpha$ (nominal distance).
- Analysis: The study analyzed the correlation between human reliability and model alignment, compared performance on observable versus appraisal dimensions, and examined distributional mismatches in abstention rates.

Key Results

Correlation with Reliability: There is a positive co-variation between human inter-annotator reliability and model alignment. Dimensions with higher human agreement tend to yield higher model agreement. Dimensions with low human reliability often show high variance in model performance, suggesting that model scores on these dimensions may reflect model priors rather than recoverable visual evidence.
Observable vs. Appraisal: Models generally achieve higher macro-scores on dimensions closer to observable attributes (e.g., physical structures) compared to appraisal dimensions (e.g., "Overall Impression").
Distributional Mismatch: For the "Overall Impression" dimension, models and annotators exhibit significant distributional mismatches. Specifically, models select "Not applicable" at higher rates than human annotators, while annotators more frequently use "Cannot judge." This indicates a divergence in how models and humans interpret uncertainty and the semantics of abstention.
Source Effects: Models achieved higher agreement with human consensus on photographs compared to synthetic renders, highlighting a domain shift related to generation artifacts, texture, and lighting.

Key Contributions

Reliability-Aware Benchmarking: The paper proposes a framework where benchmark reports must include inter-annotator reliability alongside model scores. This allows stakeholders to distinguish between model failure and inherent instability in the target concept.
Negotiated Benchmarking: It introduces the concept of benchmarks as "negotiated artifacts." Rather than treating label spaces as fixed, the paper advocates for procedures where affected communities can contest and revise definitions, scoring rules, and abstention policies.
Empirical Anchor: It provides a concrete, community-annotated dataset and evaluation protocol that makes disagreement and abstention observable, serving as a testbed for these methodological claims.
Disclosure Standards: The paper outlines a disclosure set (Table 1) for benchmark creators, including label specifications, judgment collection details, reliability reports, and revision records, to make assumptions visible.

Significance and Claims
The paper claims that for VLMs used in urban governance, treating appraisal targets as stable ground truths is a methodological failure that can lead to misleading conclusions about model progress and deployment risk. By treating disagreement and abstention as outcomes rather than errors, the proposed approach makes uncertainty visible.

The significance lies in shifting the focus from maximizing a single accuracy score to understanding the conditions under which alignment occurs. The authors argue that reliability-aware and negotiated benchmarking does not merely improve evaluation metrics but serves as a governance mechanism. It enables the revision of contested categories and ensures that model outputs used for institutional decisions remain open to contestation and accountability. The paper concludes that benchmark quality for appraisal targets depends as much on transparent assumptions and revision mechanisms as it does on statistical design.