💻 computer science

Can Large Language Models Substitute for Human Raters? Reliability and Validity of Urban Design Quality Assessment

This study evaluates the reliability and validity of multimodal LLMs for urban design assessment in Seoul, finding that while they perform well on visually explicit features, their inability to fully replicate human ratings for contextual or proportional variables limits their suitability to element-specific rather than universal applications.

Original authors: Wookjae Yang, Jihyun Hwang, Reid Ewing

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wookjae Yang, Jihyun Hwang, Reid Ewing

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to grade the beauty and safety of a neighborhood. Traditionally, you'd hire three expert inspectors to walk the streets, look at the buildings, count the trees, and fill out a detailed report card. This is accurate, but it takes a long time and costs a lot of money.

Now, imagine you have three super-smart AI robots (called Multimodal Large Language Models, or LLMs) that can look at the same street photos and fill out the exact same report cards instantly. The big question this paper asks is: Can we just fire the human inspectors and let the robots do the whole job?

The short answer from the study is: Not yet. The robots are great at some things, but they miss the mark on others.

Here is a breakdown of what the researchers found, using simple analogies:

1. The "Big, Obvious" vs. The "Small, Subtle"

Think of the neighborhood like a messy room.

What the Robots Get Right: If you ask the robots to count the big, obvious items—like "How many red brick walls are there?" or "Are there any large commercial signs?"—they are surprisingly consistent. They are like a robot that can perfectly count the number of red chairs in a room. They see the big, structural things clearly.
What the Robots Miss: If you ask them to estimate "How much of the wall is covered by windows?" or "How much of the sky can you see above the trees?", they get confused. They also struggle to spot tiny, scattered items like a single trash can on the sidewalk or a security camera hidden in a corner. It's like asking the robot to estimate the exact percentage of a rug that is blue versus red, or to find a single lost coin in a pile of leaves. They often miss these small details or guess the proportions wrong.

2. The "Agreement" Problem

The researchers had three human inspectors and three AI robots look at the same 40 spots in a Seoul neighborhood.

Humans vs. Humans: The three humans mostly agreed with each other. If one said a street was "safe," the others usually agreed.
Robots vs. Robots: The three robots also mostly agreed with each other. They were consistent in their own way.
Humans vs. Robots: This is where it gets tricky. Even though the robots were consistent with themselves, they often disagreed with the humans.
- The Analogy: Imagine the humans and robots are both taking a photo of a sunset. The humans might say, "It's 70% orange." The robots might say, "It's 30% orange." They are both looking at the same thing, but they are measuring it differently. The robots tend to see fewer streetlights, fewer trees, and fewer signs than the humans do.

3. The "Comfort" Test

To see if the robots' reports actually mattered, the researchers checked if their scores matched how comfortable people felt on those streets.

The Good News: For some things, like "Are there trees?" or "Is there a fence?", the robots and humans agreed on the pattern. When the robots said "more trees," people felt more comfortable, just like when the humans said "more trees."
The Bad News: For other things, the robots got it wrong. For example, the humans thought "parking under a building" made people feel more comfortable, but the robots didn't see that connection. The robots also missed the link between "noise" and "discomfort" that the humans caught.

The Bottom Line

The study concludes that you can't just swap humans for AI in urban design yet.

The Robots are "Specialists," not "Generalists." They are excellent at spotting big, clear architectural features (like building materials or signs).
They are "Blind" to Context. They struggle with things that require guessing proportions, spotting small details, or understanding how different parts of a street work together to create a feeling.

The Verdict: Instead of replacing the human inspectors, the best approach right now is a hybrid team. Let the robots do the easy, obvious counting (like counting big signs), but keep the humans to handle the tricky stuff (like estimating tree coverage or spotting small safety hazards). The robots are a helpful assistant, but they aren't ready to take the lead.

Technical Summary: Can Large Language Models Substitute for Human Raters? Reliability and Validity of Urban Design Quality Assessment

1. Problem Statement

The assessment of urban design quality traditionally relies on trained human observers to evaluate fine-grained environmental attributes (e.g., enclosure, transparency, façade conditions) using standardized checklists. While the proliferation of geo-tagged street-level imagery has created vast visual datasets, the labor-intensive nature of human-based rating systems severely constrains their scalability. The emergence of multimodal Large Language Models (LLMs), capable of interpreting visual inputs and applying structured scoring rubrics, raises a fundamental methodological question: Can LLMs reliably replace trained human raters in urban design assessment?

Current research gaps include a lack of systematic evaluation regarding the inter-rater reliability of AI-based assessments and the extent to which AI outputs align with human judgments. Existing AI approaches often focus on feature extraction via computer vision rather than applying the complex, contextual evaluative criteria used by human experts. This study addresses whether LLMs can serve as valid, reliable substitutes for human raters or if their utility is limited to supportive roles.

2. Methodology

Study Area and Sampling

The study was conducted in Hwagok-dong, Seoul, a dense, low-rise villa neighborhood characterized by heterogeneous building conditions, narrow streets, and significant variation in streetscape quality.

Sampling: 40 observation points were selected using a systematic spatial sampling approach (100-meter intervals) followed by random sampling to ensure representative coverage and minimize selection bias.
Data: High-resolution street-level imagery was captured at each point.

Assessment Protocol

Three distinct groups evaluated the 40 observation points using an identical standardized rubric derived from established frameworks (e.g., Ewing & Clemente, 2013):

Human Raters: Three trained observers conducted field surveys.
LLM Raters: Three multimodal models (GPT-5.3 Thinking, Gemini 3.5 Flash, and Claude Sonnet 4.6) were prompted with the same standardized instructions and visual inputs.

Variables

Urban design quality was operationalized into two types of variables:

Continuous Variables: Counts (e.g., commercial signs, streetlights) and proportional measures (e.g., window-to-wall ratio, tree coverage, sky view factor).
Categorical Variables: Presence/absence or types of features (e.g., lane markings, building materials, fences, CCTV cameras).

Statistical Analysis

The study employed a three-pronged reliability and validity framework:

Within-Group Reliability:
- Continuous Variables: Intraclass Correlation Coefficient (ICC) using a two-way random-effects model for absolute agreement [ICC(2,1)].
- Categorical Variables: Fleiss' kappa ( $\kappa$ ) to quantify agreement among multiple raters.
Cross-Group Agreement (Human vs. LLM):
- Continuous Variables: Bland–Altman analysis to quantify systematic bias and 95% limits of agreement (LOA).
- Categorical Variables: Fleiss' kappa calculated between the human group and the LLM group.
Construct Validity:
- Ordinary Least Squares (OLS) regression models were estimated to predict perceived comfort level (dependent variable) using design variables rated by humans and LLMs separately. This tested whether AI-rated variables retained meaningful associations with human outcomes.

3. Key Results

3.1 Inter-Rater Reliability

Human Raters: Demonstrated generally strong reliability for visually explicit, countable features (e.g., commercial signs, retail facilities, window security bars; ICC > 0.6). Reliability dropped for variables requiring proportional estimation or detection of small details (e.g., illegal parking, benches).
LLM Raters: Showed a selective performance pattern.
- High Reliability: LLMs performed comparably to humans for structurally stable, visually bounded attributes (e.g., building materials, retail presence, window security bars).
- Low Reliability: Reliability dropped significantly for variables requiring proportional estimation (e.g., tree coverage, sky view factor) or detection of small-scale, partially occluded elements (e.g., streetlights, safety facilities).
Comparison: Human raters were more consistent than LLMs across most variables, particularly those requiring contextual interpretation.

3.2 Human-LLM Agreement

Systematic Bias: Bland–Altman analysis revealed that LLMs systematically assigned lower values than humans for most continuous variables.
Limits of Agreement: Wide limits of agreement were observed for variables involving proportional judgment (e.g., window-to-wall ratio, sky view factor, tree coverage), indicating that human and AI ratings are not interchangeable for these metrics.
Categorical Disagreement: Cross-group Fleiss' kappa was generally modest. While agreement was substantial for "excessive signage" ( $\kappa = 0.670$ ), it was weak or negative for small-scale elements like trash cans, CCTV cameras, and wires/poles. In some cases (e.g., trash cans), human and LLM classifications were systematically different rather than just inconsistent.

3.3 Construct Validity

Model Fit: Both human-rated and LLM-rated models explained a similar proportion of variance in perceived comfort (43.0% vs. 42.6%), suggesting LLMs capture aggregate environmental information.
Variable-Level Divergence:
- Consistent Associations: Tree coverage, window-to-wall ratio, and fence presence showed consistent directional associations with comfort in both models.
- Divergent Associations: Variables such as pilotis parking, lane markings, and building materials showed significant differences in direction or significance between the two models. For instance, pilotis parking was positively associated with comfort in the human model but not the LLM model.

4. Key Contributions

Differentiation of Reliability, Agreement, and Validity: The study distinguishes between internal consistency (reliability) and cross-group equivalence (agreement). It demonstrates that an AI system can be internally consistent (reliable) while still failing to reproduce human judgments (low agreement), challenging the assumption that high reliability implies substitutability.
Element-Specific Performance: The research identifies that LLM performance is element-specific rather than universal. LLMs are effective for visually explicit, architecturally legible attributes but struggle with qualities requiring proportional estimation, contextual inference, or the detection of fine-grained details.
Methodological Framework: The study provides a rigorous statistical framework (combining ICC, Fleiss' kappa, and Bland–Altman analysis) for evaluating AI as a measurement instrument in urban planning, moving beyond simple predictive accuracy to assess measurement interchangeability.

5. Significance and Claims

The paper concludes that multimodal LLMs are not yet general substitutes for trained human raters in urban design quality assessment.

Selective Suitability: LLMs are best suited as selective measurement tools for specific classes of visually explicit and structurally stable variables. They are not yet capable of fully replicating the nuanced, contextual judgments required for assessing qualities like openness, greenery, and maintenance conditions in complex, low-rise environments.
Hybrid Workflow: The authors propose a hybrid workflow where AI handles visually explicit variables to scale up assessment, while human raters remain essential for context-sensitive, proportional, or low-frequency features.
Modest Claims on Validity: While LLMs preserve some meaningful associations with human well-being (specifically for variables like tree coverage and enclosure), they do not reproduce the full pattern of relationships observed in human-rated measures. The study asserts that AI should be viewed as a complementary and screening tool rather than a fully autonomous evaluator, particularly in dense residential environments where subtle environmental cues are critical.

The study emphasizes that the future role of LLMs in urban design is likely to be complementary, selective, and hybrid, rather than a complete replacement of human expertise.