💬 NLP

Can AI Reason Like an Urban Planner? Benchmarking Large Language Models Against Professional Judgment

This paper introduces Urban Planning Bench (UPBench), a framework evaluating 25 large language models against professional planning judgment, revealing that while AI excels at analytical synthesis, it struggles with context-dependent regulatory and normative tasks due to specific epistemic limitations like regulatory hallucination and phronetic deficit.

Original authors: Yijie Deng, He Zhu, Wen Wang, Junyou Su, Minxin Chen, Wenjia Zhang

Published 2026-06-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yijie Deng, He Zhu, Wen Wang, Junyou Su, Minxin Chen, Wenjia Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question

Imagine you have a super-smart robot that has read every book, article, and law ever written about cities. You ask it: "How should we fix this crowded neighborhood?"

The big question this paper asks is: Can this robot actually think like a human urban planner, or is it just a very good mimic?

To find out, the researchers built a special test called UPBench (Urban Planning Bench). Think of it like a "driver's license exam" for AI, but instead of driving a car, the AI has to solve complex city problems.

How They Tested the AI

The researchers didn't just ask the AI simple trivia. They created a giant grid (a 4x5 matrix) to test the AI on two main things:

What it knows: Four types of knowledge (Planning Rules, Mixing different subjects, How government works, and Real-world practice).
How it thinks: Five levels of thinking, from simple memory (Remembering facts) to complex judgment (Making tough decisions).

They tested 25 different AI models (both American and Chinese) against this test.

The Surprising Results: The "Upside-Down" Test

Usually, we expect AI to be great at simple things (like memorizing facts) and bad at hard things (like making complex judgments).

But this paper found the opposite. The AI's performance looked like a "U" shape or a rollercoaster:

The Easy Part (Remembering): The AI was fantastic at recalling facts. If you asked, "What is the definition of a zoning law?" it got it right almost 90% of the time.
The Hard Part (Understanding): Here is the shocker. When asked to explain why a concept works or how two ideas connect, the AI crashed. Its score dropped to about 55%. It could recite the words, but it didn't truly get the meaning.
The "Smart" Part (Analyzing): Weirdly, the AI got good again when asked to analyze complex scenarios. It could write long, logical-sounding essays about city problems.
The Human Part (Judging): When asked to make a final, value-based decision (like "Should we build a park here or a hospital?"), the AI struggled the most.

The Analogy: Imagine a student who can recite the entire rulebook of a sport perfectly (Remembering) and can write a beautiful essay about the history of the game (Analyzing). But if you ask them to actually play the game and make a split-second decision on the field based on the crowd's mood and the referee's mood (Understanding/Judging), they freeze up.

The Four Ways AI Fails (The "Epistemic Diagnostics")

The paper found that when AI fails at planning, it fails in four specific, predictable ways:

Regulatory Hallucination (The "Fake Lawyer"):
The AI confidently invents laws that don't exist. It might say, "According to Section 402 of the city code..." when that section is completely made up. It sounds like a lawyer, but it's lying.
- Analogy: It's like a tour guide who invents a secret tunnel in a museum that doesn't exist, but describes it so vividly you almost believe it's real.
Conceptual Conflation (The "Word Blender"):
The AI mixes up similar-sounding ideas. It treats two different planning theories as if they are the same thing.
- Analogy: It's like a chef who thinks "salt" and "sugar" are the same because they are both white powders. They look similar, but if you use them interchangeably, the dish is ruined.
Wickedness Paralysis (The "Indecisive Robot"):
Real city problems are "wicked"—they have no perfect answer and involve conflicting values (e.g., housing vs. nature). The AI lists every possible factor but refuses to pick a side. It says, "It depends," instead of making a tough call.
- Analogy: Imagine a referee in a soccer game who sees a foul but says, "Well, both teams have valid points, so let's just keep playing." A real planner has to blow the whistle and make a call.
Phronetic Deficit (The "Missing Common Sense"):
This is the biggest gap. AI lacks "phronesis," which is a fancy Greek word for practical wisdom. It's the gut feeling a planner gets from years of experience, knowing how a specific neighborhood will react, or understanding the unspoken politics of a city council.
- Analogy: An AI can read a map of a forest, but it doesn't know that the ground is muddy in that specific spot because it rained last night, or that the locals hate that particular path. It lacks the "street smarts" that come from being there.

The "Where You're From" Problem

The paper also found that AI models perform better in the country they were trained in.

American AI is great at American laws but gets confused by Chinese city rules.
Chinese AI is great at Chinese rules but gets confused by American ones.
The Lesson: Planning knowledge isn't just "facts"; it's deeply tied to local culture, laws, and history. You can't just download a "global" planner; the AI needs to be trained on the specific local context.

What This Means for the Future (According to the Paper)

The paper suggests a strategy called "Differential Delegation." This means knowing exactly what to let the AI do and what to keep for humans:

Let the AI do: The "grunt work." Summarizing long reports, finding similar case studies, or brainstorming a list of ideas. It's great at the "Remember" and "Analyze" parts.
Keep for Humans: The "judgment." Interpreting specific local laws, making the final decision on controversial issues, and understanding the human/political context. The AI is too prone to "hallucinating" laws and "paralyzing" on tough choices to be trusted with these alone.

The Bottom Line

AI is not going to replace urban planners. Instead, it acts like a very fast, very well-read intern who knows a lot of facts but lacks common sense and cannot make tough moral or legal calls.

The paper concludes that the most valuable part of being a planner isn't knowing the rules (which AI can do); it's knowing how to apply those rules in a messy, real-world situation with real people. That "human touch" is something AI currently cannot replicate.

Technical Summary: Benchmarking Large Language Models Against Professional Judgment in Urban Planning

1. Problem Statement

The rapid deployment of Large Language Models (LLMs) in urban planning practice has precipitated an urgent epistemological question: to what extent can artificial intelligence replicate the contextual sensitivity, value awareness, and institutional literacy that constitute professional planning judgment? While existing literature offers broad ethical critiques or narrow technical applications (e.g., automated plan evaluation or sentiment analysis), there is a critical lack of systematic, domain-specific frameworks to evaluate whether AI systems can reason with the phronesis (practical wisdom) required for real-world planning. Unlike medicine or law, which have established benchmarks (Med-PaLM, LegalBench) to separate AI capabilities from human expertise, planning lacks an empirical instrument to measure what planners actually know and which dimensions of that knowledge remain irreducibly human. This gap leaves the profession vulnerable to over-reliance on AI for high-stakes tasks or the unnecessary rejection of AI for routine work, without evidence-based guidance on the boundaries of AI competence.

2. Methodology: Urban Planning Bench (UPBench)

To address this gap, the authors introduce Urban Planning Bench (UPBench), a domain-specific evaluative framework grounded in the discipline's own knowledge structure rather than generic computer science benchmarks.

2.1. Assessment Architecture

UPBench utilizes an Evidence-Centered Design (ECD) framework organized along a 4×5 matrix:

Four Knowledge Pillars: Derived from international licensure exams (AICP, RTPI, PIA) and curricula:
1. Principles of Urban Planning: Foundational theories and analytical concepts (Episteme).
2. Cross-Disciplinary Integration: Synthesis of adjacent fields like engineering, ecology, and economics (Techne).
3. Planning Governance: Legal, regulatory, and institutional conditions (Interpretive).
4. Planning Practice: Integrative professional judgment in project contexts (Phronesis).
Five Cognitive Levels: Adapted from Bloom's revised taxonomy (Anderson & Krathwohl, 2001):
1. Remember: Factual recall of concepts and theories.
2. Understand: Explaining relationships and theoretical logic.
3. Apply: Deploying knowledge to solve specific problems.
4. Analyze: Critically evaluating propositions and claims.
5. Evaluate: Judging the validity of scenarios and supplying precise terminology.
  (Note: The "Create" level was excluded to focus on professional judgment rather than open-ended production.)

2.2. Data Construction and Validation

Scenario Generation: The benchmark comprises 405 assessment scenarios derived from a cross-national pipeline. Sources include AICP preparation materials, the Chinese National Registered Urban Planner Examination (2,150 items), and graduate curricula from MIT, UC Berkeley, Tongji, and Peking University.
Cross-Contextual Equivalence: To ensure the benchmark tests reasoning rather than regulatory nomenclature, Chinese institutional concepts (e.g., "one-document-three-certificates") were functionally mapped to their American counterparts (e.g., entitlement sequences) while preserving cognitive demands.
Dual-Track Evaluation Protocol:
- Track 1 (Automated): High-capacity LLMs act as judges for structured tasks (Remember, Understand, Apply), scoring responses against expert-authored rubrics and analyzing Chains of Thought (CoT).
- Track 2 (Expert Panel): A five-member panel of planning academics and practitioners evaluates open-ended reasoning (Analyze, Evaluate) for depth, contextual sensitivity, and professional appropriateness.
- Calibration: The two tracks were aligned via prompt optimization, achieving a Spearman $\rho$ of 0.67, deemed acceptable given the inherent epistemological differences between structured and judgment-intensive assessment.

2.3. Models Evaluated

The study evaluated 25 LLMs, including 22 open-weight models (dominated by Alibaba's Qwen ecosystem, Meta's Llama, and Google's Gemma) and 3 proprietary API models (GPT-4o-mini, Gemini 2.5 Flash, Claude Haiku). The focus on open-weight models prioritizes reproducibility and transparency.

3. Key Results

3.1. The Non-Monotonic Cognitive Curve

Contrary to the assumption that LLMs excel at lower-order tasks and fail at higher-order ones, UPBench revealed a non-monotonic performance curve:

Remember (89.6%) and Analyze (81.8%) showed high performance.
Understand (55.3%) exhibited a "catastrophic collapse," dropping nearly 34 percentage points from the Recall level.
Apply (76.2%) recovered but remained below Analyze.
Evaluate (37.9%) performed as expected (lowest).
This indicates that planning's "lower-order" knowledge (conceptual understanding) is deeply embedded in institutional and jurisdictional context, making it harder for pattern-generalization strategies than ostensibly "higher-order" analytical synthesis.

3.2. Knowledge Domain Asymmetries

Performance varied across the four pillars, mapping onto the Aristotelian knowledge typology:

Cross-Disciplinary Integration (69.3%): Highest performance; models excel at synthesizing diverse literatures.
Planning Governance (68.8%): Moderate performance; high accuracy on factual recall but prone to "regulatory hallucination" in application.
Principles of Urban Planning (67.6%): Strong on canonical frameworks but weak on contextual application.
Planning Practice (65.6%): Lowest performance; models struggle with situated, strategic, and politically aware reasoning.

3.3. Four Epistemic Diagnostics (Failure Modes)

Qualitative analysis identified four systematic failure modes that illuminate dimensions of planning expertise resistant to computational replication:

Regulatory Hallucination: The fabrication of specific regulatory requirements or legal citations that do not exist, blending elements from different jurisdictions into plausible but factually incorrect responses.
Conceptual Conflation: The erosion of theoretical precision, where models treat distinct planning paradigms (e.g., medieval organic growth vs. modernist functionalism) as interchangeable.
Wickedness Paralysis: The retreat from normative commitment. When faced with conflicting values (e.g., equity vs. property rights), models enumerate factors exhaustively but refuse to make the value-laden judgments required for professional decision-making.
Phronetic Deficit: The absence of situated practical wisdom. Models lack the tacit, experience-based knowledge regarding community dynamics, political feasibility, and the gap between plan rhetoric and on-the-ground reality.

3.4. Cross-National Institutional Asymmetry

Performance gaps between US and Chinese scenarios were largely mediated by model origin rather than task difficulty. US-origin models showed a significant advantage (+13.6%) on US scenarios, while China-origin models showed near-zero gaps (-1.7%). This suggests that training data composition is the primary driver of competence, challenging the notion of a single "global" model capable of equitable cross-cultural performance.

4. Key Contributions

Methodological: Introduction of UPBench, the first comprehensive, theory-grounded framework for evaluating AI planning competence, constructed from within planning's own evaluative traditions (plan quality evaluation, licensure standards) rather than imported from computer science.
Theoretical: The use of AI failure patterns as an "epistemic mirror" to empirically demonstrate the structure of planning knowledge. The study confirms that planning expertise is irreducibly institutional, normative, temporal, and relational, characterized by phronesis rather than mere technical procedure.
Practical: A framework for differential delegation, providing an evidence-based guide for planning practitioners and educators to determine which tasks can be responsibly augmented by AI (e.g., literature review, preliminary synthesis) and which require irreducible human judgment (e.g., regulatory interpretation, normative conflict resolution).

5. Significance and Claims

The paper claims that the inverted cognitive gradient—where AI struggles more with planning's "simple" institutional knowledge than with its "complex" analytical tasks—reveals that planning expertise is structured differently from other professions. The authors argue that the profession's distinctive value proposition lies not in analytical sophistication (which AI increasingly provides) but in integrative institutional judgment and phronetic wisdom.

The study concludes that while LLMs can serve as productive tools for cross-disciplinary synthesis and broad analytical reasoning, they exhibit persistent incapacity in jurisdiction-specific regulatory interpretation, normative conflict resolution, and context-sensitive procedural application. Consequently, the authors advocate for a reorientation of planning education from knowledge transmission toward cultivating institutional literacy, paradigmatic precision, normative courage, and AI critical literacy. The findings suggest that the "human-in-the-loop" is not merely a safety check but a fundamental requirement for the legitimacy of planning judgment, as the core of the profession is constituted through embeddedness in specific governance contexts and communities rather than through computation alone.