🤖 AI

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

This paper introduces 3DCity-LLM, a unified framework featuring a coarse-to-fine feature encoding strategy and a new 1.2M-sample dataset, which significantly advances multi-modality large language models' capabilities in 3D city-scale perception and understanding by outperforming existing state-of-the-art methods.

Original authors: Yiping Chen, Jinpeng Li, Wenyu Ke, Yang Luo, Jie Ouyang, Zhongjie He, Li Liu, Hongchao Fan, Hao Wu

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiping Chen, Jinpeng Li, Wenyu Ke, Yang Luo, Jie Ouyang, Zhongjie He, Li Liu, Hongchao Fan, Hao Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant that is great at describing what's in a single room or identifying a specific cat in a photo. But now, you ask it to look at an entire city from a drone's view, find the nearest hospital, calculate how far it is from a specific park, and then suggest how to redesign the traffic flow to make it safer for pedestrians.

Currently, most AI assistants would get overwhelmed. They might know what a "building" looks like, but they struggle to understand how thousands of buildings fit together, how far apart they are, or how a city functions as a whole system.

This paper introduces 3DCity-LLM, a new "brain" designed specifically to solve this problem. Here is a simple breakdown of how it works, using some everyday analogies.

1. The Problem: The "Zoom Lens" vs. The "Map"

Think of existing AI models as having a zoom lens. They are excellent at zooming in on one object (like a car or a tree) and describing it. But if you zoom out to see the whole city, they get lost. They can't easily connect the dots between a hospital, a railway station, and the traffic in between.

Furthermore, the data they usually learn from is like a 2D photo album. It shows pictures of cities, but it doesn't tell the AI the exact height of a building, the precise distance between two streets, or the 3D shape of a park. It's like trying to navigate a city using only a flat drawing without knowing the elevations or distances.

2. The Solution: The "City Architect" (3DCity-LLM)

The authors built a new AI framework called 3DCity-LLM. Instead of just looking at a picture, this AI acts like a City Architect who has three special tools:

The Magnifying Glass (Target Object): It zooms in to understand a specific building or object (e.g., "What is this red building?").
The String Theory (Relationships): It uses invisible strings to measure how objects relate to each other (e.g., "This hospital is 400 meters southeast of the train station").
The Bird's-Eye View (Global Scene): It steps back to see the whole neighborhood, understanding how parks, roads, and zones fit together.

By combining these three views, the AI can answer complex questions like, "Which hospital is closest to the train station, and where is its emergency room located?" with precise numbers and logical reasoning.

3. The Training Ground: The "City Encyclopedia" (3DCity-LLM-1.2M)

To teach this AI, the researchers couldn't just use a few examples. They needed a massive library. They created a new dataset called 3DCity-LLM-1.2M.

Size: It contains 1.2 million examples. That's like reading 1.2 million different city guides.
The Content: It's not just simple questions like "What is this?" It includes complex tasks like:
- Object Analysis: "What is this building used for?"
- Relationship Math: "How far is Building A from Building B?"
- City Planning: "If we build a new road here, how will it affect pedestrian safety?"
The Twist: The dataset simulates different people asking the questions. Sometimes a tourist asks, "Where can I see a cool landmark?" and sometimes a city planner asks, "Is this zoning compliant?" This teaches the AI to speak the right language for the right person.

4. The Exam: The "Human Judge" (New Evaluation)

Here is a clever part of the paper. Usually, we test AI by checking if its answer matches the "correct" answer word-for-word (like a spell-checker). But in city planning, there are many ways to say the same thing.

Wrong Way: If the AI says, "The park is 50 meters away," and the correct answer is "The park is 50m to the south," a spell-checker might say they are different.
Right Way: The authors used other AI models as judges. These "Judge AIs" read the answer and ask: "Does this make sense logically? Is it factually true based on the map?"

This ensures that the AI gets credit for being smart and accurate, even if it uses different words than the textbook.

5. The Results: The "Graduate Student"

When they tested 3DCity-LLM against other top AI models:

It was better at describing individual objects.
It was much better at calculating distances and directions.
It could actually plan solutions for city problems (like suggesting where to put a crosswalk) with high reliability.

Summary

In short, the authors built a super-urbanist AI. They gave it a massive, detailed 3D map of cities (the dataset), taught it to look at things from three different angles (the framework), and tested it with human-like judges. The result is an AI that doesn't just "see" a city; it understands how a city works, can do the math on distances, and can even help us plan better cities for the future.

1. Problem Statement

While Multi-modality Large Language Models (MLLMs) have achieved success in object-centric or indoor scenarios, scaling them to 3D city-scale environments remains a significant challenge. Existing approaches face three primary limitations:

Data Scarcity & Quality: Current datasets are often limited to 2D aerial/street views (lacking 3D spatial info), restricted to short-form QA, or focused on specialized tasks (e.g., grounding) without the scale or diversity needed for training robust LLMs.
Complexity of Urban Scenes: City environments contain thousands of heterogeneous entities with intricate spatial relationships. Models must not only recognize objects but also model interactions, functional roles, and global scene semantics.
Evaluation Deficiencies: Traditional text-similarity metrics (BLEU, ROUGE) fail to evaluate open-ended, long-form urban responses. They penalize semantically correct answers that differ in wording from the ground truth, failing to capture logical coherence or factual alignment with 3D evidence.

2. Methodology

A. 3DCity-LLM Framework

The authors propose a unified framework built upon a pre-trained LLM backbone (LLaVA-7B) with a coarse-to-fine feature encoding strategy. The model processes multi-modality inputs (text queries, 3D point clouds, and 2D images) through three parallel branches:

Object Encoding: Extracts features for the target object, including:
- Local View: Top-view RGB patches via CLIP.
- Shape: 3D point sampling via Uni3D.
- Landmark: Semantic embedding of object names (e.g., "City Hall") via BERT.
Relationship Encoding: Captures context by identifying $K$ $K$ -nearest neighbors. It encodes:
- Context Geometry: Relative positions and shape features of neighbors, weighted by attention mechanisms.
- Context Landmark: Semantic embeddings of neighboring objects.
Scene Encoding: Represents the global city context via:
- Global View: A top-down RGB map of the entire scene.
- Global Landmark: Aggregated semantic features of all named landmarks in the scene.

These features are projected into the LLM's embedding space. The model uses an instruction-driven paradigm to dynamically activate specific branches based on the task type (object-level, relationship-level, or scene-level).

B. Training Strategy

The model undergoes a two-stage training process:

Feature Alignment: Trained on simple captioning tasks to align text, 2D images, and 3D point cloud modalities.
Scene Understanding: Fine-tuned on high-level analysis and planning tasks to enable complex reasoning and holistic scene comprehension.

C. The 3DCity-LLM-1.2M Dataset

To support training, the authors constructed a massive dataset containing 1.2 million high-quality samples derived from SensatUrban, UrbanBIS, and City-BIS.

Generation Pipeline: An automated pipeline extracts structured city attributes (3D coordinates, distances, topology) and uses advanced VLMs (e.g., ChatGPT-5) to generate QA pairs.
Contextual Simulation: The generation process simulates diverse user personas (tourists, officials, staff) to ensure linguistic diversity and realistic reasoning depth.
Task Taxonomy: The dataset covers seven task categories:
1. Object Caption
2. Object Localization
3. Object Analysis
4. Relationship Computation
5. Scene Caption
6. Scene Analysis
7. Scene Planning (goal-oriented reasoning)

D. Evaluation Protocol

The paper introduces a multi-dimensional evaluation protocol to address the limitations of traditional metrics:

Text-Similarity Metrics: BLEU-4, ROUGE-L, METEOR.
LLM-Based Semantic Assessment: Two new metrics, Logicality (coherence and reasoning flow) and Reliability (factual correctness and alignment with 3D evidence), are scored by independent LLM evaluators (ChatGPT-5, Qwen3-VL, DeepSeek-V3) with explicit reasoning traces.

3. Key Contributions

3DCity-LLM Framework: A novel architecture that unifies object, relationship, and scene-level understanding through coarse-to-fine feature encoding, specifically designed for city-scale 3D perception.
3DCity-LLM-1.2M Dataset: The largest and most diverse city-scale vision-language dataset to date, featuring explicit 3D numerical information, 7 task categories, and diverse user-oriented simulations.
Advanced Evaluation Protocol: A rigorous assessment framework combining standard metrics with LLM-based logical and reliability scoring, ensuring a faithful evaluation of open-ended urban reasoning.

4. Experimental Results

Extensive experiments were conducted on the 3DCity-LLM-1.2M and City-3DQA benchmarks.

Performance vs. Baselines: 3DCity-LLM significantly outperformed state-of-the-art methods (including LLaVA, Chat 3D, City-VLM, and zero-shot LVLMs like ChatGPT-5.1 and Qwen3-VL).
- Object-Level Tasks: Achieved a BLEU-4 of 30.64 (vs. 28.67 for the next best, Chat Scene).
- Scene-Level Tasks: Achieved a BLEU-4 of 20.11 and METEOR of 37.84, demonstrating superior holistic understanding.
- City-3DQA: Achieved the highest sentence-wise accuracy (68.55) and competitive city-wise accuracy.
Ablation Studies: Confirmed that the combination of local view, shape, landmark, and relationship features yields the best performance. Removing any component (e.g., landmarks or geometry) resulted in measurable drops in accuracy and reliability.
Qualitative Analysis: The model demonstrated robustness in handling diverse query phrasings and generalizing to unseen objects and scenes. It successfully performed spatial reasoning (calculating distances/orientations) and goal-oriented planning (e.g., suggesting traffic improvements).

5. Significance

Bridging the Gap: This work successfully bridges the gap between current MLLM capabilities and the complex requirements of 3D city-scale perception, moving beyond indoor or object-centric limitations.
Urban Intelligence: By enabling models to understand spatial relationships, functional roles, and planning constraints, 3DCity-LLM offers a promising direction for applications in urban planning, smart city management, and autonomous navigation.
Methodological Shift: The introduction of a multi-dimensional evaluation protocol sets a new standard for assessing generative AI in spatial tasks, moving beyond simple text matching to evaluate reasoning and factual grounding.
Resource Availability: The release of the 1.2M dataset and source code provides a critical foundation for future research in 3D vision-language understanding.