💻 computer science

Leveraging Multimodal LLMs for Built Environment and Housing Attribute Assessment from Street-View Imagery

This paper presents a novel framework that leverages fine-tuned and distilled multimodal large language models to automatically and efficiently evaluate building conditions and housing attributes nationwide using Google Street View imagery, achieving high accuracy comparable to human raters while significantly reducing computational costs and labeling effort.

Original authors: Siyuan Yao, Siavash Ghorbany, Kuangshi Ai, Arnav Cherukuthota, Meghan Forstchen, Alexis Korotasz, Matthew Sisk, Ming Hu, Chaoli Wang

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyuan Yao, Siavash Ghorbany, Kuangshi Ai, Arnav Cherukuthota, Meghan Forstchen, Alexis Korotasz, Matthew Sisk, Ming Hu, Chaoli Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a city planner or a homeowner trying to fix up an old neighborhood. You know that many houses need repairs, but you don't have enough money or time to send a team of experts to knock on every door and inspect every roof. It's like trying to count every grain of sand on a beach with a magnifying glass—impossible for a human to do alone.

This paper presents a clever solution: teaching a super-smart AI to be your "digital inspector" by looking at photos of houses taken from the street (like Google Street View).

Here is the story of how they did it, broken down into simple steps:

1. The Problem: Too Many Houses, Too Few Humans

Traditionally, checking a house's condition requires a human expert to walk around, look at the paint, check the windows, and guess if the roof is sagging. This is slow, expensive, and hard to do for millions of houses.

2. The Star Player: The "Super-Brain" AI

The researchers started with a massive Artificial Intelligence model called Gemma 3 27B. Think of this model as a genius art critic who has read every book on architecture and seen millions of photos.

The Test: They showed this "genius" photos of houses and asked, "On a scale of 1 to 5, how bad is this house?" (1 = a pile of rubble, 5 = a brand-new mansion).
The Result: The AI was surprisingly good. It agreed with human experts almost as well as two human experts agree with each other. It could look at a cracked window and a peeling roof and say, "This is a 2 (Poor)."

3. The Speed Problem: The Genius is Too Slow

Here's the catch: The "genius" AI is so smart that it's also very heavy. It's like a Formula 1 car—incredibly fast on the track, but it guzzles gas and costs a fortune to run. If you tried to use this giant model to check millions of houses, it would take forever and cost a lot of money.

4. The Solution: Teaching a "Student" (Knowledge Distillation)

To fix the speed issue, the researchers used a technique called Knowledge Distillation.

The Analogy: Imagine the "Genius" AI is a Master Chef. The Master Chef tastes a dish and describes exactly how it should taste. Instead of hiring the Master Chef to cook every single meal for a city, the Master Chef teaches a Junior Chef (a smaller, faster AI model) how to cook the dish by tasting the Master's notes.
The Result: They trained a smaller, lighter AI (the Junior Chef) using the Master's "answers."
- The Junior Chef is 30 times faster than the Master.
- It is almost as accurate.
- It can run on a regular computer, not just a super-computer.

5. The "Magic Dashboard"

Knowing the condition of a house is great, but what do you do with that information? The researchers built a Visualization Dashboard.

The Analogy: Think of this as a Google Maps for home repairs.
When you click on a house on the map, the dashboard doesn't just give you a number. It shows you a "Report Card." It tells you: "The roof looks okay, but the paint is peeling, and the neighborhood feels a bit unsafe." It even estimates how much it might cost to fix.
This helps homeowners decide: "Should I fix this house, or is it time to move?"

6. The "Human vs. Robot" Check

The researchers were careful. They didn't just trust the AI blindly. They ran a study where they compared the AI's opinions against a group of human architecture experts.

The Finding: The AI was actually more consistent than humans. Humans get tired, hungry, or have bad days, so one expert might call a house a "3" while another calls it a "4." The AI, however, is like a robotic judge that never gets tired and always applies the same rules. This makes it perfect for checking thousands of houses fairly.

The Big Picture

This paper is about scaling up.

Before: We could only check a few houses at a time because humans are slow.
Now: We can use these "Digital Inspectors" to check millions of houses across the entire United States in a matter of hours.

Why does this matter?
It helps us find the "leaky" houses that need help before they become dangerous. It helps governments and charities target their money to the places that need it most, making our cities safer, warmer, and more affordable for everyone. It's like giving every house in America a quick, free check-up from a doctor who never sleeps.

1. Problem Statement

The United States faces a critical shortage of affordable housing, aging infrastructure, and rising energy costs. Retrofitting existing housing stock is a cost-effective solution, but assessing the physical condition of buildings (specifically the exterior envelope) is traditionally resource-intensive, relying on manual inspections that cannot be scaled nationwide.

Limitations of Current Methods: Traditional computer vision (CNNs) models often rely on narrow indicators (e.g., wall cracks) and struggle to integrate diverse visual elements for holistic assessment. Vision-language models (VLMs) show promise but lack the capacity for multi-factor reasoning and structured scoring based on formal criteria.
Goal: To develop a scalable, automated, and cost-effective framework for evaluating residential building conditions and extracting extensive housing attributes from Google Street View (GSV) imagery using Multimodal Large Language Models (MLLMs).

2. Methodology

The authors propose a pipeline that leverages MLLMs for both direct assessment and knowledge distillation to smaller models.

A. Data Collection and Preprocessing

Dataset: 12,063 GSV images from six US states (CA, FL, GA, IN, NY, TX).
Annotation: 1,281 images were independently rated by 7 human experts on a 1–5 scale (1: Uninhabitable to 5: Excellent) to generate Mean Opinion Scores (MOS). The remaining 10,782 images were used for unsupervised knowledge distillation.
Preprocessing: Images are cropped to isolate individual buildings using GroundingDINO to remove street-level clutter.

B. Model Architecture and Training

Base Model Selection: The framework utilizes Gemma 3 (specifically the 27B parameter version) as the primary multimodal backbone.
Prompt Engineering: Inputs include the cropped image and a structured text prompt containing:
- Formal rating criteria (definitions for scores 1–5).
- Instructions to evaluate specific aspects (paint, windows, structure, maintenance).
- Output formatting constraints.
Fine-Tuning Strategy:
- Parameter-Efficient Fine-Tuning (PEFT): Due to hardware constraints, the authors use QLoRA (Quantized Low-Rank Adaptation) to fine-tune Gemma 3 27B on a small subset of human-labeled data (500 images). This updates only ~16% of parameters, reducing memory usage and training costs.
Knowledge Distillation:
- Teacher-Student Framework: The fine-tuned Gemma 3 27B acts as a "teacher" to generate pseudo-labels for the large unlabeled dataset.
- Student Models: These pseudo-labels are used to train:
  - A smaller LLM (Gemma 3 4B).
  - Lightweight vision models (EfficientNetV2-M, SwinV2-B, ResNet-50, MobileNetV3).
- Objective: To transfer the reasoning capabilities of the large model to smaller, faster architectures without requiring additional human annotation.

C. Extended Attribute Extraction

Beyond overall condition, the study investigates the ability of closed-source proprietary MLLMs (GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6) to extract 12 distinct attributes (e.g., Architectural Era, Safety, Walkability, Energy Efficiency) using a multiple-choice QA format on zoomed-out street-view images.

D. Visualization

A Visualization Dashboard was developed to integrate model outputs, allowing homeowners and stakeholders to view property assessments, neighborhood context, and detailed audit reports.

3. Key Contributions

Novel Framework: A scalable pipeline for nationwide building condition assessment using MLLMs, moving beyond simple classification to holistic, criteria-based reasoning.
Human-AI Alignment: A rigorous study demonstrating that fine-tuned MLLMs can achieve alignment with human experts that surpasses individual raters.
Efficiency via Distillation: A successful demonstration of distilling complex MLLM reasoning into lightweight CNNs and Transformers, achieving a 30× speedup while maintaining comparable accuracy.
Comprehensive Attribute Analysis: An evaluation of MLLMs on a broad set of built-environment attributes, analyzing stability, inter-rater reliability, and alignment with human experts.
Open-Source Deployment: Identification of Gemma 3 27B as the most practical open-source model for single-GPU deployment, balancing performance and resource constraints.

4. Key Results

Performance Metrics (SRCC & PLCC)

Zero-Shot Baseline: Among open-source models, Gemma 3 27B achieved the best balance, with an SRCC of 0.77 and PLCC of 0.77 on a single GPU, outperforming larger models like LLaMA 4 Scout (which required 2 GPUs) and smaller models like Gemma 3 4B (SRCC 0.45).
Fine-Tuning: Fine-tuning Gemma 3 27B on just 500 human-labeled images improved performance to SRCC 0.83 and PLCC 0.82, surpassing the average agreement of human raters (SRCC 0.78).
Knowledge Distillation:
- Gemma 3 4B (Distilled): Achieved SRCC 0.81 (surpassing the base 27B) with a 3× speedup.
- Vision Models (EfficientNetV2-M/SwinV2-B): Achieved SRCC 0.73 with a 30× speedup compared to the LLM, demonstrating that lightweight models can effectively learn the "reasoning" of the LLM via pseudo-labels.

Human-AI Alignment Study

Stability: Claude-Opus-4.6 showed the highest internal stability (0.941) across repeated trials, while GPT-5.4 was the least stable.
Human Alignment: Gemini-3.1-Pro achieved the highest alignment with human expert judgments (Pearson $r$ = 0.844), suggesting that high internal consistency does not always equate to better human alignment.
Reliability: LLMs demonstrated significantly higher inter-rater reliability (Krippendorff's $\alpha$ > 0.94) compared to human experts ( $\alpha$ = 0.635), indicating LLMs are more deterministic and less subjective.

5. Significance and Impact

Scalability: The framework enables the assessment of millions of buildings with minimal human labeling effort, making nationwide retrofit planning feasible.
Cost-Effectiveness: By distilling knowledge into lightweight models, the system can run on consumer-grade hardware or at scale with low inference costs.
Data-Driven Policy: The generated data supports evidence-based decision-making for homeowners, researchers, and local agencies regarding energy efficiency, safety, and housing equity.
Future Directions: The authors identify future work in exploring prompt engineering strategies, addressing potential biases in LLM outputs (e.g., architectural style bias), and expanding the dashboard into a full decision-support system.

In conclusion, this paper establishes that fine-tuned Multimodal LLMs, when combined with knowledge distillation, offer a robust, efficient, and highly accurate alternative to manual building inspections, bridging the gap between advanced AI reasoning and practical, large-scale urban analysis.