Using street view images and visual LLMs to predict heritage values for governance support: Risks, ethics, and policy implications

Technical Summary: Using Street View Images and Visual LLMs to Predict Heritage Values for Governance Support

Problem Statement
The implementation of the European Union's Energy Performance of Buildings Directive (EPBD) requires member states to develop National Building Renovation Plans by 2026. These plans necessitate a comprehensive overview of the national building stock, including buildings with heritage values, to guide exemptions or adapted energy requirements. In Sweden, a significant barrier exists: there is no comprehensive national register of buildings with heritage values. Existing records are fragmented, relying on official listings under the Historic Environment Act (covering ~0.15% of the stock) and incomplete municipal lists under the Planning and Building Act. Furthermore, existing data often lacks reliability or granularity. The Swedish National Heritage Board sought to utilize novel tools to fill these data gaps efficiently, specifically exploring the use of visual Large Language Models (LLMs) to assess heritage indicators from street-level imagery at a national scale.

Methodology
The study employed a zero-shot prompting approach using OpenAI's multimodal model (gpt-4o-2024-08-06) to analyze street view images of the Swedish building stock.

• Data Acquisition and Processing: The analysis began with 272,174 Energy Performance Certificates (EPCs) linked to the national property register. Through geospatial processing (matching building footprints to road centerlines and calculating sightlines), the authors retrieved 281,018 usable Google Street View images representing 186,038 unique buildings. After filtering for visibility and quality, 154,710 unique EPCs were successfully assessed.
• Prompt Engineering: Instead of fine-tuning the model, the researchers utilized zero-shot prompting. The prompt was designed based on the Swedish National Heritage Board's and Boverket's official checklists for ocular building surveys. The model was assigned a persona and instructed to output structured data in JSON format, identifying specific façade features (e.g., construction year, number of floors, window panes, materials) and assigning a "predicted heritage indicator" score (1–100).
• Validation and Governance Integration: To mitigate the risks of LLM hallucinations and bias, the study incorporated a human-in-the-loop validation process. Expert workshops were held with heritage specialists from the Swedish National Heritage Board and Uppsala University. A web-based mapping tool was developed to visualize the predicted scores alongside façade images for expert review. The final decision was to use the LLM output not for individual building certification, but as a statistical indicator for the building stock level to inform the EPBD implementation.

Key Results

• Predictive Performance: The LLM demonstrated varying degrees of accuracy. It successfully identified well-documented landmarks (e.g., the Turning Torso) and showed reasonable alignment with Energy Performance Certificate data regarding the number of floors. However, it struggled with construction year predictions, showing a tendency to overestimate extreme ages (e.g., predicting 1000-year-old buildings) and underrepresent specific decades (e.g., 1910s, 1940s).
• Bias and Representativeness: The study identified significant biases in the model's outputs.
  • Geographic and Socio-economic Bias: Buildings in Stockholm and in affluent areas (higher average disposable income) received significantly higher heritage indicator scores compared to similar buildings in other regions or lower-income areas.
  • Material Bias: The model assigned lower heritage scores to wooden buildings, which are common in modest, everyday environments, while favoring stone and brick structures.
  • Conservative Approach: The model exhibited a conservative bias, rarely assigning extreme high scores (90+). Consequently, the researchers set a threshold of 50 to identify "predicted heritage value," which captured only an additional 1–2% of the building stock beyond officially protected categories.
• Application in EPBD: Based on the expert validation and the statistical distribution of scores, the Swedish National Heritage Board adopted the LLM-derived indicators for the EPBD process. The method allowed the inclusion of an additional 5.0 million square meters of heated floor area into the heritage consideration category, filling a critical data gap for national planning.

Key Contributions

1. Governance Framework for AI: The paper provides a concrete case study of how public authorities can integrate zero-shot LLM predictions into policy-making. It demonstrates a workflow where AI handles the "menial" task of data extraction from vast image datasets, while human experts retain oversight for validation and strategic decision-making.
2. Methodological Transparency: By using structured prompts based on existing official checklists and requiring JSON outputs, the authors mitigated the risks of "sycophancy" (the model agreeing with the user) and made the results more interpretable and auditable than unstructured text generation.
3. Bias Identification: The study explicitly documents the socio-economic and geographic biases inherent in current multimodal LLMs when applied to heritage assessment, highlighting that these models tend to reproduce existing societal valuations rather than critically re-evaluating them.
4. Scalability: The research proves that zero-shot prompting can be scaled to analyze hundreds of thousands of buildings without the need for expensive, region-specific training data, offering a rapid, albeit imperfect, solution for data-scarce environments.

Significance and Claims
The paper claims that while zero-shot LLM predictions cannot replace expert heritage assessment or serve as a definitive legal register for individual buildings, they offer a valuable, efficient tool for building stock-level analysis. The primary significance lies in enabling authorities to generate a "first pass" overview of heritage indicators across the entire national building stock, which was previously impossible due to data fragmentation and resource constraints.

The authors emphasize that the value of this approach is not in the absolute accuracy of every single prediction, but in its ability to support the development of National Building Renovation Plans by identifying potential heritage assets that would otherwise remain invisible in official registers. However, the paper concludes with a strong caution: these tools must be used with critical oversight, as they inherently reproduce biases regarding wealth, location, and architectural style. The method is presented as a supplementary decision-support mechanism that reduces the burden on heritage experts, allowing them to focus on strategic tasks and complex case-by-case evaluations, rather than as a replacement for human judgment in heritage governance.