HeatPrompt: Zero-Shot Vision-Language Modeling of Urban Heat Demand from Satellite Images

This paper introduces HeatPrompt, a zero-shot vision-language framework that leverages pretrained Large Vision Language Models to extract thermal-relevant features from satellite images and generate accurate heat-demand estimates, significantly outperforming baseline models in data-scarce urban environments.

The Gist

Imagine you are a city planner trying to figure out how much heating a neighborhood needs to stay warm in winter. Usually, to do this accurately, you need a massive spreadsheet for every single building: "Built in 1950, brick walls, thin windows, old boiler."

But here's the problem: That spreadsheet doesn't exist. Most cities don't have that level of detail, and privacy laws often stop them from collecting it.

This is where a new tool called HeatPrompt comes in. Think of it as a "Super-Intelligent Satellite Detective" that can look at a photo of a neighborhood and guess the heating needs without ever needing a spreadsheet.

Here is how it works, broken down into simple concepts:

1. The Old Way vs. The New Way

• The Old Way (The "Blind Architect"): Imagine trying to guess how much fuel a house needs by only knowing its address and size. You might guess, but you'd be wrong a lot because you don't know if the roof is leaking or if the windows are single-pane glass. This is what current computer models do; they rely on incomplete data.
• The New Way (HeatPrompt): Instead of guessing, HeatPrompt uses a satellite photo and asks a super-smart AI (called a Vision-Language Model, or VLM) to act like a seasoned energy expert.

2. The "Magic Prompt"

The researchers didn't just feed the photo to the AI. They gave it a specific job description, like a boss giving instructions to a new employee.

They told the AI: "Look at this satellite image. Pretend you are an expert city energy planner. Describe what you see that would make a house hot or cold. Look for things like: Is the roof old and rusty? Is it covered in green plants? Are there many houses packed tightly together?"

The AI then writes a short "caption" or a list of observations, just like a human would.

• Example Output: "I see an old, red-tiled roof with no insulation. There are many houses packed close together, but very few trees to block the wind."

3. Turning Words into Numbers

Once the AI writes these descriptions, HeatPrompt turns those words into a mathematical code (a vector). It's like translating a poem into a secret number that a calculator can understand.

Then, a simple calculator (a mathematical model) takes these "word-numbers" and combines them with basic map data (like "how big is the area?") to predict the total heat demand.

4. Why is this a Big Deal?

The paper shows that this method is a game-changer for two main reasons:

• It's Much More Accurate: When they tested it, the new method was 93% better at predicting heat needs than the old "blind" methods. It reduced the error rate by 30%. It's like going from a weather forecast that's right half the time to one that's right almost every time.
• It's Transparent (No "Black Box"): Usually, AI models are like black boxes: you put data in, and a number comes out, but you don't know why. HeatPrompt is different. Because it uses the AI's written descriptions, a human planner can look at the result and say, "Ah, the AI predicted high heat demand because it saw old roofs and no trees. That makes sense!"

The "Detective" Analogy

Imagine you are trying to guess how much money a person spends on groceries.

• Method A (Old): You only know their zip code. You guess "Average."
• Method B (HeatPrompt): You look at a photo of their house. You see a big garden (maybe they grow their own food?), a fancy car (maybe they eat out?), and a large family (more mouths to feed). You use those visual clues to make a much smarter guess.

The Bottom Line

HeatPrompt is a tool that lets cities use satellite photos to "read" the energy needs of buildings, even when they don't have the official data files. It helps cities plan how to switch from fossil fuels to cleaner energy by knowing exactly where the heat is needed most, without needing to knock on every door to ask for permission.

It turns a blurry satellite image into a clear, actionable plan for a warmer, greener future.

Technical Summary

1. Problem Statement

Accurate heat demand mapping is essential for decarbonizing space heating and optimizing district heating networks. However, municipalities often lack the granular, building-level data required for precise calculations (e.g., construction year, insulation levels, heating systems) due to data incompleteness, outdated records, or privacy regulations.

• Limitations of Current Methods:
  • Bottom-up models: Rely on detailed building metadata which is often missing.
  • Top-down models: Disaggregate aggregate data but lack spatial precision.
  • Traditional ML: Often requires historical sensor data or partial building features, limiting applicability in data-scarce regions.
• The Gap: Converting raw satellite pixel data into quantitative energy metrics remains challenging. Existing remote sensing approaches struggle to capture complex semantic attributes (like roof age or vegetation density) that correlate with thermal load without extensive manual labeling.

2. Methodology: HeatPrompt

The authors propose HeatPrompt, a zero-shot vision-language framework that estimates annual heat demand by combining satellite imagery, basic GIS data, and semantic features extracted via Large Vision-Language Models (VLMs).

A. Data Acquisition and Preprocessing

• Source Data: Uses Esri World Imagery (RGB satellite images) overlaid with municipal heat planning isolines (multipolygons) from the Rheinland-Palatinate (RLP) Energy Atlas.
• Sample Construction:
  • Each sample consists of a $512 \times 512$ RGB image centered on a polygon's centroid.
  • A binary mask ($M_i$) is rasterized to highlight the region of interest (ROI).
  • The mask is inserted into the alpha channel to create an RGBA composite image.
  • Auxiliary Data: Enriched with OpenStreetMap (OSM) building footprints (type, height), LOD2 geometry, and Census data (age class, floor area).
• Dataset: 1,677 samples containing triplets $(I_i, M_i, y_i)$, where $y_i$ is the annual heat demand (MWh/a).

B. Semantic Feature Extraction (The Core Innovation)

Instead of training a CNN to predict heat demand directly from pixels, HeatPrompt uses a Zero-Shot VLM (specifically GPT-4o, though CLIP and Qwen2.5-VL were tested) as a "domain-specific energy planner."

1. Prompting: The RGBA composite image is fed to the VLM with a prompt instructing it to act as a municipal heat planner.
2. Extraction: The model identifies and lists the five most salient visual factors affecting heating demand (e.g., "roof age," "building density," "vegetation cover," "solar-ready surfaces") along with confidence scores.
3. Embedding: These textual captions are converted into fixed-length semantic vectors ($z^{sem}_i \in \mathbb{R}^{512}$) using Nomic text-embeddings.
4. Feature Fusion: The final feature vector combines:
   • GIS attributes ($A_i$, perimeter $\ell_i$).
   • Semantic embeddings ($z^{sem}_i$) from the VLM.
   • Structured building composition features (aggregated height, floor count, construction decade, building type ratios).

C. Regression Model

• Architecture: A Multi-Layer Perceptron (MLP) with two hidden layers (64 units each), dropout (0.2), and weight decay.
• Training: Trained for up to 250 epochs with early stopping.
• Validation: Stratified 5-fold cross-validation based on heat-demand quintiles to ensure distribution preservation.

3. Key Contributions

1. Zero-Shot Semantic Heat Mapping: The first framework to model annual heat demand using RGB satellite images and zero-shot semantic captions from VLMs, eliminating the need for manual labeling of energy-level features.
2. Interpretable Pipeline: Unlike "black-box" CNNs, HeatPrompt produces human-readable tokens (e.g., "old roof," "high density") that explain why a specific heat demand was predicted, offering transparent insights for urban planners.
3. Reproducible Benchmark: The authors released open-source code establishing the first vision-language benchmark for urban heat-demand estimation, including data preparation and evaluation protocols.

4. Experimental Results

The study compares HeatPrompt against a bottom-up simulation baseline and various machine learning models.

• Baseline Performance:
  • Physical simulation baseline: $R^2 = 0.32$, MAE = 287.1.
  • Best traditional ML (MLP on structured data only): $R^2 = 0.51$, MAE = 239.0.
• Impact of Semantic Features (Ablation Study):
  • Adding visual embeddings significantly improved performance across all models.
  • GPT-4o (HeatPrompt): Achieved the highest performance with $R^2 = 0.62$ and MAE = 200.7.
  • Improvement: This represents a 93.7% relative increase in $R^2$ and a 30% reduction in MAE compared to the physical simulation baseline.
  • Comparison: GPT-4o outperformed other vision backbones like CLIP ($R^2=0.60$) and EfficientNet-B0 ($R^2=0.58$).
• Qualitative Analysis:
  • The model successfully identified semantic trends: High-demand zones correlated with low vegetation, fewer solar panels, and higher structural density.
  • Differentiation: The model could distinguish between geometrically similar buildings with different visual characteristics (e.g., a modern roof with greenery vs. an old tiled roof with bare surroundings), assigning distinct heat demand values (100 vs. 150 kWh/m²) that physical models would miss.

5. Significance and Future Work

• Significance: HeatPrompt demonstrates that VLMs can serve as powerful "semantic proxies" for missing physical metadata. It enables accurate, interpretable, and automated heat demand estimation in data-scarce regions without requiring expensive field surveys or task-specific retraining of deep learning models.
• Future Directions:
  • Developing energy-aware visual-language embeddings specifically tailored to urban energy semantics (e.g., directly inferring insulation quality or PV coverage).
  • Exploring lightweight fine-tuning of promptable VLMs to focus the embedding space on energy-relevant factors.
  • Creating a reusable embedding layer for planners to query and visualize latent correlations between building morphology and thermal performance.

In conclusion, HeatPrompt bridges the gap between remote sensing and energy modeling by leveraging the reasoning capabilities of VLMs to extract actionable, interpretable features from satellite imagery, significantly outperforming traditional simulation and regression approaches.