💻 computer science

ShadeBench: A Benchmark Dataset for Building Shade Simulation in Sustainable Society

This paper introduces ShadeBench, a comprehensive multimodal dataset and benchmark featuring geographically diverse urban scenes with simulated shade maps, satellite imagery, and 3D building data to facilitate scalable urban shade analysis, support downstream tasks like shade generation and segmentation, and advance data-driven research for heat-resilient urban planning.

Original authors: Longchao Da, Mithun Shivakoti, Xiangrui Liu, T Pranav Kutralingam, Yezhou Yang, Hua Wei

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Longchao Da, Mithun Shivakoti, Xiangrui Liu, T Pranav Kutralingam, Yezhou Yang, Hua Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to plan a city where people stay cool during a scorching summer. To do this, you need to know exactly where the shadows will fall from buildings at 8:00 AM, noon, or 6:00 PM. But right now, our "maps" for this are messy. Some are outdated, some ignore the fact that the sun moves differently in the Northern vs. Southern Hemisphere, and there's no standard way to test if a computer program is good at predicting these shadows.

Enter ShadeBench. Think of it as the "Olympics" or a "standardized driving test" for computers trying to understand and predict urban shadows.

Here is a simple breakdown of what the paper does:

1. The Problem: The "Ghost Building" Confusion

Imagine you are trying to draw a map of a city using an old sketch (OpenStreetMap data) and then comparing it to a fresh photo taken from space (satellite imagery).

The Issue: The old sketch might miss a new building, or the satellite photo might show a building that the sketch forgot. If you try to simulate shadows based on the old sketch, the shadows will fall in the wrong places because the "ghost" buildings aren't there, or real buildings are missing.
The ShadeBench Fix: The researchers built a "geometric aligner." Think of this as a smart editor that looks at the satellite photo and the old sketch side-by-side. If the photo shows a building that the sketch doesn't have, the editor either erases that building from the photo (masking it) or uses AI to "paint over" that spot with something that looks like a parking lot or grass. This ensures the computer is only simulating shadows for buildings that actually exist in both the data and the real world.

2. The Physics: The Sun's "Compass"

Most old shadow simulations act like the sun moves the same way everywhere on Earth.

The Issue: In the Northern Hemisphere (like Arizona), the sun moves across the southern sky. In the Southern Hemisphere (like Sydney), it moves across the northern sky. If your simulation ignores this, the shadows will rotate the wrong way.
The ShadeBench Fix: They used a real-world physics formula (NOAA Solar Position Algorithm) to tell the computer exactly where the sun is for any city, on any day, at any hour. This means the dataset correctly shows shadows moving clockwise in one hemisphere and counter-clockwise in the other, just like nature intended.

3. The Dataset: A "Time-Travel" Library

The result is a massive library of city scenes. For every single city block, the dataset provides:

The Real Photo: A satellite image of the city.
The Skeleton: A simple outline of the buildings.
The 3D Model: A digital 3D mesh of the buildings.
The Time-Travel Shadows: A series of images showing exactly what the shadows look like from 8:00 AM to 6:00 PM, every hour, for different seasons.
The Story: A text description explaining the sun's position (e.g., "It's 2:00 PM, the sun is high and to the south").

4. The Three "Tests" (Tasks)

The paper uses this library to test computers on three specific challenges:

Test A: Shadow Generation (The Artist)
- The Task: Give the computer a picture of a city and a text prompt like "It's 4:00 PM in July." Ask it to draw what the shadows will look like.
- The Result: They tested different AI models. The best one (DeepShade) was able to draw shadows that looked very realistic and followed the correct physics, while others made mistakes or looked blurry.
Test B: Shadow Segmentation (The Detective)
- The Task: Show the computer a picture of a city and ask it to circle only the shadowed areas.
- The Result: This is surprisingly hard for computers. Shadows are often faint and blend into the ground. The paper found that even advanced AI models struggle to find the exact edges of shadows, especially in the early morning or late afternoon when shadows are long and thin.
Test C: 3D Reconstruction (The Architect)
- The Task: Show the computer a 2D satellite photo and ask it to build a 3D model of the buildings.
- The Result: ShadeBench provides the "answer key" (the perfect 3D model) so researchers can measure exactly how close the computer's 3D guess is to reality.

Why This Matters

The authors aren't claiming this will immediately save lives or change city planning today. Instead, they are saying: "We finally built a fair, standardized playground where researchers can test their shadow-predicting tools."

Before this, everyone was using different maps and different rules, so you couldn't tell if one computer program was actually better than another. Now, with ShadeBench, researchers can run their programs through the same "driving test" to see who can best predict where the shade will be, helping us eventually design cooler, more comfortable cities.

Technical Summary: ShadeBench

Problem Statement

Urban heat exposure is intensifying due to the urban heat island effect, posing significant risks to public health and outdoor labor, particularly in regions like the southwestern United States. While shade is a critical mitigation factor, accurately modeling and analyzing urban shade at scale remains challenging. Existing approaches suffer from three primary limitations:

Data Misalignment: Synthetic datasets often rely on OpenStreetMap (OSM) data for ground truth, which can be outdated or incomplete. This creates geometric misalignment between simulated building layouts and real-world satellite imagery. Furthermore, prior datasets frequently neglect geographic variability in solar trajectories, failing to account for distinct diurnal and seasonal shade patterns between the Northern and Southern Hemispheres.
Lack of Standardized Evaluation: Simulation-based baselines are rarely assessed under unified protocols, making it difficult to rigorously compare the accuracy, robustness, and temporal dynamics of different modeling approaches.
Limited Downstream Support: The absence of standard benchmarks hinders the development of models for downstream tasks such as generative shade simulation, satellite-based shade segmentation, and 3D building reconstruction, which are essential for heat-resilient urban planning.

Methodology

The authors introduce ShadeBench, a comprehensive dataset and benchmark designed to address these gaps through a rigorous construction pipeline and physically grounded simulation.

Dataset Construction Pipeline

The pipeline integrates map-based priors with sensor-based observations to ensure physical realism and geometric consistency:

Data Sources: The dataset leverages OpenStreetMap (OSM) for building metadata (address, height, floors, roof shape) and satellite imagery for real-world observations.
Geometric Alignment: To resolve discrepancies between OSM-derived simulations and satellite imagery (e.g., missing or outdated buildings), the authors propose a Geometric Aligner. This module detects edge-based inconsistencies using Canny edge detection. It then employs two strategies to reconcile misalignment:
1. Mask-based Alignment: Suppressing satellite-observed structures that do not exist in the simulation.
2. Generative Refinement: Using a GenAI-based image editing model (e.g., Nano-banana) to inpaint inconsistent regions with semantically plausible non-building surfaces (e.g., parking lots), ensuring visual continuity.
Physically Grounded Solar Modeling: To capture hemispheric differences, the pipeline utilizes the NOAA Solar Position Algorithm (SPA). This calculates location-dependent solar elevation and azimuth angles based on latitude, longitude, date, and time. These parameters drive the directional light source in a Blender-based rendering environment, ensuring that shade evolution patterns (e.g., shadow rotation) correctly reflect the reversal of solar trajectories across the equator.

Dataset Composition

ShadeBench covers six continents, 16 countries, and 34 cities, featuring diverse urban layouts (dense vs. sparse) and traffic conditions. Each sample includes:

Aligned satellite images ( $X'_j$ ).
Building topology/skeleton images ( $X_i$ ).
Time-series simulated shade ground-truth images ( $X_{i,t}$ ) for various hours and seasons.
Structured textual descriptions ( $D_{i,t}$ ) encoding solar attributes (declination, elevation, time).
3D building meshes ( $M_i$ ) exported from Blender in Wavefront .OBJ format.

Benchmark Tasks and Evaluation

ShadeBench establishes standardized evaluation protocols for three downstream tasks:

Generative Shade Simulation: Evaluates models (Diffusion, ControlNet, DeepShade) on their ability to synthesize physically correct shade patterns given building geometry and solar conditions. Metrics include Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS).
Shade Segmentation: Assesses the ability of segmentation models (SAM, SAM2, GeoSAM) to delineate fine-grained, low-contrast shadow boundaries. Evaluations are conducted in two settings: on simulated images (idealized) and on real satellite images (realistic) at different times of day (6 AM, 12 PM, 6 PM). Metrics include Dice coefficient and Intersection over Union (IoU).
3D Building Reconstruction: Provides a testbed for recovering 3D building geometry from 2D observations using physically grounded supervision (Blender meshes). The authors propose using Chamfer Distance (CD) to quantitatively assess reconstruction quality.

Key Results

Generative Simulation: In experiments comparing Diffusion models, ControlNet, and DeepShade, DeepShade consistently outperformed baselines across both dense and sparse urban scenarios. It achieved higher structural similarity and lower perceptual distance, indicating superior capability in capturing fine-grained boundaries and temporal regularities.
Shade Segmentation: While foundation models (SAM, SAM2) performed well on simulated images (high Dice/IoU), their performance degraded significantly when applied to real satellite imagery, particularly during early morning or late afternoon when shadows are elongated and low-contrast. This highlights a substantial domain gap between idealized simulations and real-world visual conditions.
3D Reconstruction: The paper presents a proof-of-concept using SAM-3D to demonstrate that the dataset enables geometry-aware evaluation. The authors emphasize that the primary contribution here is the establishment of a unified benchmark for 3D reconstruction under realistic alignment and solar conditions, rather than the proposal of a new reconstruction model.

Significance and Claims

The authors position ShadeBench as a foundational resource for data-driven urban climate research and heat-resilient planning. The paper claims three primary contributions:

A Large-Scale, Aligned Dataset: It provides a rigorous construction pipeline that aligns simulation ground truth with real observations and incorporates location-dependent solar trajectories to ensure physically realistic dynamics across hemispheres.
Unified Evaluation Protocols: It establishes standardized benchmarks for comparing simulation-based shade modeling methods, enabling systematic assessment of accuracy, robustness, and temporal dynamics.
Support for Downstream Tasks: It facilitates research in shade generation, segmentation, and 3D reconstruction, addressing the previous lack of standard benchmarks for these specific urban analytics tasks.

The authors conclude that while generative models show promise, accurately simulating fine-grained, temporally consistent shade patterns remains a challenge, particularly under complex layouts and extreme illumination. They suggest that future work requires tighter integration of geometry, physics, and perception, and that ShadeBench serves as a necessary testbed for these advancements. The code and dataset are made publicly available to support reproducible research.