GeoFormer: A Lightweight Swin Transformer for Joint Building Height and Footprint Estimation from Sentinel Imagery

This paper introduces GeoFormer, a lightweight Swin Transformer-based multi-task learning framework that jointly estimates building height and footprint from open-access Sentinel and DEM data with higher accuracy and efficiency than CNN baselines, achieving globally consistent results without region-specific fine-tuning.

The Gist

Imagine you are trying to draw a map of a city, but you don't have a street-level view. You only have a blurry, high-altitude photo taken from a satellite. Your goal is to guess two things for every patch of land: how tall the buildings are and how much of the ground they cover.

This is exactly what the researchers in this paper set out to do. They created a new AI tool called GeoFormer to solve this puzzle using free satellite data.

Here is the story of how they did it, explained simply:

1. The Problem: The "Pixel Soup"

Imagine looking at a city from a plane. If you zoom in too close (like looking at a single 10-meter square), you might see a mix of a roof, a tree, a shadow, and a road all squished together. It's like looking at a bowl of fruit salad and trying to guess the exact weight of just the strawberries. It's messy and confusing.

Most previous AI models tried to guess the height of buildings by looking at these tiny, messy squares. They often got it wrong because they couldn't see the "big picture" of the neighborhood.

2. The Solution: The "Neighborhood Watch" (100m Grid)

Instead of looking at tiny 10-meter squares, the researchers decided to look at 100-meter squares. Think of this as zooming out to look at an entire city block or a small neighborhood at once.

By looking at the whole block, the AI can ignore the messy details (like one specific tree or shadow) and focus on the average height and density of the buildings in that area. It's like judging the average height of people in a room by looking at the whole crowd, rather than trying to measure one person standing behind a pillar.

3. The Secret Sauce: The "Smart Eye" (Swin Transformer)

The researchers built a new type of AI brain called GeoFormer.

• Old AI (CNNs): Imagine an old AI that looks at a picture through a tiny, fixed window, moving it one step at a time. It's like a person with a tunnel vision who has to walk across a room to understand the whole scene.
• New AI (GeoFormer): This AI uses something called a Swin Transformer. Think of this as a person with smart, shifting eyes. It can look at a small detail, then instantly shift its focus to see how that detail connects to the wider neighborhood. It understands the "context" much better.

The researchers found that this "smart eye" approach was 7.5% more accurate than the old methods, but it was also 35 times smaller and lighter. It's like replacing a heavy, fuel-guzzling truck with a sleek, electric sports car that gets the job done faster and with less energy.

4. The Ingredients: What the AI Eats

To make its guesses, GeoFormer eats three specific types of "food" (data) that are free for everyone to use:

1. Sentinel-1 (The Radar): This is like a night-vision camera that can see through clouds and darkness. It bounces radio waves off buildings to see their shape.
2. Sentinel-2 (The Color Camera): This is a standard optical camera that sees colors. It helps the AI tell the difference between a concrete roof, a green park, or a red brick wall.
3. DEM (The Elevation Map): This is a 3D map of the ground itself. It tells the AI, "Is this building on a hill, or is the ground flat?" This is crucial for guessing height.

The Discovery: The researchers tested what happens if you remove one ingredient.

• If you take away the Color Camera, the AI gets very confused (accuracy drops by nearly 40%).
• If you take away the Elevation Map, the AI gets bad at guessing height (accuracy drops by 15%).
• If you take away the Radar, it gets slightly worse, but not terrible.
• Conclusion: The AI needs all three to work its best, but the Color Camera is the most important ingredient.

5. The "Fair Test" (GeoSplit)

One of the biggest problems in AI is "cheating." If you train an AI on a map of New York and then test it on a map of New York, it might just memorize the streets instead of learning how to guess heights.

To prevent this, the researchers used a clever trick called GeoSplit. Imagine cutting a pizza into 10 slices. They trained the AI on 8 slices, but they made sure the test slices were completely separate from the training slices. They didn't just pick random spots; they picked whole slices of the city. This ensured the AI was actually learning the rules of building heights, not just memorizing specific addresses.

6. The Results: A Global Map

The team tested their AI on 54 different cities across the world, from dense Asian megacities to European towns.

• Accuracy: They guessed building heights with an average error of only 3.19 meters (about 10 feet). That's incredibly accurate for a global map!
• Speed & Size: The model is so small it could run on a standard laptop, yet it outperformed much larger, older models.
• Real-World Test: They even tested it on a city in Turkey that was hit by a massive earthquake. Without being re-trained, the AI looked at the "before" and "after" satellite images and correctly predicted that the buildings had collapsed and the area was now empty. It saw the disaster without ever being taught about disasters.

Why Does This Matter?

This isn't just a cool science project. This data helps us:

• Predict Floods: Knowing how tall buildings are helps us model how water will flow through a city.
• Fight Climate Change: It helps us understand how cities trap heat (the "Urban Heat Island" effect).
• Plan for Disasters: If an earthquake hits, we can quickly estimate which areas are most at risk.

In short, GeoFormer is a lightweight, super-smart AI that uses free satellite photos to build a 3D map of the world's cities, helping us understand our planet better without needing expensive, secret data.

Technical Summary

1. Problem Statement

Accurate, globally consistent data on Building Height (BH) and Building Footprint (BF) is critical for urban climate modeling, disaster risk assessment, and population mapping. However, existing solutions face significant limitations:

• Data Scarcity: High-resolution 3D urban data is often unavailable, particularly in the Global South.
• Cost and Coverage: Methods using Very-High-Resolution (VHR) optical imagery, airborne LiDAR, or commercial SAR are too expensive or have limited spatial coverage for global deployment.
• Dependency: Many existing models rely on proprietary data, pre-existing vector footprints (e.g., OpenStreetMap), or auxiliary layers (cadastral maps) that are inconsistent globally.
• Resolution Trade-offs: While 10m resolution is common, it suffers from "sub-pixel contamination" in dense urban areas where a single pixel may straddle multiple buildings or shadows. Conversely, many global products operate at 100m–250m but often treat BH and BF as isolated tasks, ignoring their physical coupling.
• Generalization: Many models are trained and tested on single cities, failing to generalize across morphologically diverse urban environments.

2. Methodology

A. Data Strategy

• Input Data: The model utilizes only open-access, globally available data:
  • Sentinel-1 SAR: VV and VH polarizations (10m).
  • Sentinel-2 MSI: Red, Green, Blue, and Near-Infrared bands (10m).
  • DEM: SRTM Digital Elevation Model (30m).
• Target Resolution: The study adopts a 100m grid resolution. This choice is motivated by:
  1. Reducing sub-pixel mixing artifacts common at 10m in dense cities.
  2. Aligning with major global products (e.g., GHS-BUILT, WorldPop) and downstream applications (e.g., WUDAPT, mesoscale weather models).
  3. Enabling computationally feasible global processing.
• Reference Labels: Derived from the SHAFTS dataset using "Fishnet Analysis," which aggregates vector building inventories into 100m grid cells to calculate mean height ($H_{ave}$) and footprint ratio ($\lambda_p$).
• GeoSplit (Spatial Partitioning): To prevent data leakage caused by context windows overlapping between training and test sets, the authors employ a geo-blocked radial sector strategy. Each city is divided into 10 radial sectors, which are then allocated to training, validation, and test sets (8:1:1 ratio). This ensures strict spatial independence and captures both central and peripheral urban morphologies.

B. Model Architecture: GeoFormer

GeoFormer is a lightweight Swin Transformer-based multi-task learning framework.

• Input Processing: Multi-source data is fused into an 8-band tensor (S1, S2, DEM) plus a binary mask for valid regions.
• Backbone: Utilizes the Swin Transformer architecture, which processes the input as non-overlapping patches (e.g., 3×3, 5×5, 9×9 context windows). It employs Window-based Multi-head Self-Attention (W-MSA) and Shifted Window Attention (SW-MSA) to capture both local and global spatial dependencies efficiently.
• Multi-Task Heads:
  • BH Head: A lightweight MLP with ReLU activation to regress building height.
  • BF Head: A lightweight MLP with Sigmoid activation to constrain footprint ratio between 0 and 1.
• Loss Function: A multi-task loss combining Adaptive Huber Loss for both tasks, weighted by learnable task uncertainties ($\sigma_{bh}, \sigma_{bf}$) to balance the optimization of height and footprint.

3. Key Contributions

1. Novel Framework: Development of GeoFormer, a compact (0.32M parameters) Swin Transformer model that jointly estimates BH and BF at 100m resolution using only open-source data.
2. Superior Efficiency: Demonstrates that windowed local attention is more effective than convolution for scene-level parameter retrieval, achieving 7.5% better accuracy than the best CNN baseline (UNet) with 35× fewer parameters than ResNet-18.
3. Rigorous Evaluation: Introduced GeoSplit to ensure strict spatial independence across 54 morphologically diverse cities, addressing the common flaw of data leakage in receptive-field-based models.
4. Comprehensive Ablation: Systematically proved that:
   • A 5×5 (500m) receptive field is optimal.
   • DEM is indispensable for height estimation.
   • Multispectral reflectance is the dominant predictive signal.
5. Open Science: Public release of code, model weights, and the resulting global 100m BH/BF product.

4. Results

A. Performance Metrics

On a held-out test set of 54 cities:

• Building Height (BH): Achieved an RMSE of 3.19 m and $R^2$ of 0.66. This outperforms the best CNN baseline (UNet-MTL, RMSE 3.45 m) by 7.5%.
• Building Footprint (BF): Achieved an RMSE of 0.050 and $R^2$ of 0.80.
• Efficiency: GeoFormer (base) has only 0.32M parameters, significantly lighter than ResNet-18 (11.19M) and ConvNeXt-T (27.83M), while maintaining competitive inference times (~1.05 ms on RTX 3090).

B. Ablation Studies

• Receptive Field: The 5×5 window provided the best trade-off. Larger windows (9×9) led to over-smoothing and slightly degraded performance.
• Model Capacity: Increasing model size (Enlarged GeoFormer) led to overfitting (lower training error but higher test error), confirming the compact architecture is optimal for this data regime.
• Input Modalities:
  • Removing DEM caused the largest drop in BH accuracy (+15.0% RMSE), proving its necessity for vertical cues.
  • Removing Optical (Sentinel-2) caused the most severe performance collapse (+37.9% BH RMSE), indicating spectral data is the primary driver.
  • Removing SAR caused moderate degradation, highlighting its complementary role in capturing structural density.

C. Generalization and Transferability

• Cross-City Transfer: Tested on Suwon, South Korea (an unseen city with different morphology), GeoFormer achieved a BH RMSE of 3.57 m without fine-tuning, demonstrating robust spatial generalization.
• Zero-Shot Disaster Application: Applied to Kahramanmaraş, Turkey (post-2023 earthquake) without retraining. The model successfully detected large-scale structural collapse (reduced BH and BF) in the city core, validating its potential for rapid post-disaster assessment.

5. Significance

• Global Reproducibility: By relying solely on free Sentinel and DEM data, GeoFormer enables the generation of consistent 3D urban data for any location on Earth, including the Global South where proprietary data is scarce.
• Computational Feasibility: The lightweight nature of the model makes it suitable for routine global updates and deployment on standard hardware, overcoming the computational barriers of high-resolution global processing.
• Scientific Insight: The study establishes that for scene-level urban parameter retrieval, windowed attention mechanisms outperform traditional convolutions, and that multi-source fusion (SAR + Optical + DEM) is essential for robust estimation.
• Practical Application: The resulting global product supports critical downstream applications in climate modeling, flood risk assessment, and disaster response, filling a critical gap in global urban monitoring.