💻 computer science

Joint Instance Segmentation and Geometric Attribute Regression for Roof Structures in Aerial Imagery

This paper presents a method that extends Mask R-CNN to jointly perform instance-level roof segmentation and regress geometric attributes (height, slope, and azimuth) from single aerial images, utilizing a conditional azimuth loss and log-normalized height representation to achieve accurate 3D building model reconstruction without requiring 3D data at inference time.

Original authors: Luuk Versteeg, Rob G. J. Wijnhoven, Martin R. Oswald

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luuk Versteeg, Rob G. J. Wijnhoven, Martin R. Oswald

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, high-resolution photo of a city taken from a helicopter. Now, imagine you want to know the exact shape, height, and tilt of every single roof in that picture without ever leaving your chair or visiting the site. That is exactly what this paper describes: a smart computer program that looks at a single aerial photo and instantly "measures" the buildings it sees.

Here is a simple breakdown of how they did it, using some everyday analogies.

The Big Idea: The "One-Shot" Architect

Usually, to build a 3D model of a city, you need expensive laser scanners (LiDAR) flown over the area, or you need to visit every building with a tape measure. This paper says, "No need for all that expensive gear."

They built a system that learns from a few expensive 3D maps (specifically from the Netherlands) and then applies that knowledge to regular 2D photos. Think of it like a student who studies a few detailed blueprints in a library and then, when shown a simple sketch of a house, can accurately guess how tall it is and how steep the roof is. Once the student learns the lesson, they don't need the blueprints anymore; they just need the sketch.

How the System Works

The researchers took a standard computer vision tool (called Mask R-CNN, which is good at finding and outlining objects) and gave it a "superpower": a new brain branch dedicated to guessing numbers.

The Outline (Segmentation): First, the system draws a mask around every roof it sees, separating one house from its neighbor.
The Measurements (Regression): Then, for every single roof it outlined, it predicts three things:
- Height: How tall is the building?
- Slope: How steep is the roof? (Is it flat like a pancake or steep like a slide?)
- Azimuth: Which way is the roof facing? (North, South, East, or West?)

The Two "Cheat Codes" They Invented

The paper highlights two clever tricks they used to make the system much smarter than previous attempts.

1. The "Ignore the Flat Roof" Trick (Conditional Azimuth Loss)

The Problem: If you look at a flat roof from above, it has no "direction." It's just a flat surface. However, the data they used to train the AI (called 3DBAG) accidentally gave these flat roofs random compass directions (like "North" or "South") just because the computer had to pick something.
The Analogy: Imagine trying to teach a dog to point North. If you tell the dog "Point North" when it's sitting on a flat table, and "Point South" when it's on the same table five minutes later, the dog gets confused.
The Fix: The researchers told the AI: "If the roof is flat, ignore the compass direction completely." By stopping the AI from trying to learn from these confusing, random directions, the error in guessing roof direction dropped by nearly half.

2. The "Log-Book" Trick (Log-Normalized Height)

The Problem: Most houses are short (maybe 3 to 9 meters), but a few are skyscrapers or church towers (100+ meters). If you ask a computer to learn heights normally, it gets obsessed with the few giant towers and forgets the thousands of small houses. It's like a teacher grading a class where one student gets a 1000 and everyone else gets a 5; the teacher focuses only on the 1000.
The Analogy: Instead of measuring height in straight meters, they used a "logarithmic" scale. Think of it like the Richter scale for earthquakes: a small jump in the number represents a huge jump in reality. This squashes the giant numbers down so the computer pays equal attention to the small houses and the big towers.
The Result: This made the height predictions 17% more accurate.

How Good Is It?

The system was tested on over 17,000 aerial photos of Dutch cities. Here is how well it performed:

Roof Direction: It guessed the compass direction within about 7 degrees (roughly the width of your pinky finger held at arm's length).
Roof Steepness: It guessed the angle within about 4 degrees.
Height: It guessed the height within about 1 meter (roughly the height of a tall adult).
Finding Roofs: It successfully outlined about 56% of the roofs perfectly (a very strong score for this type of complex task).

The Catch (Limitations)

The paper is honest about where the system struggles:

Confusing Shapes: If a roof slopes inward from all sides (like a pyramid), the system sometimes gets confused about which way it's facing because it only has one photo.
Old Data: Sometimes the photo shows a new house, but the training data says there's an old barn there. The system might get penalized for being right because the "ground truth" is wrong.
No Teamwork: The system treats every roof as an island. It doesn't realize that two houses next to each other usually have similar heights. It guesses them independently, which can lead to weird inconsistencies.

The Bottom Line

This paper proves that you don't need expensive 3D lasers to get a decent 3D understanding of a city. If you have a high-quality aerial photo, this AI can build a simplified 3D model of the buildings, complete with height and roof angles. This is a big deal because aerial photos are available almost everywhere, making it possible to analyze building geometry for things like solar energy planning or insurance risk on a massive scale.

Technical Summary: Joint Instance Segmentation and Geometric Attribute Regression for Roof Structures in Aerial Imagery

Problem Statement
Accurate geometric knowledge of the built environment—specifically building height, roof slope, and orientation (azimuth)—is critical for sustainability planning, energy assessment, insurance risk evaluation, and housing management. However, acquiring this data at a national scale is challenging. While LiDAR provides accurate 3D models, it is expensive and infrequent. Manual inspection is not scalable. Existing learning-based methods often focus on single attributes (e.g., height only) or operate at the semantic level rather than the instance level, failing to distinguish multiple roof segments on a single building. Furthermore, many approaches require multi-view imagery or specific acquisition conditions that limit their applicability.

Methodology
The authors propose a method to jointly predict instance-level roof segment masks and three continuous geometric attributes (building height, roof slope, and roof azimuth) from a single aerial orthophoto.

Architecture: The approach extends the Mask R-CNN framework with a dedicated Attribute Regression Branch.
- Backbone: The model utilizes a DINOv3 ConvNeXt-Base backbone, which leverages self-supervised pretraining via knowledge distillation. This was found to significantly outperform ResNet-50 and EfficientNetV2-M, particularly in generating precise mask boundaries.
- Feature Extraction: A Feature Pyramid Network (FPN) extracts features. The attribute branch shares 7×7 RoI-aligned features with the classification and bounding box branches.
- Output: For each detected roof segment, the network outputs a binary mask, a bounding box, and three attributes: height ( $h$ ), slope angle ( $\alpha$ ), and azimuth ( $\phi$ ). The azimuth is encoded as a 2D unit vector $(\cos \phi, \sin \phi)$ to handle circular discontinuity.
Dataset: The model is trained and evaluated on a large-scale dataset of 17,353 Dutch aerial images (8 cm/pixel resolution). Ground truth is derived automatically from 3DBAG, a nationwide dataset of LoD2.2 building reconstructions combining cadastral footprints with airborne LiDAR (AHN). The dataset contains over 2 million instance annotations.
Key Technical Innovations:
1. Conditional Azimuth Loss: Flat roofs (slope $\le 15^\circ$ ) in the 3DBAG ground truth are assigned arbitrary azimuth values because a horizontal surface has no meaningful compass orientation. The authors introduce a gating mechanism that suppresses the azimuth loss for these flat segments. This prevents the model from learning noisy labels and reduces azimuth prediction error significantly.
2. Log-Normalized Height Representation: Building heights in the dataset are heavily right-skewed (most are <20m, with a long tail of tall structures). Direct regression causes tall buildings to dominate the gradient signal. The authors apply a log-normalization transformation ( $h_{norm} = (\log h - \mu)/\sigma$ ) to reshape the target distribution into a near-Gaussian form, balancing the gradient signal across all height ranges.
3. Two-Stage Training: The model is trained in two stages. First, only the Mask R-CNN components (detection and segmentation) are trained to ensure reliable instance matching. Second, the backbone is frozen, and the attribute regression branch is trained jointly with the other branches.

Results
Evaluated on a held-out test set with strict geographic separation, the method achieves the following performance metrics:

Instance Segmentation: An AP50 of 0.566 and a mean Average Precision (mAP) of 0.298.
Attribute Regression (Mean Absolute Error):
- Roof Slope: ~4.08°
- Azimuth: ~6.96° (improved to ~7° with conditional loss)
- Building Height: ~1.03 meters

Ablation Studies and Analysis:

Backbone: DINOv3 ConvNeXt-Base outperformed other backbones, achieving an mAP of 0.204 in the detection-only stage compared to 0.129 for ResNet-50.
Loss Design: The conditional azimuth loss reduced azimuth MAE by 41% (from ~21.8° to ~12.9°) without degrading other attributes.
Height Representation: The log-normalized representation improved height regression by 17% compared to direct regression on raw heights (reducing MAE from 1.878m to 1.556m in ablation, and to ~1.03m in the final model with higher resolution).
Resolution: Increasing input resolution from 512×512 to 768×768 consistently improved all metrics.
Per-Cluster Performance: The model performs best on medium-to-high steep roofs (match rates >69%). Performance degrades for very tall buildings and low-steep roofs, where visual cues are weaker or surroundings are complex.

Significance and Claims
The paper claims that this work is the first to jointly predict instance-level building masks with continuous regression of height, slope, and azimuth from a single 2D aerial image.

The primary significance lies in the scalability of the approach. By using expensive 3D reference data (LiDAR-derived 3DBAG) only for training, the resulting model can be deployed anywhere aerial imagery is available to infer 3D geometric attributes. This enables the reconstruction of simplified 3D building models (LoD2) without requiring multi-view input or LiDAR at inference time. The authors assert that this provides a scalable foundation for applications in energy planning, insurance risk assessment, and sustainable building management, addressing the gap where accurate geometric data is currently unavailable at the scale of national building stocks.

The authors also note limitations, particularly regarding the granularity of the 3DBAG ground truth (which can split visually coherent surfaces into multiple polygons) and temporal misalignments between imagery and 3D data, which may cause quantitative metrics to underrepresent the model's actual qualitative performance.