💻 computer science

Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery

This paper presents a scalable multi-modal building inspection framework that fuses satellite and street-level imagery using a Perceiver IO architecture with a shared DINOv2 backbone, demonstrating that an RGB-M masking strategy and flexible token-based fusion significantly improve the detection of street-level roof attributes while accommodating variable input views across a large-scale, ten-country dataset.

Original authors: Niels Sombekke, Rob G. J. Wijnhoven, Martin R. Oswald

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Niels Sombekke, Rob G. J. Wijnhoven, Martin R. Oswald

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a house to a friend who has never seen it. If you only show them a photo taken from a drone (satellite), they can see the shape of the roof and what's around it, but they can't tell if the shingles are slate or tile, or if there's a dormer window sticking out. If you only show them photos taken from the street, they can see the texture of the walls and the roof edge, but they miss the big picture of the roof's layout.

This paper presents a new "super-observer" that combines both views to get the full story. Here is how they did it, explained simply:

The Problem: Too Many Views, Too Many Angles

The researchers wanted to build a system that could automatically inspect thousands of buildings to check their roofs. They had two types of data:

Satellite photos: Looking down from space.
Street-level photos: Looking up from the ground (like Google Street View).

The tricky part was that some buildings have dozens of street photos, some have none, and some are blocked by trees or cars. Traditional computer programs struggle with this "messy" variety. They usually force everything into a neat, fixed box, which throws away important details or gets confused when the input changes.

The Solution: The "Perceiver IO" (The Smart Translator)

The team built a new AI architecture called Perceiver IO. Think of this AI as a smart translator who can listen to a conversation with any number of people (from 0 to 8 street views) and a single narrator (the satellite view), and then summarize the whole thing into a single, perfect report.

Instead of squashing all the photos into a single blurry summary (which is what older methods do), this AI keeps the fine details of every little patch of the image. It uses a "shared brain" (a model called DINOv2) that has already learned to recognize patterns in millions of images, and then teaches it specifically to look at buildings.

The Secret Sauce: The "Ghost Outline" (RGB-M Masking)

One of the biggest challenges is telling the AI which part of the photo is the house and which part is the neighbor's tree or a passing car.

The researchers tested four ways to show the AI the house:

Show the whole photo: The AI gets distracted by the background.
Crop it: Cut out everything except the house. This works, but it removes the context (like the street style) that helps the AI guess the materials.
Inverted crop: Show only the background. (This proved the background actually has useful clues, but isn't enough on its own).
The "Ghost Outline" (RGB-M): This was the winner. They took the photo and added a fourth "channel" (like adding a new color layer) that acts as a transparent ghost outline of the building.

The Analogy: Imagine looking at a house through a window.

Hard cropping is like taping a piece of paper over the window so you can only see the house, but you lose the view of the street.
The Ghost Outline is like having a smart friend stand next to you and point a laser pointer at the house while you look through the window. You see the house clearly and you can still see the street context around it. The AI learned that this "laser pointer" method was the best way to focus without losing context.

What They Found

They tested this on a massive dataset of over 32,000 buildings across 10 countries. Here is what happened:

The "Street" Advantage: For things you can only see from the ground (like the texture of slate tiles or the shape of a dormer window), the combined AI was much better than the satellite-only AI. For example, identifying "slate" roofs improved by a huge margin because the street view showed the texture clearly.
The "Sky" Advantage: For things you can only see from above (like the overall shape of a skylight or the color of the whole roof), the satellite-only AI was still slightly better. Adding street views sometimes confused the AI for these specific items.
The Best Overall: The combined "Perceiver IO" system was the most flexible. It didn't matter if a building had 8 photos or 0 photos; the system handled it gracefully without breaking.

The Takeaway

The paper concludes that while a satellite view is great for a general overview, combining it with street views creates a much richer, more accurate picture of a building's condition. The key to making this work was using a flexible AI architecture that doesn't force data into a fixed shape and using a "ghost outline" technique to help the AI focus on the building without ignoring its surroundings.

This creates a scalable way to inspect buildings automatically, which is a big step forward for maintaining our cities without needing humans to visit every single house.

Technical Summary: Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery

Problem Statement
Maintaining and insuring the built environment at a national scale requires detailed knowledge of individual building attributes, such as roof materials and structural elements. Traditional manual inspections are costly, slow, and generate significant carbon emissions. While remote sensing offers a scalable alternative, relying solely on satellite imagery limits analysis to attributes visible from above, missing critical facade-level details. Conversely, street-level imagery provides close-up views of facades and materials but suffers from occlusion, variable coverage, and inconsistent camera angles. The core challenge lies in fusing these heterogeneous modalities—satellite and street-level imagery—into a unified framework that can handle variable numbers of street-level inputs per building, isolate the target structure from its surroundings without losing contextual cues, and jointly predict multi-label roof attributes.

Methodology
The authors propose a multi-modal classification framework centered on a Perceiver IO architecture operating on spatial patch tokens derived from a shared DINOv2 backbone.

Dataset Construction: A large-scale dataset was constructed comprising 32,135 buildings (61,672 roof segments) across ten countries. Each segment is paired with a single satellite image and a variable number of street-level views (averaging 4.54 per segment, up to 8 sampled during training). The dataset supports two multi-label tasks: identifying six roof elements (e.g., solar panels, dormers) and seven roof materials (e.g., slate, bitumen).
Masking Strategies: To isolate the target building, four strategies were evaluated: (1) full image, (2) hard cropping (zeroing background), (3) inverted cropping (zeroing the building), and (4) RGB-M, where the binary building footprint mask is appended as a fourth input channel. The RGB-M strategy was found to provide a "soft spatial prior," allowing the model to focus on the building while retaining environmental context, outperforming hard cropping which disrupts pre-trained patch statistics.
Backbone and Fine-tuning: The study compares ResNet-50 and DINOv2-S. Results indicate that fully fine-tuning DINOv2-S significantly outperforms linear probing and partial fine-tuning, suggesting that the domain shift from natural images to aerial/street-level building imagery requires updating early feature layers.
Fusion Architecture: Three fusion strategies were compared:
1. Concatenation: Merging global feature vectors from satellite and max-pooled street-level views.
2. Feature Vector Transformer: Processing a sequence of global vectors with a transformer encoder.
3. Perceiver IO: The proposed method processes raw spatial patch tokens from both modalities. It uses cross-attention to compress a variable-length sequence of tokens (from one satellite image and $N$ street-level images) into a fixed-size latent representation. This preserves fine-grained patch-level information and naturally handles variable input counts without padding or truncation.

Key Results

Masking: The RGB-M strategy consistently yielded the best performance across both modalities and backbones, achieving 0.939 mAP for roof elements and 0.729 for roof materials on satellite imagery with DINOv2-S.
Fusion Performance: The Perceiver IO fusion model outperformed both concatenation and Feature Vector Transformer baselines. While the satellite-only baseline retained a slight advantage in macro-averaged mAP (due to classes like skylights and bitumen being best viewed from above), the Perceiver IO model delivered substantial per-class gains for attributes visible from the street. Notable improvements included +11.3 AP for slate, +4.5 AP for glass, and +1.3 AP for dormers.
Architecture Sensitivity: The Perceiver IO performance was found to be more sensitive to latent dimensionality ( $D_z$ ) than the number of latent vectors ( $N_z$ ). A configuration of $N_z=128$ and $D_z=512$ provided the best trade-off.
Attention Analysis: Grad-CAM visualizations confirmed that the RGB-M strategy produces balanced attention patterns, focusing on the building while utilizing boundary context. Conversely, for classes like chimneys, attention was sometimes diffuse, suggesting the model may rely on contextual shortcuts rather than direct localization.

Significance and Claims
The paper establishes a scalable, flexible architecture for multi-modal building inspection capable of accommodating heterogeneous inputs and multiple output tasks. The authors claim that their work differs from prior art by fusing at the spatial token level rather than the feature vector level, thereby preserving fine-grained details. Furthermore, the use of Perceiver IO allows for the natural handling of variable numbers of street-level views without performance degradation.

The study concludes that multi-modal fusion is most valuable when modalities provide genuinely complementary views. While the satellite-only model remains superior for macro-averaged metrics due to the prevalence of top-down visible classes, the fusion approach significantly enhances the detection of street-level specific attributes. The authors suggest future work could involve encoding camera metadata as geometric embeddings, utilizing the token-based architecture for patch-level masking, and extending the framework to multi-task learning (segmentation and regression).