💻 computer science

BuildMamba: A Visual State-Space Based Model for Multi-Task Building Segmentation and Height Estimation from Satellite Images

BuildMamba is a unified multi-task framework leveraging visual state-space models to achieve state-of-the-art building segmentation and height estimation from satellite imagery by introducing a Mamba Attention Module, a Spatial-Aware Mamba-FPN, and a Mask-Aware Height Refinement module to overcome boundary bleeding and height underestimation while ensuring computational efficiency.

Original authors: Sinan U. Ulu, A. Enes Doruk, I. Can Yagmur, Bahadir K. Gunturk, Oguz Hanoglu, Hasan F. Ates

Published 2026-03-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sinan U. Ulu, A. Enes Doruk, I. Can Yagmur, Bahadir K. Gunturk, Oguz Hanoglu, Hasan F. Ates

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a city from a very high altitude, like a bird flying straight down. From this view, skyscrapers and small houses all look like flat, colorful shapes on a map. They look like 2D drawings. But if you wanted to build a 3D model of this city for urban planning, you would need to know two things: where the buildings are (their footprints) and how tall they are.

This is a tricky puzzle for computers. A single photo doesn't have depth clues like our eyes do. It's like trying to guess the height of a stack of books just by looking at the top of the stack from above.

This paper introduces a new computer program called BuildMamba that solves this puzzle. Here is how it works, explained simply:

The Problem: The "Flat" View

Current computer programs try to guess building heights from satellite photos, but they often make mistakes.

The "Bleeding" Effect: They often blur the edges, making a tall building look like it's melting into the ground.
The "Short" Guess: They tend to underestimate how tall skyscrapers are, thinking they are shorter than they really are.
The "Heavy" Cost: To see the whole city context (like how one building relates to another far away), traditional computers have to do a massive amount of math, which is slow and expensive.

The Solution: BuildMamba

The authors built a new system called BuildMamba. Think of it as a super-smart detective that looks at a satellite photo and draws a perfect 3D map of the city. It does this by combining two jobs at once: finding the building outlines and guessing the height.

Here are the three special "tools" (modules) inside BuildMamba that make it so good:

1. The "Mamba Attention Module" (The Spotlight)

Imagine you are in a dark room full of furniture. You need to find a specific chair. Instead of looking at the whole room at once, you shine a flashlight on the most important parts.

How it works: This module acts like a dynamic spotlight. It scans the image and says, "Hey, this part of the roof is important, let's focus there," while ignoring the boring, empty sky. It helps the computer pay attention to the right details without getting confused by shadows or weird roof textures.

2. The "Spatial-Aware Mamba-FPN" (The Multi-Tool)

Usually, computers look at an image in two ways: zoomed in (to see small details like a window) or zoomed out (to see the whole neighborhood). Often, these two views don't talk to each other well.

How it works: This tool is like a Swiss Army knife that can switch between zoom levels instantly. It uses a special "state-space" technology (called Mamba) that is very fast at connecting the small details with the big picture. It's like having a team of detectives where one looks at the bricks and another looks at the skyline, and they instantly share notes to build a complete picture. This allows the system to handle both small houses and huge skyscrapers accurately.

3. The "Mask-Aware Height Refinement" (The Editor)

Sometimes, the computer guesses the height but gets a little "noisy"—like a radio with static, or a 3D model with jagged, bumpy surfaces.

How it works: This module acts as a strict editor. It looks at the "building outline" the computer already drew (the mask) and says, "Okay, if this is a building, the height should be smooth here. If the outline says it's a road, the height should be zero." It uses the shape of the building to clean up the height guess, removing the "static" and making the edges sharp and clean.

The Results: Why It Matters

The authors tested BuildMamba on three different sets of satellite images from different cities around the world.

It's Faster and Smarter: Unlike older methods that get confused by tall buildings or complex city layouts, BuildMamba kept its cool.
Better Accuracy: It guessed building heights much more accurately than the previous best methods. For example, on one test, it reduced the error by nearly a meter compared to the competition.
No Extra Help Needed: Many other systems need extra data (like 3D laser scans) to work well. BuildMamba only needs a standard satellite photo (RGB), making it easier to use anywhere in the world.

The Bottom Line

BuildMamba is like giving a computer a pair of "3D glasses" just by showing it a flat photo. By using a new, efficient way of looking at the whole city at once (the Mamba technology) and adding smart tools to focus on details and clean up mistakes, it creates a much more accurate and reliable 3D map of our cities than ever before. This helps us understand our urban environments better without needing expensive or hard-to-get extra data.

Technical Summary of "BuildMamba: A Visual State-Space Based Model for Multi-Task Building Segmentation and Height Estimation from Satellite Images"

Problem Statement
Accurate building segmentation and height estimation from single-view RGB satellite imagery are critical for urban analytics, including planning, infrastructure monitoring, and risk assessment. However, this task is inherently ill-posed due to the near top-down imaging geometry, which obscures 3D structures, and the reliance on indirect appearance cues like texture and shadows. Existing approaches, which often adapt monocular depth architectures, suffer from several limitations:

Structural Weakness: They frequently rely on implicit correlations between image texture and height, lacking explicit structural understanding of building shapes.
Artifacts: Common issues include boundary bleeding, systematic underestimation of high-rise structures, and sensitivity to homogeneous rooftop textures and illumination changes.
Dependency on Auxiliary Data: Many robust methods require auxiliary modalities (e.g., SAR, LiDAR, or DSMs) or auxiliary supervision (e.g., land-use masks) that are often unavailable during inference, limiting scalability in RGB-only settings.
Computational Trade-offs: Traditional global modeling via Transformers incurs quadratic computational complexity, while CNNs struggle with long-range dependencies.

Methodology: BuildMamba
The authors propose BuildMamba, a unified multi-task framework designed to jointly perform building segmentation and height estimation using only RGB input. The architecture is built upon a visual state-space backbone (VMamba) to balance global context modeling with computational efficiency. The framework consists of four primary components:

Visual State-Space Backbone (VMamba):
Instead of standard attention mechanisms, the model utilizes Selective State-Space Models (S6) generalized for 2D visual data. This employs a Cross-Scan Module (CSM) that unfolds image patches into four distinct sequences traversing the feature map in different directions. This allows the model to capture long-range global dependencies with linear computational complexity ( $O(N)$ ) while preserving spatial locality, addressing the efficiency bottlenecks of Transformers.
Mamba Attention Module (MAM):
Integrated at the output of each encoder stage, MAM dynamically recalibrates spatial importance. It processes features along height and width dimensions via adaptive average pooling, generating an attention map that emphasizes informative regions and suppresses noise. This enhances the fusion of local and global features extracted by the dual-path encoder.
Spatial-Aware Mamba-FPN (S-MambaFPN):
To address the limitations of standard Feature Pyramid Networks (FPN) in propagating long-range dependencies, the authors replace the refinement operator with a gated state-space scan. This module combines a baseline Mamba branch with a spatial-aware scanning branch that uses depthwise convolutions (3×3 and 7×7) to capture local and mid-range cues. The outputs are combined via element-wise multiplication, allowing the network to fuse spatial evidence with long-range context at each pyramid level.
Mask-Aware Height Refinement (MHR):
This module exploits the asymmetry between segmentation and height estimation, where segmentation masks often converge faster and provide more reliable structural boundaries. The MHR module treats the segmentation mask as a structure-aware confidence prior. It takes the raw height prediction and the segmentation mask as a two-channel input, predicting a residual height correction. A gated update function modulates this residual based on segmentation confidence, effectively suppressing height artifacts (speckle noise) in low-texture regions and sharpening boundaries without collapsing height estimation into a trivial function of the mask.

Loss Function
The framework is trained with a composite objective function:

Segmentation Loss: A hybrid of Cross-Entropy, Dice loss (to handle class imbalance), and a boundary-aware term using Laplacian edge maps to enforce structural consistency.
Regression Loss: Huber loss is used for height estimation to handle outliers and the long-tailed distribution of building heights, ensuring differentiability while preventing extreme values from destabilizing gradients.

Key Contributions
The paper claims the following contributions:

Unified RGB-Only Framework: A multi-task model that explicitly couples segmentation and height regression through structural constraints, eliminating the need for auxiliary modalities like DSMs or SAR.
Novel Architectural Modules: The integration of the Mamba Attention Module (MAM) and the Spatial-Aware Mamba-FPN (S-MambaFPN) to enhance global-local feature interaction and multi-scale aggregation.
Mask-Aware Refinement: A specific module that uses segmentation outputs as confidence priors to clean regression artifacts and correct height residuals.
Boundary-Aware Strategy: A loss strategy that emphasizes height discontinuities along building edges to reduce boundary smoothing.

Experimental Results
BuildMamba was evaluated on three benchmarks: DFC19, DFC23, and the challenging Huawei BHE dataset (which features diverse urban environments, mislabeling, and missing data).

Segmentation Performance: BuildMamba achieved state-of-the-art IoU scores of 0.90 on DFC19 and 0.93 on DFC23. On the complex Huawei BHE dataset, it achieved an IoU of 0.60, outperforming the second-best model (UANet) by 7.0%.
Height Estimation Performance: The model achieved an RMSE of 1.06 m on DFC19 and 1.77 m on DFC23. On the Huawei BHE dataset, it achieved an RMSE of 9.23 m and a $\delta_1$ accuracy of 0.806.
Comparison: The model consistently outperformed baselines such as UANet, FarSeg++, DSMNet, and HTC-DC. Notably, it surpassed the previous best RGB-only method (HTC-DC) by approximately 31.6% in RMSE on DFC23.
Ablation Studies: Experiments confirmed that each component (VMamba backbone, MAM, S-MambaFPN, and MHR) contributes incrementally to performance. The Mamba backbone was shown to offer a better accuracy-complexity trade-off compared to ResNet-101 and Swin-S backbones.

Significance and Claims
The authors position BuildMamba as a solution to the specific challenges of monocular satellite-based height estimation, particularly the trade-off between global context and computational efficiency. The paper claims that by leveraging visual state-space models, BuildMamba establishes a new performance upper bound for RGB-only methods. The significance lies in its ability to provide robust, structurally consistent predictions in diverse and weakly labeled real-world scenarios without relying on auxiliary data, thereby enabling scalable 3D urban reconstruction from single-view imagery. The authors note that this work is a significantly extended version of their previous preliminary framework, offering a more rigorous analysis of scalability and robustness across multiple datasets.