⚡ electrical engineering

A Controlled Benchmark of Visual State-Space Backbones with Domain-Shift and Boundary Analysis for Remote-Sensing Segmentation

This paper presents a strictly controlled benchmark of visual state-space models for remote-sensing segmentation, revealing that while they offer favorable accuracy-efficiency trade-offs, their performance is limited by modest scaling gains, asymmetric cross-domain generalization, and boundary delineation failures, suggesting future improvements should prioritize robustness and boundary-aware decoding over encoder scaling alone.

Original authors: Nichula Wasalathilaka, Dineth Perera, Oshadha Samarakoon, Buddhi Wijenayake, Roshan Godaliyadda, Vijitha Herath, Parakrama Ekanayake

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Nichula Wasalathilaka, Dineth Perera, Oshadha Samarakoon, Buddhi Wijenayake, Roshan Godaliyadda, Vijitha Herath, Parakrama Ekanayake

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to look at satellite photos of the Earth and draw perfect lines around things like roads, houses, forests, and lakes. This is called "semantic segmentation."

For a long time, we used two main types of "brains" (AI models) for this job:

The Local Detective (CNNs): Great at seeing small details like a brick wall or a pothole, but sometimes misses the big picture.
The Global Observer (Transformers): Great at understanding the whole city layout, but gets overwhelmed and slow when looking at high-resolution images.

Recently, a new type of brain called Visual State-Space Models (SSMs), or "Mamba" models, has arrived. They promise to be the best of both worlds: fast, efficient, and able to see both the trees and the forest. But there's a catch: nobody really knew if they were actually better because every researcher tested them differently (using different tools, different training methods, and different "decoders" to draw the final lines).

The Big Experiment: A "Controlled Kitchen"

The authors of this paper decided to stop guessing and start measuring. They built a "Controlled Benchmark."

Think of it like a cooking competition where every chef must use the exact same oven, the exact same ingredients, and the exact same recipe, but they are only allowed to change the chef's knife.

The Knife: The AI "Encoder" (the part that looks at the image). They tested three new "Mamba" knives (VMamba, MambaVision, Spatial-Mamba) against the old standard knives (CNNs and Transformers).
The Recipe: A fixed, lightweight decoder (the part that draws the lines).
The Ingredients: Two famous datasets: LoveDA (a mix of city and country scenes) and ISPRS Potsdam (super-high-resolution aerial photos).

The Three Big Discoveries

1. Bigger Isn't Always Better (The "Elephant vs. Mouse" Analogy)

The researchers tried making the Mamba models bigger and bigger, hoping they would get smarter.

The Result: It was like giving a mouse a bigger backpack. It didn't really help it run faster or jump higher.
The Lesson: Simply making the model larger (scaling up) only gave tiny improvements. A small, efficient Mamba model performed almost as well as a giant, expensive one.

2. The "City-to-Country" vs. "Country-to-City" Mystery

They tested how well the models could adapt when moving from one type of environment to another.

The Result: It was much easier to teach a model trained in the countryside to recognize cities than the other way around.
The Analogy: Imagine a farmer who learns to drive a tractor on dirt roads. If you put him in a city with traffic lights and tight turns, he might struggle a bit, but he understands the basics of driving. However, if you take a Formula 1 driver (trained only on smooth, predictable city tracks) and put them on a muddy, unpredictable farm field, they might crash immediately.
Why? Cities are very structured (straight lines, grids). Countryside is messy and varied. The "messy" training data teaches the AI to be more flexible and robust.

3. The "Blurry Edge" Problem

This was the most surprising finding. The researchers looked at where the AI made mistakes.

The Result: The AI was actually pretty good at identifying the middle of a building or a road. But it was terrible at drawing the edges.
The Analogy: Imagine a painter who is great at filling in the color of a house but keeps painting the roof slightly too big or the windows slightly too small. The "boundary" is where the AI gets confused.
The Cause: When the AI moves from a city dataset to a country dataset, the edges get "blurry" (due to different lighting, shadows, or how the map was drawn). The current AI models are like people trying to trace a shaky line with a heavy marker—they just can't get the fine details right.

The Final Verdict

The paper concludes that Visual State-Space Models (Mamba) are a great new tool. They are efficient and fast, often beating the old methods.

However, the secret to making them even better isn't just building bigger models. The future lies in:

Teaching the AI to be more robust when moving between different environments (like city to country).
Fixing the "Edge" problem. We need to improve the part of the system that draws the lines (the decoder) so it doesn't get confused by blurry boundaries.

In short: We found a new, efficient engine (Mamba), but to win the race, we need to fix the steering wheel (boundary handling) rather than just adding more horsepower (bigger models).

1. Problem Statement

Remote sensing (RS) semantic segmentation faces a dual challenge: capturing fine local structures (e.g., road boundaries, building edges) while integrating broad scene context from high-resolution imagery.

Current Limitations: Convolutional Neural Networks (CNNs) are efficient at local feature extraction but lack global context. Vision Transformers (ViTs) excel at global context but suffer from quadratic computational complexity, making them memory-intensive at high resolutions.
Emerging Alternative: Visual State-Space Models (SSMs), such as Mamba, offer linear scaling and efficient long-range modeling. However, their practical advantages over CNNs and Transformers in RS remain unclear.
The Gap: Existing studies often fail to provide fair comparisons because reported results vary due to differences in decoder architectures, training schedules, data pipelines, and feature interfaces. This makes it difficult to isolate the true performance of the encoder backbone. Furthermore, standard metrics like mean Intersection-over-Union (mIoU) often mask specific failure modes, particularly regarding boundary delineation under domain shifts.

2. Methodology

The authors propose a strictly controlled benchmark designed to isolate the impact of the encoder backbone by fixing all other variables in the segmentation pipeline.

Controlled Protocol:
- Unified Interface: All backbones (SSMs, CNNs, Transformers) output a standardized 4-stage feature pyramid with specific strides (4, 8, 16, 32) and normalized channel dimensions (64, 128, 256, 512).
- Fixed Decoder: A lightweight, CNN-based U-Net style decoder is used for all experiments.
- Training Consistency: Identical input resolution (512×512), data augmentations, optimization schedules (AdamW), loss functions, and evaluation protocols.
- Loss Function: A composite loss combining Lovász-Softmax (for Jaccard index optimization), Focal loss (for class imbalance), and a specific Boundary Loss ( $L_{Boundary}$ ) that penalizes misclassifications within 2 pixels of ground-truth edges.
Backbones Evaluated:
- Visual SSMs: VMamba (4-directional cross-scan), MambaVision (hybrid Mamba/Attention), and Spatial-Mamba (structure-aware state fusion).
- Baselines: CNN (DeepLabv3 encoder) and Transformer (UNetFormer encoder).
Datasets:
- LoveDA: Focuses on cross-domain generalization between Urban and Rural scenes (0.3m resolution).
- ISPRS Potsdam: Focuses on very high-resolution (5cm) fine-grained urban parsing.
Evaluation Metrics:
- Standard metrics: mIoU, Overall Accuracy (OA), F1-score.
- Diagnostic Metrics: Boundary vs. Interior mIoU analysis using a $\tau=2$ pixel dilation to identify where errors occur.
- Efficiency: Parameters, FLOPs, GPU memory, and Frames Per Second (FPS).

3. Key Contributions

Reproducible Benchmark: The first strictly controlled comparison of three visual SSM families (VMamba, MambaVision, Spatial-Mamba) against CNN and Transformer baselines using a unified 4-stage feature interface and fixed decoder.
Comprehensive Analysis: An evaluation of in-domain accuracy, cross-domain robustness, boundary sensitivity, and computational efficiency under a single measurement protocol.
Deployment-Oriented Insights: Validation on both LoveDA and ISPRS Potsdam to guide model selection for real-world RS applications involving domain shifts.

4. Key Results and Findings

A. Accuracy-Efficiency Trade-offs

SSM Performance: Visual SSMs generally achieve favorable accuracy-efficiency trade-offs compared to the controlled baselines.
Scaling Laws: Intra-family scaling (increasing model size) yields only modest gains in accuracy. For example, on LoveDA, scaling VMamba from Tiny to Base showed minimal mIoU improvement despite significant increases in parameters and FLOPs.
Top Performers:
- MambaVision-L2 achieved the best Urban-to-Rural generalization (41.74 mIoU) but at a high computational cost (250M params).
- VMamba-Small remained highly competitive (40.62 mIoU) with a fraction of the complexity, offering a better balance for resource-constrained deployments.
- Spatial-Mamba underperformed significantly on LoveDA, suggesting that for fragmented RS scenes, feature structure and decoder compatibility are more critical than the SSM initialization alone.

B. Cross-Domain Generalization (Asymmetry)

Directional Bias: There is a strong asymmetry in cross-domain transfer. Models trained on Rural scenes generalize to Urban scenes significantly better than the reverse (Urban $\to$ $\to$ Rural).
- Example: VMamba-Small improved from 40.62 mIoU (Urban $\to$ Rural) to 53.52 mIoU (Rural $\to$ Urban).
Cause: Rural scenes possess greater appearance diversity, promoting transferable representations. Conversely, Urban-trained models tend to overfit to regularized geometry and sharp edges that do not exist in rural imagery.

C. Boundary Delineation as the Dominant Failure Mode

The Gap: Diagnostic analysis revealed a massive performance gap between Interior and Boundary pixels.
- Example: VMamba-Small achieved ~59% mIoU on interior pixels but only ~28% on boundary pixels (a gap of ~30 points).
Implication: Even stronger encoders struggle to resolve geometric ambiguity at edges (thin roads, building outlines). The primary bottleneck is not the encoder's capacity to extract features, but the boundary delineation capability.
Failure Modes: Errors are concentrated on boundary-heavy pixels rather than interior regions, particularly for natural categories (forest, agriculture, water) under domain shifts.

5. Significance and Conclusion

This study establishes a practical reference benchmark for future Mamba-based segmentation backbones. The key takeaways for the remote sensing community are:

Encoder Scaling is Not the Silver Bullet: Simply increasing the size of the SSM encoder yields diminishing returns.
Focus on Robustness and Boundaries: Future improvements should prioritize robustness-oriented design and boundary-aware decoding (e.g., better handling of mixed pixels and geometric cues) rather than solely scaling the encoder.
Domain Shift Strategy: When deploying models across domains, transferring from Rural to Urban is more viable than the reverse.
Protocol Standardization: The paper demonstrates that fair comparison requires isolating the encoder via a unified feature interface and fixed decoder, a protocol that should be adopted in future RS segmentation research.

The authors conclude that while SSMs are promising, the next leap in performance will likely come from architectural innovations in the decoder and boundary handling mechanisms rather than the backbone alone.