💻 computer science

Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

This paper proposes "Align and Segment" (AnS), a novel unsupervised learning framework that simultaneously learns to align misaligned image-label pairs via affine transformations and performs high-quality building segmentation without requiring any ground-truth labels.

Original authors: Venkanna Babu Guthula, Oswin Krause, Dimitri Gominski, Hui Zhang, Johan Mottelson, Ankit Kariryaa, Nico Lang, Christian Igel

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Venkanna Babu Guthula, Oswin Krause, Dimitri Gominski, Hui Zhang, Johan Mottelson, Ankit Kariryaa, Nico Lang, Christian Igel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, high-tech coloring book. On one page, there's a crisp, clear satellite photo of a city. On the other, there's a stencil of where the buildings should be. But here's the catch: someone glued the stencil onto the photo, and they did a terrible job. The stencil is shifted a few inches to the left, maybe tilted a bit, or rotated. If you try to teach a computer to learn what a building looks like using this messy pair, the computer gets confused. It might just learn to draw the buildings in the wrong spots, matching the bad stencil instead of the real photo.

This is the exact problem the researchers at the University of Copenhagen tackled. They call their solution Align and Segment (AnS).

The "Shortcut" Trap

Usually, when we train AI, we give it perfect pairs: a photo and the exact correct outline. But in the real world, especially with maps from different sources (like OpenStreetMap vs. satellite images), these pairs are often "misaligned."

The paper argues that if you just throw a standard AI at this problem, it will take the "easy way out." Instead of figuring out how to fix the shift, the AI will simply learn to draw the buildings exactly where the messy stencil says they are, even if that's wrong. It's like a student who memorizes the wrong answer key just to pass the test, rather than actually learning the subject. The authors explicitly show that a simple, "naive" approach where you try to fix the alignment and the drawing separately often fails because the AI gets stuck in this shortcut.

The Magic Trick: Two Heads, One Brain

To fix this, the authors built a system with two special parts working together, like a detective and an artist.

The Artist (SNet): This part tries to draw the building outlines based on the satellite photo.
The Detective (TNet): This is the new, clever part. It looks at the messy stencil and the Artist's drawing. Its job is to figure out exactly how much the stencil is shifted, rotated, or tilted. It then "un-does" that shift, sliding the stencil back into the right place so it matches the photo.

The cool part is that they teach these two to learn at the same time. The Detective learns to fix the stencil, and the Artist learns to draw the buildings correctly based on the fixed stencil.

How They Stopped the Cheat

The researchers knew the Detective might get lazy and just say, "I didn't shift anything," while the Artist just copied the messy stencil. To stop this, they used two tricks:

The "Shuffle" Test (Consistency Loss): They would randomly spin or flip the messy stencil before showing it to the Detective. If the Detective is smart, it should be able to say, "Ah, you spun this 90 degrees, so I need to spin it back!" If the Detective just ignores the spin, it fails the test. This forces the Detective to actually learn the geometry of the shift.
The "Mirror" Trick (Augmentation): They would flip the images horizontally or vertically during training. If the stencil was shifted 50 pixels to the right in the original, flipping the image makes it look like it's shifted 50 pixels to the left. By seeing both versions, the AI learns that the shift isn't a permanent rule of the world; it's just a mistake that needs fixing.

What They Found

The team tested this on data from cities like Las Vegas, Paris, and Khartoum. They created two types of test scenarios:

Random Mess: Where the stencils were shifted in random directions and amounts.
Systematic Mess: Where every single stencil was shifted exactly 50 pixels to the right (a very common real-world problem).

In the "Systematic Mess" tests, standard AI models failed miserably, getting stuck in the shortcut. But the AnS method? It worked.

On the random mess, it achieved a score (called IoU) of 0.79 for drawing buildings and 0.84 for fixing the alignment.
On the tricky systematic mess (the 50-pixel shift), it scored 0.78 for buildings and 0.88 for alignment.

They also tested it on a dataset with 41 different cities (ReBO) and real-world data from San Juan using OpenStreetMap. In these cases, where they didn't have the "perfect" answer key to check against, the system still managed to align the maps and draw the buildings with high precision.

What It Can't Do (Yet)

The authors are careful to point out what this isn't.

It doesn't fix missing buildings. If the stencil forgot to draw a house, the AI won't magically invent it.
It currently only fixes shifts that are simple (sliding, rotating, tilting). It can't fix complex warping where the map looks like it's been melted.
The "Artist" part still produces slightly blurry edges sometimes, because the system is focused on fixing the alignment first.

The Bottom Line

This isn't a magic wand that makes all map data perfect instantly. But it is a powerful new way to teach computers to clean up their own messes. By teaching the AI to fix the alignment while it learns to draw, they showed that we can use huge amounts of existing, messy map data (like OpenStreetMap) to train better building detectors, without needing a human to manually fix every single image first. It turns a "broken" dataset into a useful one, all without needing a "golden" perfect label to start with.

Technical Summary: Align and Segment (AnS)

Problem Statement
Supervised learning for image segmentation typically requires spatially aligned image and label sets. However, in remote sensing and other domains, labels often originate from different sources (e.g., OpenStreetMap, crowd-sourced data) than the imagery (e.g., satellite or drone), resulting in significant spatial misalignment. This misalignment acts as a form of label noise that can severely deteriorate model performance.

A critical challenge identified in this work is systematic misalignment (biased noise), such as consistent translation errors caused by erroneous orthorectification or land registry inaccuracies. Standard segmentation networks possess sufficient capacity to simply learn these misaligned labels, effectively "shortcutting" the learning process by predicting the noisy labels directly rather than learning the true underlying segmentation. Existing approaches to this problem often assume small, unbiased misalignments or require "golden" (clean, ground-truth) labels for supervision, which are scarce in many real-world scenarios, particularly in regions lacking high-quality building data.

Methodology: Align and Segment (AnS)
The authors propose Align and Segment (AnS), a model-agnostic, unsupervised framework that simultaneously learns label alignment and semantic segmentation without requiring any golden labels. The architecture consists of two primary components:

Segmentation Network (SNet): A canonical semantic segmentation network (e.g., U-Net with ConvNeXt encoder) that maps input images to probabilistic segmentation masks.
Transformation Network (TNet): A lightweight module inspired by Spatial Transformer Networks (STN). It takes the misaligned label mask and the SNet's predicted segmentation mask as input to predict the parameters of an affine transformation ( $\hat{\theta}$ ). This transformation is applied to the misaligned label to produce a realigned mask ( $I_{\hat{y}a}$ ).

To prevent the SNet from learning the misaligned labels directly (shortcut learning) and to ensure the TNet learns the correct transformation, the authors introduce two key mechanisms:

Consistency Loss ( $L_{Con}$ ): A self-supervised regularization term. The system applies a known random transformation to the misaligned label and feeds it to the TNet. The TNet is penalized if it cannot predict the inverse of this known transformation. Crucially, this loss is independent of the SNet's output, forcing the TNet to learn alignment based on the relationship between the two segmentation masks rather than relying on the SNet's potentially noisy predictions.
Data Augmentation: To counteract systematic biases (e.g., a constant 50-pixel shift), the authors apply geometric augmentations (flips, rotations) to both the image and the misaligned label pairs. This ensures that in expectation, the translation bias is removed, preventing the model from settling into a local optimum where it learns the bias rather than the true alignment.

The total training objective combines the standard cross-entropy loss between the realigned labels and the SNet prediction with the consistency loss.

Key Contributions

Novel Methodology: The proposal of AnS, a plug-and-play approach that enables direct learning from misaligned labels using any existing semantic segmentation network, without requiring golden labels or assuming unbiased transformations.
Disentanglement Strategy: The demonstration that combining a self-consistency loss with specific data augmentations effectively mitigates shortcut learning, allowing the network to disentangle alignment correction from segmentation prediction.
Empirical Validation: Extensive experiments on synthetic and real-world datasets (Las Vegas, Paris, Khartoum, San Juan, and a multi-city ReBO dataset) showing that AnS generalizes across different cities, building densities, and noise levels (both random and systematic).

Results

Performance: On synthetic datasets with systematic bias ( $D_{bias}$ ), AnS significantly outperformed unsupervised baselines (Map Alignment, AutoCorrect, Spatial Correction) and achieved competitive results against supervised baselines (MapRepair, Alignment Correction Network) that typically require clean labels or specific training conditions.
Robustness: The method remained robust to misalignment magnitudes up to 100 pixels in random noise scenarios and approximately 70 pixels in systematic bias scenarios.
Ablation Studies: Experiments confirmed that both the consistency loss and data augmentation are complementary; removing either led to performance degradation, particularly in systematically biased settings.
Real-World Application: When applied to OpenStreetMap building footprints paired with WorldView-3 imagery, AnS successfully corrected misalignments. While the resulting segmentation masks exhibited some blurriness (attributed to inherent label errors and lack of specialized boundary loss), the alignment transformations were accurately estimated.

Significance
The paper argues that the misalignment between independent imagery and existing footprint data (like OpenStreetMap or Google Open Buildings) is a major bottleneck preventing the large-scale utilization of these valuable, hand-curated datasets for training deep learning models. By demonstrating that high-quality building segmentation and precise label-image alignment can be learned simultaneously without "golden" labels, AnS offers a pathway to generate inexpensive, high-quality building footprint data with extensive coverage. This is particularly significant for applications in urban planning and risk assessment in informal settlements where curated data is scarce. The work establishes that joint learning of alignment and segmentation is not only feasible but essential for leveraging the vast amounts of existing, albeit misaligned, geospatial data.