Generalizable Multiscale Segmentation of Heterogeneous Map Collections

This paper introduces Semap, a diverse benchmark dataset, and a robust multiscale segmentation framework to enable generalizable semantic segmentation across heterogeneous historical map collections, thereby facilitating the integration of vast cartographic archives into historical geographic studies.

The Gist

Imagine you have a giant, dusty attic filled with thousands of old maps. Some are detailed city plans from Paris, others are rough sketches of the American frontier, and some are colorful world maps from the 1600s. They all look different, use different colors, and draw roads and rivers in unique ways.

For a long time, computer scientists trying to teach machines to "read" these maps have been like a student who only studies for one specific exam. They built special AI models to read only Parisian maps, or only Swiss topographic maps. If you showed that AI a map from a different country or era, it would get confused and fail.

This paper is about teaching the AI to be a polyglot cartographer—a map-reader that can understand any map, no matter where it's from or how old it is.

Here is how they did it, broken down into simple concepts:

1. The Problem: The "Specialist" Trap

Think of the old approach like hiring a chef who only knows how to make perfect pizza. If you ask them to make a sushi roll, they might try to put pepperoni on rice, and it would be a disaster.
Most previous AI models were "pizza chefs." They were trained on huge, uniform sets of maps (like a whole series of identical city atlases). They worked great on those specific maps but failed miserably when faced with the "long tail" of history: the millions of unique, weird, and diverse maps that don't fit into neat categories.

2. The Solution: A "Gym" for the AI

To fix this, the researchers built a new training ground called Semap.

• The Dataset: Instead of feeding the AI 1,000 identical maps of Paris, they gathered 1,439 maps that are all different. Some are tiny insurance plans; others are huge world maps. Some are black and white; others are hand-colored. It's like giving the AI a gym membership where it has to lift every type of weight, not just one.
• The "Fake" Maps (Procedural Synthesis): Since they didn't have enough real, hand-labeled maps to train the AI, they invented a way to generate "fake" maps using code. Imagine a digital artist who can instantly draw a map of a forest, then a city, then a desert, changing the colors and styles randomly. They created over 12,000 of these synthetic maps.
  • Why do this? It's like training a firefighter with smoke machines and fake fires before sending them into a real burning building. It teaches the AI the rules of what a road or a river looks like, without needing a real historical map for every single example.

3. The Strategy: Seeing the Big Picture and the Small Details

Maps are tricky because they have things that are huge (like an ocean) and things that are tiny (like a single house).

• The "Zoom" Trick: The AI looks at the map twice. First, it zooms out to see the whole neighborhood (to understand the big picture). Then, it zooms in to see the details (to spot the small houses). It combines these two views to make a final decision.
• The "Swin" Brain: They used a specific type of AI brain (called a Swin Transformer) that is naturally good at looking at things at different sizes at the same time. It's like having eyes that can focus on a single ant and a whole forest simultaneously.

4. The Results: The "Universal Translator"

When they tested this new AI:

• It got smarter, not dumber: Usually, when you mix different types of data, AI gets confused. But here, the diversity made the AI stronger. It became a "universal translator" for maps.
• It works everywhere: Whether the map was from Indonesia, Turkey, or the US, from the 1600s or the 1900s, the AI performed consistently well. It didn't have a "favorite" region.
• The Score: It beat all previous record-holders by a significant margin, proving that a "one-size-fits-all" approach actually works better than building a custom model for every single map collection.

5. Where It Still Stumbles

The AI isn't perfect yet.

• Thin Lines: It sometimes struggles with very thin lines, like a tiny dotted path or a faint river boundary. It's like trying to read a very faint pencil sketch; the AI might miss the line or confuse it with a shadow.
• Confusing Colors: If a map uses blue to paint land (like in some old artistic maps), the AI might think the land is water. It relies heavily on color, so when artists get creative, the AI gets confused.

Why This Matters

This research is a game-changer for historians and geographers.
Before, if you wanted to study how cities grew over 300 years, you could only study the few map series that were uniform and easy to read. You had to ignore the "messy" archives.
Now, with this "universal" AI, we can finally unlock the long tail of history. We can process hundreds of thousands of unique, individual maps to see how the world changed in granular detail. It turns a dusty attic of unreadable maps into a massive, searchable library of human history.

In short: They taught a computer to stop being a specialist and start being a generalist, using a mix of real maps and computer-generated practice maps, so it can finally read the entire history of cartography.

Technical Summary

1. Problem Statement

Historical map collections are vast and valuable for fields like economic history, urban studies, and climate science. However, existing research in map recognition predominantly focuses on specialist models trained on homogeneous map series (e.g., specific topographic series or city atlases). This approach suffers from several limitations:

• Lack of Generalizability: Models trained on uniform datasets fail to transfer effectively to diverse, heterogeneous map collections (the "long tail" of cartographic archives).
• Data Scarcity: There is a shortage of generic, high-quality ground-truth labels for pretraining models across varied scales, styles, and geographic regions.
• Fragmentation: Labeled data is rarely reused across projects, and results often do not apply to distinct cartographic contexts, hindering the integration of diverse historical archives into large-scale geographic studies.

The core challenge is to develop a generalizable semantic segmentation framework capable of handling the extreme diversity of historical maps (varying scales, styles, and content) without requiring a new specialist model for every collection.

2. Methodology

The authors propose a two-pronged approach: the creation of a diverse benchmark dataset and a robust training/inference framework combining procedural synthesis with multiscale integration.

A. The Semap Dataset

To address data scarcity and diversity, the authors introduced Semap, a new open benchmark dataset:

• Composition: 1,439 manually annotated map patches (768×768 pixels) drawn from the ADHOC database (Aggregated Database on the History of Cartography), which contains ~100k digitized maps.
• Diversity: The dataset covers a wide range of scales (from insurance plans to world maps), styles, and geographic regions (Europe, US, etc.).
• Semantic Classes: It defines 6 classes:
  1. Background: Borders, legends, ornaments.
  2. Boundary: Object outlines (e.g., building footprints, land plots).
  3. Built: Buildings, walls, porticoes.
  4. Road Network: Streets, bridges, railways.
  5. Water: Rivers, seas, lakes.
  6. Non-built: Agricultural land, natural areas, parks.
• Annotation Effort: Approximately 400 hours of manual work, including updating existing datasets (HCMSSD) to include the new "boundary" class.

B. Procedural Data Synthesis

To augment the limited real-world annotations, the authors employed procedural data synthesis rather than generative AI (GANs/Diffusion):

• Source: Derived from MapTiler Planet contemporary geodata.
• Process:
  • Sampling: Geodata is sampled based on the scale and spatial distribution of the ADHOC database.
  • Stylization: A probabilistic algorithm applies graphical processes (colors, hatching, textures, icons) to mimic historical styles. Parameters are derived from Semap annotations.
  • Features: Includes relief representation (hillshading, isolines), randomized text labels, and artificial graticules.
  • Volume: 12,122 synthetic raster samples were generated.
• Rationale: Procedural generation offers greater control over texture frequency and avoids the "mode collapse" risks of generative models, ensuring the model learns morphological cues rather than specific stylistic artifacts.

C. Model Architecture and Training

• Architecture: Mask2Former with a Swin-L (Swin Transformer Large) backbone. This choice leverages the hierarchical design of Swin Transformers to handle multiscale objects effectively.
• Training Strategy:
  • Pretraining: The model is first trained on a mix of 90.9% synthetic data and 9.1% real Semap data.
  • Fine-tuning: The model is subsequently fine-tuned exclusively on real Semap data.
  • Loss Function: A composite loss combining Binary Cross-Entropy (mask prediction), Cross-Entropy (class attribution), and Dice Loss (to handle class imbalance). The "boundary" class is weighted lower (0.6) to prevent it from hindering the learning of major land classes.
• Inference Strategy (Multiscale Integration):
  • Large map images are partitioned into 768×768 patches with 64-pixel overlaps.
  • Dual-Scale Inference: Predictions are generated at both the original resolution and half-resolution.
  • Consensus: Raw logits from overlapping windows and both scales are averaged to produce the final segmentation mask, improving the recognition of large objects and reducing border effects.

3. Key Contributions

1. Semap Dataset: A new, diverse, open-source benchmark for historical map segmentation that reflects the "long tail" of cartographic archives, including a novel "boundary" class for vectorization potential.
2. Procedural Synthesis Framework: A method for generating visually plausible synthetic historical maps using procedural graphics, which effectively augments training data without the instability of generative models.
3. Generalizable Model: Demonstration that a diversity-driven approach (training on heterogeneous data) yields more robust and transferable models than specialist models trained on homogeneous series.
4. Multiscale Inference: A strategy that significantly boosts performance on large-scale objects by integrating predictions across different resolutions.

4. Results

The proposed method achieved State-of-the-Art (SOTA) performance on both the Semap test set and established benchmarks (HCMSSD-Paris and HCMSSD-World).

• Performance Metrics (Semap Test Set):
  • Mean IoU (mIoU): 74.2% (across geographic classes).
  • Class-wise IoU: Built (79.8%), Non-built (81.8%), Water (72.2%), Road Network (62.9%).
  • Precision/Recall: High precision (85.4% mean) and strong recall (79.4% mean).
• Benchmark Comparisons:
  • Outperformed previous UNet-based and Graph Neural Network (SCGCN) approaches.
  • On the HCMSSD-World (highly diverse) dataset, the model achieved an mIoU of 76.3% (transfer learning), a massive improvement over the previous best of ~45% (UNet-ResNet101).
  • On HCMSSD-Paris, it achieved 76.0% mIoU, surpassing the previous best by ~22 percentage points.
• Ablation Studies:
  • Removing multiscale integration reduced mIoU by ~4.2 points.
  • Removing synthetic pretraining reduced mIoU by ~5.1 points.
  • Both components were critical for generalization, with synthetic data primarily boosting recall and multiscale integration improving both precision and recall.
• Bias Analysis: Statistical analysis (OLS regression) showed that segmentation performance is largely independent of metadata variables (collection institution, publication country, scale, year), with an $R^2$ of only 0.043. This confirms the model's robustness and lack of systematic bias across different historical contexts.

5. Significance and Conclusion

This work fundamentally shifts the paradigm of historical map recognition from specialist, series-specific models to generalizable, diversity-driven models.

• Unlocking the "Long Tail": By proving that models can generalize across heterogeneous collections, this research enables the automated processing of hundreds of thousands of individual, non-standardized map documents that were previously inaccessible to large-scale computational analysis.
• Methodological Shift: It demonstrates that diversity in training data, when paired with appropriate architectures (Swin Transformers) and strategies (procedural synthesis, multiscale inference), acts as a catalyst for robustness rather than a hindrance.
• Future Impact: The ability to extract geographic land classes (built, water, non-built) from diverse archives opens new avenues for historical geography, allowing researchers to model long-term territorial evolution, urban sprawl, and environmental changes with unprecedented granularity.

The authors conclude that while challenges remain in detecting thin linear features (boundaries, small roads), the proposed framework successfully bridges the gap between the vast diversity of cartographic archives and the need for automated, scalable geographic analysis.