💻 computer science

Plan2Map: A Multimodal Benchmark for Document-Grounded Geospatial Boundary Reconstruction from Planning Records

This paper introduces Plan2Map, a multimodal benchmark and the GeoPlanAgent system for reconstructing geospatial boundaries from UK planning records by integrating document text and map visuals, demonstrating that a structured, tool-in-the-loop approach significantly outperforms direct VLM baselines in accuracy and reliability.

Original authors: Fabian Degen, Oishi Deb, Jindong Gu, Junchi Yu, Samuele Marro, Philip Torr, Jialin Yu

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fabian Degen, Oishi Deb, Jindong Gu, Junchi Yu, Samuele Marro, Philip Torr, Jialin Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very old, slightly crumpled piece of paper from a town hall. On this paper, there is a legal rule that says, "You cannot build a new house in this specific area." The paper has a map drawn on it with a red line circling that area, and some text explaining the rules.

The problem? That red line is just a drawing. A computer doesn't know where that red line is on the actual Earth. It can't tell if the line covers a park, a street, or a specific house. To a computer, it's just red ink on a page.

Plan2Map is a new "test" created by researchers to see if computers can solve this puzzle. They gathered 208 of these real-world planning documents and paired them with the correct answer (the exact digital map of that area). Their goal: Can a computer look at the messy paper and draw the exact digital map on its own?

The Challenge: The "Lost in Translation" Problem

The researchers found that if you just ask a super-smart AI (like a high-tech chatbot) to look at the paper and "draw the map," it usually fails. It's like asking someone to translate a poem into a map; they might understand the words, but they can't figure out the geography. The AI gets confused by the handwriting, the weird angles of the map, or the fact that the map is upside down.

The Solution: GeoPlanAgent (The "Detective Team")

Instead of asking one AI to do everything at once, the researchers built a team of specialized AI "agents" that work together like a detective squad. They call this team GeoPlanAgent.

Here is how the team solves the puzzle, step-by-step:

The Reader (The Librarian):
First, an AI agent reads the document like a librarian. It doesn't try to draw anything yet. It just looks for clues: "Ah, it mentions 'Norwich Road' and 'Postcode 123'." It also finds the page with the map and checks if the map is upside down. It writes down a list of clues for the next team member.
The Locator (The GPS Navigator):
Next, a second agent takes those clues. It goes to a digital map of the UK and searches for "Norwich Road" and that postcode. It finds a rough spot on the map, kind of like saying, "The treasure is somewhere in this neighborhood." It's not perfect yet, but it's a good starting point.
The Matcher (The Jigsaw Puzzle Solver):
This is the magic step. The team takes the blurry, old map from the paper and tries to fit it over the crisp, modern digital map, like matching a puzzle piece. They slide the old map around until the roads and buildings line up perfectly with the digital map. Once they find the match, they know exactly where the paper map sits in the real world.
The Tracer (The Artist):
Now that they know where the map is, a specialized tool (called SAM 3) looks at the red line on the paper and traces it. Because the computer now knows the scale and the location, it can turn that red line into a precise digital shape (a polygon) that fits exactly onto the real-world map.
The Critic (The Quality Control Inspector):
Finally, a third agent looks at the result. It asks, "Does this red line actually cover the right buildings? Do the roads match?" If the answer is "No," it sends the team back to step 2 to try again. If the answer is "Yes," it approves the final map.

The Results

When the researchers tested this "Detective Team" against the 208 documents:

The Team Won: They successfully recreated the digital maps for about 68% of the cases with very high accuracy.
The Solo AI Lost: When they asked a single, powerful AI to do the whole job alone (without the team steps), it only got about 10% right. It was like asking a human to draw a map of their house while blindfolded.

Why This Matters

This isn't just about drawing lines. It's about turning thousands of old, paper-based rules into digital data that governments and developers can actually use. Right now, checking if a building project is allowed often requires a human to look at a dusty map and guess where the line is. This system shows that, with the right "teamwork" between different AI tools, computers can finally read these old maps and turn them into useful digital information.

In short: Plan2Map is the test, and GeoPlanAgent is the team of specialists that proved computers can learn to read old planning maps, provided they don't try to do it all in one giant leap.

Technical Summary: Plan2Map

Problem Definition

The paper addresses the challenge of document-grounded geospatial boundary reconstruction. In the context of UK planning records, specifically Article 4 Directions, authorities restrict permitted development rights for specific geographic areas. While these restrictions are legally binding, the source documents (PDFs) typically provide only indirect spatial evidence: textual notices, legal schedules, and scanned or embedded map plates with hand-drawn or hatched boundaries. They rarely provide machine-readable geometry (e.g., GeoJSON).

The core difficulty lies in reconstructing a valid, structured spatial object (a polygon) from distributed textual, visual, and geographic evidence without access to a pre-existing digital boundary. The system must:

Read heterogeneous PDFs containing scanned pages and embedded maps.
Identify spatial evidence (postcodes, grid references, map labels).
Ground the relevant map in real-world geography.
Reconstruct the boundary polygon and project it into geospatial coordinates.

This is distinct from standard document understanding (which returns text) or simple field extraction, as it requires a multi-step reasoning process to produce an externally verifiable geospatial output.

Methodology: GeoPlanAgent

To solve this, the authors propose GeoPlanAgent, a decomposition-based, tool-augmented agentic system. Rather than asking a Vision-Language Model (VLM) to generate a GeoJSON in a single pass, the system decomposes the task into distinct stages:

Reader Agent: Processes the raw PDF to extract structured spatial evidence (postcodes, OS grid references, site addresses, map page indices, and scale). It also determines if the document covers an entire administrative district (triggering a shortcut).
Worker Agent: Orchestrates the reconstruction pipeline using a loop of tools:
- Locate Sub-agent: Uses the extracted evidence to query an offline geocoder (OS Open Names) to generate approximate candidate coordinates for the map center, along with an uncertainty radius ( $\sigma$ ).
- Map Registration: Aligns the planning map against a national basemap (OS Open Zoomstack) using feature matching (MINIMA-LoFTR) and RANSAC to fit an affine transform. This step includes a sliding-window search to refine the coarse location.
- Boundary Segmentation: Uses SAM 3 (Segment Anything Model) fine-tuned with LoRA (Low-Rank Adaptation) to segment the boundary mask from the registered map image.
- Projection: Projects the segmented mask into WGS84 coordinates using the affine transform derived from registration, yielding a candidate GeoJSON.
- Commit Logic: Decides whether to accept a candidate based on RANSAC inlier counts, scale consistency, and road-name agreement.
Critic Agent (Optional): Performs a second-pass visual verification on the top-3 candidates, comparing the planning map overlay against the basemap to approve, switch, or trigger a re-localization.

The system is implemented using pydantic-ai and defaults to Gemini 3 Flash for LLM calls to ensure reproducibility.

Key Contributions

The paper introduces four primary contributions:

Plan2Map Benchmark: A manually curated dataset of 208 UK Article 4 Direction records. Each case pairs a source planning PDF with a verified reference GeoJSON boundary and a location-map visualization. The dataset spans 1958–2025, covering varied document qualities, boundary shapes (from simple rectangles to complex multi-part geometries), and 29 local planning authorities.
Task Definition and Protocol: A formal definition of the reconstruction task where the system receives only the source document, while the reference GeoJSON is held out for scoring. Evaluation metrics include Intersection-over-Union (IoU), centroid error, and cost.
GeoPlanAgent System: A modular, decomposition-based architecture that separates document reading, localization, map registration, and boundary extraction. This design allows for targeted diagnosis of failure modes.
Diagnostic Analysis: An empirical study demonstrating that direct VLM-to-GeoJSON prediction is unreliable for this task, while specific components (like supervised boundary segmentation and tool-backed geocoding) are critical for success.

Experimental Results

Evaluated on the full 208-case benchmark, GeoPlanAgent achieves:

0.736 Mean IoU and 0.904 Median IoU.
67.8% of predictions achieve an IoU $\ge$ 0.8.
A median centroid error of 4.6 meters.
78.8% of cases have a centroid error within 10% of the boundary diameter (Acc@0.1D).

In contrast, direct VLM baselines (e.g., Gemini-3.1-Pro, GPT-5.5-Pro) performing end-to-end prediction without tools achieve significantly lower performance:

Mean IoU ranges from 0.04 to 0.11.
No direct baseline achieves an IoU $\ge$ 0.8 on any case in the stratified subset.
Centroid errors for direct baselines range from 386m to 1.1km.

Ablation Studies reveal:

Localization: The tool-backed Locate sub-agent reduces median error to 176m, while the subsequent map-registration step tightens this to 5m (a 38x improvement).
Segmentation: Supervised fine-tuning of SAM 3 with LoRA improves mean pixel IoU by 0.30 over the vanilla SAM 3 baseline, confirming that boundary extraction is no longer the dominant bottleneck once adapted.
Direct VLM Failure: Even frontier models fail to reliably trace boundaries or localize maps in a single pass, highlighting the necessity of the decomposition approach.

Significance and Claims

The paper positions Plan2Map as a concrete testbed for multimodal geospatial reconstruction from public planning records. Its significance lies in:

Practical Impact: Addressing the "Extract initiative" and the need to convert legacy planning records into structured digital data for digital planning systems.
Methodological Insight: Demonstrating that for complex spatial tasks, decomposition and tool use are superior to monolithic VLM generation. The paper argues that direct prediction remains unreliable, and that the primary bottlenecks in current systems are localization and map registration, not just boundary tracing.
Scalability: While the current benchmark is UK-specific (leveraging rich open geospatial infrastructure like OS Open Names), the authors claim the agent architecture is jurisdiction-agnostic. Extending the system to other countries would primarily require re-pointing the geospatial tools to local equivalents.

The authors conclude that Plan2Map and GeoPlanAgent provide a foundation for studying document-grounded geospatial reconstruction, with future work focused on improving robustness in low-quality scans and handling cases where boundaries are described purely by text without visual maps.