GISclaw: An Open-Source LLM-Powered Agent System for Full-Stack Geospatial Analysis

The paper introduces GISclaw, an open-source, full-stack geospatial analysis agent system that integrates LLMs with a persistent Python sandbox and diverse open-source GIS libraries to support vector, raster, and tabular data, featuring novel architectural designs and evaluation protocols that achieve high task success rates on the GeoAnalystBench benchmark.

The Gist

Imagine you have a brilliant, hyper-intelligent assistant who speaks perfect English and knows how to write computer code. But there's a catch: this assistant has never actually done geography or map-making before. If you ask it to "find the best spot for a new fire station," it might write code that looks perfect but tries to use a tool that doesn't exist, or it might forget to check if the map is upside down.

This is the problem GISclaw solves.

Here is the story of GISclaw, explained without the jargon.

🗺️ The Problem: The "Map-Making" Bottleneck

Geographic Information Systems (GIS) are like the ultimate toolkits for understanding the world. They help us plan cities, track wildfires, and predict floods. But using them is hard. You need to know how to code, how to handle different types of data (like satellite photos vs. spreadsheet lists), and you need expensive, proprietary software.

Most people who know what analysis they need (e.g., "Show me where the flood risk is highest") don't know how to code it.

🤖 The Solution: GISclaw (The "Smart Intern")

The authors built GISclaw, an open-source system that acts like a super-smart intern for geographers. It takes your plain English request ("Analyze flood risks") and turns it into working code.

Here is how it works, using three simple metaphors:

1. The "Persistent Sandbox" (The Never-Ending Workbench)

Most AI code generators write a script, run it, and then forget everything. If the script crashes, they have to start over.
GISclaw is different. It lives in a persistent Python sandbox. Imagine a workbench where you can leave your tools, your half-finished models, and your notes out. If the intern makes a mistake, they don't throw the whole project away; they just look at their notes, fix the specific error, and keep building. This allows for complex, multi-step projects like "Load the map, fix the coordinates, run the math, and draw the picture" without losing track.

2. The "Three Golden Rules" (The Training Manual)

The AI is smart, but it doesn't know the specific rules of geography. The authors gave it three strict rules to follow:

• Rule 1: "Look Before You Leap" (Schema Analysis): Before writing any code, the AI must look at the actual data files to see what's inside. It can't guess column names; it has to read the file. This stops it from trying to calculate "poverty rates" when the file actually says "income."
• Rule 2: "No Fancy Tools" (Package Constraint): The AI is told, "You can only use free, open-source tools." This stops it from trying to use expensive, paid software (like ArcGIS) that the system doesn't have.
• Rule 3: "Listen to the Expert" (Domain Knowledge): If a human expert says, "Remember, fire stations need a 2km buffer," the AI listens. It doesn't just guess the distance; it uses the expert's rule.

3. The "One Brain vs. Two Brains" Experiment

The researchers tested two ways the AI could work:

• Single Agent (The Solo Artist): One AI brain tries to do everything at once. It thinks, acts, checks, and repeats.
• Dual Agent (The Manager and the Worker): One AI (the Manager) breaks the big task into small steps. A second AI (the Worker) does the steps. If the Worker fails, the Manager tries a new plan.

The Surprise Finding:
You might think "Two brains are better than one." Not always.

• For the Super-Brains (like GPT-5 or DeepSeek): The "Manager" actually got in the way. The Manager over-complicated things, causing the Super-Brain to get confused and fail. The Solo Artist worked best.
• For the Smaller Brains (like the 14B model): The "Manager" helped. It broke the big, scary task into tiny, manageable pieces, helping the smaller AI succeed where it would have otherwise given up.

The Lesson: Don't over-engineer the system. If you have a genius AI, let it work alone. If you have a smaller, cheaper AI, give it a manager to help it focus.

📊 The Results: How Good Is It?

The team tested GISclaw on 50 difficult geography tasks (like predicting mineral deposits or mapping heat islands).

• Success Rate: With the best setup, it succeeded 96% of the time.
• Cost: It can run on a standard gaming computer (using a free, open-source model) for $0, or on expensive cloud servers for a few dollars.
• The "Functional Equivalence" Problem: In geography, there are often 5 different ways to do the same math. A standard "spell-checker" for code would say, "This code is wrong because it looks different from the answer key." GISclaw uses a smarter checker that says, "This code is right because it produces the exact same map, even if the code looks different."

🚀 Why This Matters

GISclaw proves that we don't need to wait for "perfect" AI to automate geography.

1. It's Open: Anyone can use it, not just people with expensive licenses.
2. It's Flexible: It works with satellite images, maps, and spreadsheets all at once.
3. It's Honest: It admits when it's stuck and asks for help or tries a different angle.

In a nutshell: GISclaw is the bridge between "I have a great idea for a map" and "Here is the finished map." It turns the complex, scary world of geospatial coding into a conversation you can have with a helpful, persistent, and rule-following robot.

Technical Summary

1. Problem Statement

The convergence of Large Language Models (LLMs) and Geographic Information Science (GIS) aims to automate complex spatial analysis. However, existing LLM-powered GIS agents face three critical system-level limitations:

• Narrow Data-Type Coverage: Most systems are limited to vector data or offer only basic raster support. None provide an integrated workflow spanning vector, raster, and tabular data (full-stack analysis) within a single agent execution.
• Platform Dependency & Model Monoculture: Many agents are tightly coupled to proprietary platforms (e.g., ArcGIS) or specific LLMs (e.g., GPT-4), preventing systematic comparisons and hindering deployment in air-gapped or resource-constrained environments.
• Fragmented Evaluation: Current metrics (binary success, CodeBLEU) fail to address the "functional equivalence problem" in GIS, where semantically correct code can have vastly different lexical structures (e.g., gpd.overlay() vs. shapely.buffer()).

2. Methodology: The GISclaw System

GISclaw is an open-source, model-agnostic agent system designed to bridge the gap between natural language instructions and executable geospatial workflows.

A. System Architecture

• Persistent Python Sandbox: Unlike stateless code generators, GISclaw uses a persistent Jupyter-like environment. Variables, imports, and intermediate results survive across execution rounds, enabling iterative exploration and self-correction.
• Open-Source Library Stack: The system pre-loads a comprehensive suite of open-source libraries (GeoPandas, rasterio, scipy, scikit-learn, etc.), eliminating dependencies on proprietary software.
• Model Agnosticism: A unified LLMEngine interface supports heterogeneous backends, ranging from cloud APIs (GPT-5.4, DeepSeek) to locally deployed open-weight models (Llama-3.3-70B, Qwen2.5-14B) on consumer GPUs.
• Agent Architectures: The system implements two pluggable architectures for controlled comparison:
  1. Single Agent (SA): Follows a ReAct (Reason-Act) loop with an Error Memory module to prevent repeating failed approaches.
  2. Dual Agent (DA): Uses a Plan-Execute-Replan pipeline where a Planner decomposes tasks, and a Worker executes steps. If a step fails, the Planner acts as a Replanner to generate an alternative path.

B. Domain-Specific Prompt Engineering

To compensate for LLMs' limited geospatial training data, GISclaw enforces three engineered rules:

1. Schema Analysis: The agent must inspect data schemas (column names, CRS, types) before generating code to prevent column-name guessing errors.
2. Package Constraint: Explicitly forbids proprietary libraries (e.g., arcpy) and redirects agents to open-source equivalents.
3. Domain Knowledge Injection: Allows users to inject high-level workflows or domain-specific parameters (e.g., flood depth thresholds) to guide the agent, mitigating hallucinations.

C. Three-Layer Evaluation Protocol

To address the functional equivalence problem, the authors propose a composite evaluation metric ($S_{comp}$) combining:

1. L1: Code Structure: Uses CodeBLEU and a novel API Operation F1 metric (extracting GIS-specific operations like buffer, overlay, kriging) to measure functional correctness regardless of lexical overlap.
2. L2: Reasoning Process: Evaluates execution logs via embedding similarity (cosine similarity with gold-standard logs) and an LLM-as-Judge scoring task understanding, data handling, and self-correction.
3. L3: Output Accuracy: Type-specific verification (e.g., pixel correlation for rasters, feature count for vectors, visual scoring for maps).

3. Key Contributions

1. Full-Stack Agent System: The first open-source system supporting vector, raster, and tabular analysis with a persistent sandbox and support for 6 diverse LLM backends.
2. Architecture-Capability Matching Principle: A systematic finding that architectural complexity should be inversely proportional to model capability. Multi-agent (Dual Agent) architectures degrade the performance of strong models due to unnecessary overhead but offer marginal benefits for weaker models.
3. Unified Evaluation Protocol: A multi-layer protocol that resolves the functional equivalence problem in GIS code assessment, moving beyond simple lexical similarity.
4. Empirical Design Lessons: Identification of infrastructure-level bottlenecks (data pathing, API state management) that, when fixed, improved task success by 400% on intermediate tasks, proving that system design often matters more than model reasoning.

4. Experimental Results

The system was evaluated on GEOANALYSTBENCH (50 expert tasks) across 600 combinations of 6 models and 2 architectures.

• Performance:
  • DeepSeek-V3.2 achieved the highest success rate (96%) in Single Agent mode with a total cost of only $0.50 for 50 tasks.
  • GPT-5.4 led in Dual Agent mode (88% success) but at a significantly higher cost ($12.30).
  • Local Models: Qwen2.5-Coder-14B (running on a single RTX 3090) achieved 52% success in SA mode. Notably, when successful, its output quality (0.590) was comparable to GPT-4.1 (0.618), suggesting code-specialized fine-tuning is more impactful than raw parameter scaling for local deployment.
• Architecture Impact:
  • Strong models (DeepSeek, GPT-5.4) suffered significant performance drops in Dual Agent mode (e.g., DeepSeek dropped from 96% to 32%).
  • Weak models (Llama-70B, Qwen-14B) saw marginal gains or parity in Dual Agent mode, validating the "inverse proportionality" design principle.
• Failure Analysis:
  • Infrastructure Bugs: 400% improvement in success rates was achieved by fixing environment issues (e.g., file path casing, API state management) rather than improving the LLM.
  • Knowledge Gaps: The study identified a hierarchy of gaps:
    • API Gap: Solvable via RAG/Prompting.
    • Parametric Gap: Domain thresholds (e.g., flood depth) often missed, leading to "plausible but wrong" outputs.
    • Conceptual Gap: Physical relationships (e.g., flood depth = sea level - elevation) and semantic conventions (e.g., "map" implies PNG, not GeoJSON) that RAG cannot resolve.

5. Significance and Future Directions

• Democratization of GIS: GISclaw enables fully offline, cost-effective geospatial analysis on consumer hardware, removing barriers for domain scientists lacking programming skills.
• Engineering over Reasoning: The paper shifts the focus from "bigger models" to "better system design," demonstrating that robust infrastructure (sandbox management, error handling) is the primary determinant of agent success.
• Human-in-the-Loop Viability: For local deployments, the authors propose a collaborative paradigm where the agent handles structured analytical steps (which it performs well) while the human validates key transitions, effectively bypassing the limitations of current 14B models.
• Future Work: Directions include extending to real-time data streams, developing context-aware deduplication, and creating hybrid architectures that couple LLMs with domain-specific physical models to bridge the conceptual knowledge gap.

Availability: The system, code, and benchmarks are open-source at https://github.com/geumjin99/GISclaw.