OVerSeeC: Open-Vocabulary Costmap Generation from Satellite Images and Natural Language

OVerSeeC is a zero-shot modular framework that leverages large language models and open-vocabulary segmentation to generate executable global costmaps from satellite imagery and natural language instructions, enabling autonomous navigation to adapt to novel entities and dynamic mission constraints without requiring fixed ontologies.

The Gist

Imagine you are planning a road trip for a self-driving car, but instead of a city with clear streets, you are sending it into the wild, off-road wilderness. The car needs a map that tells it where it's safe to drive and where it should avoid.

Traditionally, these maps are like pre-printed coloring books. The creators decide in advance: "Roads are green, water is blue, and buildings are red." If the car sees something new—like a "muddy swamp" or a "baseball field"—the map doesn't know what to do because that color isn't in the book.

OVERSEEC is a new, super-smart system that throws away the pre-printed book and lets the car draw its own map on the fly based on what you say to it.

Here is how it works, broken down into three simple steps using a creative analogy:

The Analogy: The "Smart Travel Agent"

Think of OVERSEEC as a team of three specialized travel agents working together to plan a trip for a robot explorer.

1. The Interpreter (The "Translator")

The Job: You give the system a natural language instruction, like: "I want to stick to the trails and grass, but absolutely avoid the river and the baseball field."
The Magic: A standard computer program might get confused by words like "baseball field" if it was only trained on "roads" and "trees." But OVERSEEC uses a Large Language Model (LLM)—basically a super-smart AI that reads like a human. It acts as the Translator. It understands that "baseball field" is a type of "grass" but has special rules, and it figures out exactly which things you care about. It turns your messy sentence into a clear shopping list of "Things to Find."

2. The Locator (The "Detective")

The Job: Now the system has a list of things to find (River, Trail, Baseball Field), but it needs to find them on a giant, high-resolution satellite photo.
The Magic: This is the Detective. It uses a special "Open-Vocabulary" camera. Unlike old cameras that only recognize 10 specific things, this detective can look at a photo and say, "Ah, there's a river over here, and a baseball field over there," even if it has never seen a baseball field before.

• The Challenge: Satellite images are huge (like a giant billboard). The detective can't look at the whole thing at once. So, it cuts the image into small postcard-sized tiles, finds the objects on each tile, and then stitches them back together perfectly. It even sharpens the edges so the river doesn't look blurry.

3. The Synthesizer (The "Architect")

The Job: Now the system knows where the river and the trail are. But how do we turn that into a "costmap"? A costmap is just a map where "good" places are low numbers (cheap to drive) and "bad" places are high numbers (expensive to drive).
The Magic: This is the Architect. It takes your original instruction ("Avoid the river") and the Detective's findings, and it writes a custom computer program on the spot.

• It says: "Okay, for every pixel that is a river, set the cost to 100. For the trail, set it to 1. For the grass, set it to 5."
• It even understands complex logic like: "Prefer the grass, unless it's next to a building."
• The result is a brand-new, custom map tailored exactly to your specific trip, generated in seconds.

Why is this a big deal?

• No More "Out of the Box" Limits: Old systems break if you ask them to avoid a "mud pit" because they were never taught what a mud pit is. OVERSEEC just asks the AI, "What's a mud pit?" and figures it out instantly.
• Human-Like Flexibility: You can change your mind instantly. If you say, "Actually, the river is dry today, I can cross it," the system rewrites the map immediately. You don't need to retrain the robot or hire a programmer.
• The "Zero-Shot" Superpower: "Zero-shot" means it learns nothing new from scratch. It uses its existing massive knowledge (like a well-read librarian) to solve a problem it has never seen before, just by understanding your words.

The Result

In the paper, the team tested this by asking the system to navigate through complex terrains with weird rules (like avoiding a specific sports field or crossing a river only if it's dry).

• Old Systems: Got lost or drove into the river because they didn't understand the rules.
• OVERSEEC: Drew a perfect path that matched what a human would have drawn, even for things it had never seen before.

In short: OVERSEEC turns a robot's rigid, pre-programmed map into a dynamic, conversational guide that listens to your voice and draws the perfect path for your specific adventure, no matter how weird the terrain gets.

Technical Summary

1. Problem Statement

Autonomous Ground Vehicles (AGVs) operating in off-road environments require global costmaps derived from high-resolution satellite imagery for long-range route planning. Traditional approaches face two critical limitations:

• Fixed Ontologies: Existing models rely on predefined sets of terrain classes (e.g., road, water, building). They fail to recognize novel entities (e.g., "electric towers," "baseball fields") or adapt to mission-specific terrain types not seen during training.
• Rigid Cost Mappings: Standard pipelines use static class-to-cost lookup tables. They cannot interpret complex, compositional natural language instructions (e.g., "prefer grass unless it borders a building" or "avoid the river but cross if dry").

The core challenge is to generate a zero-shot, open-vocabulary costmap directly from a satellite image and a natural language prompt, without task-specific retraining or manual rule design.

2. Methodology: The OVERSEEC Framework

OVERSEEC is a modular, zero-shot framework that decomposes the costmap generation problem into three logical stages: Interpret $\rightarrow$ Locate $\rightarrow$ Synthesize.

A. Entity Identification (Interpret)

• Input: Natural language prompt ($P$) and a default set of common classes.
• Mechanism: A Large Language Model (LLM), specifically Gemma-2-27b-it, parses the prompt to:
  1. Extract relevant semantic classes ($C$), including novel entities.
  2. Categorize classes by geometry (e.g., linear vs. areal) to handle different segmentation characteristics.
  3. Identify spatial relationships and preferences (e.g., hierarchies like "baseball field $\subset$ grass").
• Output: A dynamic list of classes to segment and their geometric properties.

B. Open-Vocabulary Mask Generation (Locate)

This stage addresses the challenge of segmenting high-resolution satellite images using foundation models with fixed input size constraints.

• Sub-module 1: Open-Vocabulary Semantic Segmentation:
  • Uses CLIPSeg (Language-Grounded Semantic Segmentation Model) as the initial segmenter.
  • Tiling Strategy: The high-resolution image is tiled into smaller, overlapping chunks. CLIPSeg processes each tile to generate per-class probability maps.
  • Stitching: Overlapping predictions are averaged to create full-resolution probability maps ($P_c$).
  • Binarization: Thresholds are applied based on geometry (lower thresholds for linear features like trails, higher for areal features) to create coarse masks ($M_c$).
• Sub-module 2: Mask Refinement:
  • Uses SAMRefiner (a variant of Segment Anything Model) to sharpen boundaries and fix connectivity issues in the coarse masks.
  • The coarse mask acts as a spatial prior (prompt) for SAMRefiner, which refines the probability maps and binary masks.
• Output: Refined binary masks ($\hat{M}_c$) and thresholded probability maps ($\hat{P}^\tau_c$) for all identified classes.

C. Costmap Function Composition (Synthesize)

• Mechanism: Instead of a fixed lookup table, the LLM synthesizes an executable Python function ($f_{LLM}$) on the fly.
• Logic: The LLM translates the user's prompt into code that:
  1. Assigns weights to classes based on preference.
  2. Applies spatial predicates (e.g., "near," "unless") using mask operators (AND, OR, NOT, REMOVE).
  3. Handles semantic hierarchies (e.g., removing "baseball field" from the "grass" mask).
  4. Accumulates costs pixel-wise and normalizes the result to a range of $[0, 1]$.
• Execution: The generated function is executed on the refined masks to produce the final scalar costmap ($C$).

3. Key Contributions

1. Zero-Shot Perception Pipeline: A novel pipeline for high-resolution satellite imagery that performs open-vocabulary segmentation while preserving native resolution via tiling and refinement, enabling the detection of arbitrary, novel terrain classes.
2. LLM-Driven Cost Synthesis: Demonstrates that an LLM can interpret complex natural language preferences and synthesize executable costmap functions, enabling dynamic, preference-aligned planning without retraining.
3. Interactive GUI: A graphical interface allowing operators to rapidly iterate on costmaps by modifying natural language prompts, generating updated routes in minutes without annotation or retraining.
4. Ranked Regret Path Integral (RRPI): A new evaluation metric that quantifies how well a planned path aligns with user preference rankings, accounting for both path length and the "regret" of traversing less preferred terrain.

4. Experimental Results

The system was evaluated on datasets ($D2$, $D3$) featuring In-Distribution (ID), Out-of-Distribution (OOD), and Open-Vocabulary (OOD-OV) scenarios, comparing OVERSEEC against fixed-ontology baselines (SegFormer, DINO-UNet).

• Alignment & Performance (RQ1):
  • OVERSEEC achieved significantly lower RRPI scores and path lengths compared to baselines in ID and OOD settings.
  • In human case studies, OVERSEEC trajectories showed the lowest Hausdorff distance to human-drawn reference paths, indicating superior alignment with operator intent.
  • The system successfully encoded fine-grained geometric rules (e.g., "stay on the center of the road" vs. "stay on the edge").
• Novel-Class Generalization (RQ2):
  • In OOD-OV scenarios (e.g., identifying "baseball fields" or "electric towers" not in training data), baselines failed to recognize these entities, leading to high regret.
  • OVERSEEC correctly segmented and applied costs to these novel classes, producing trajectories that respected the user's specific constraints.
• Robustness to Distribution Shift (RQ3):
  • OVERSEEC maintained high segmentation accuracy (IoU) across different geographic regions and visual domains where fixed-ontology models degraded significantly.
  • The use of language-grounded foundation models (CLIPSeg) proved robust to changes in lighting, weather, and terrain appearance.

5. Significance

OVERSEEC represents a paradigm shift in robotic navigation for unstructured environments. By decoupling perception (finding entities) from reasoning (interpreting rules) and synthesis (generating cost logic), it overcomes the rigidity of traditional pipelines.

• Adaptability: It enables rapid mission adaptation to new terrains and changing rules without data collection or model fine-tuning.
• Human-in-the-Loop: The framework bridges the gap between high-level human intent (natural language) and low-level robot execution (costmaps), making autonomous systems more accessible to non-expert operators.
• Scalability: The modular design allows for the integration of future improvements in segmentation or LLM capabilities without redesigning the entire system.

In conclusion, OVERSEEC demonstrates that combining large-scale pre-trained foundation models in a neuro-symbolic framework enables robust, open-vocabulary, and preference-aligned global planning for autonomous vehicles.