CrossMaps: Confidence-Aware Open-Vocabulary Semantic Mapping for Rover Navigation

Technical Summary: CrossMaps

Problem Statement
Autonomous rovers operating in unstructured environments, such as planetary surfaces, face significant challenges in maintaining spatial maps due to partial observability, sensor noise, and dynamic conditions. Traditional navigation systems rely on SLAM-based geometric maps that capture spatial structure but lack semantic understanding. While recent Vision-Language Models (VLMs) have enabled open-vocabulary semantic mapping (e.g., VLMaps), deploying these systems on mobile rovers remains difficult. Observations are noisy, viewpoint-dependent, and subject to lighting variations. Existing open-vocabulary mapping methods generally fuse all evidence into a single representation with limited treatment of uncertainty, struggling to balance the plasticity required for adapting to new evidence with the robustness needed for long-horizon stability.

Methodology
The authors introduce CrossMaps, a real-time, confidence-aware open-vocabulary semantic mapping pipeline designed for mobile rover navigation. The system extends the VLMaps paradigm by integrating a dual-memory architecture and confidence-aware data fusion mechanisms.

1. Dual-Memory Architecture:

   • Short-Term Memory (STM): A dynamic grid map that aggregates recent observations. It filters noise through temporal decay and consistency checks, handling uncertain or transient data.
   • Long-Term Memory (LTM): A persistent map storing reliable semantic landmarks. Only cells from the STM that meet specific criteria for confidence, coherence, and viewpoint diversity are promoted to the LTM.

2. Pipeline Components:

   • Embedding Extraction: Incoming RGB frames are divided into multi-scale tiles (coarse and fine). Each tile is encoded using a pretrained CLIP visual transformer (ViT-L/14) to obtain semantic embeddings aligned with natural language. This tile-level aggregation trades some spatial precision for higher throughput and resilience to camera motion compared to pixel-wise embeddings.
   • Spatial Projection: Using depth images and SLAM-estimated viewpoints, depth pixels are sampled, back-projected into 3D, and associated with fused tile-level embeddings. These point-wise observations are accumulated in a top-down spatial grid map.
   • Confidence-Aware Fusion: The system explicitly models perception uncertainty by assigning confidence values at both point and cell levels using three cues:
     • Geometric Confidence: Based on sensor distance and visibility; distant observations are down-weighted.
     • Semantic Confidence: Measures agreement between new embeddings and current cell evidence via cosine similarity in CLIP space.
     • Temporal Confidence: Applies time-based decay to reduce the influence of outdated observations, allowing the STM to adapt while the LTM remains persistent.
   • Fusion Mechanism: The STM performs confidence-weighted semantic fusion. Each cell maintains a semantic accumulator and a confidence accumulator. New observations are gated by semantic consistency before being aggregated.
   • Promotion Criteria: STM cells are promoted to LTM only if they exceed thresholds for accumulated confidence ($\tau_c$) and semantic coherence ($\tau_h$), and if they are supported by multiple viewpoints (viewpoint diversity). Coherence is calculated as the norm of the mean of normalized embeddings within a cell, estimating angular variance.

3. Querying:
   Natural language queries are encoded into the same CLIP space as map cells. A spatial heatmap is generated by weighting the cosine similarity between the query and cell embeddings by the cell’s coherence. This ensures that semantically consistent regions remain visible even at lower confidence levels, while high-confidence cells act as reliable landmarks.

Key Contributions

1. A confidence-aware open-vocabulary semantic mapping framework that assigns point- and cell-level confidence to CLIP-based observations using geometric, semantic, and temporal cues.
2. A dual-memory semantic map representation (STM and LTM) that aggregates observations into cell-wise embeddings and promotes only sufficiently confident, coherent, and multi-view-supported cells to a persistent long-term landmark map.
3. A real-time semantic mapping pipeline that fuses multi-scale, tile-level CLIP embeddings with SLAM-based spatial alignment to build language-queryable 3D maps under rover motion.
4. A confidence- and coherence-based promotion and querying mechanism that uses coherence to shape heatmaps and cell selection, while using cell-wise confidence to gate long-term promotion and navigation targets.

Results and Demonstration
The framework was implemented as a modular ROS 2 node and deployed on a Waveshare UGV rover equipped with an NVIDIA Jetson Orin GPU and an RGB-D camera. The system operates in real-time alongside an RTAB-Map SLAM pipeline.

• Proof-of-Concept: The authors demonstrated the system’s ability to build semantic maps and respond to natural language queries (e.g., "plant," "hammer").
• Observations: Visualizations showed that the STM tends to be noisy, reflecting recent and uncertain observations, while the LTM displays more sparse and reliable suggestions. The confidence mechanisms effectively suppress inconsistent observations in the heatmaps.

Significance
The paper claims that CrossMaps demonstrates the feasibility of combining open-vocabulary perception with confidence-aware fusion and dual-memory representation to mitigate perceptual noise in rover navigation. By separating uncertain observations (STM) from stable semantic structures (LTM), the system enables rovers to build language-queryable semantic maps in real-time. The authors position this work as a step toward more robust navigation in unstructured environments, with future work planned to integrate segmentation-based feature extraction and extend the system toward multi-agent sharing and neuro-symbolic reasoning.