Technical Summary: A Scene Graph Backed Approach to Open Set Semantic Mapping

Problem Statement

Autonomous mobile robots operating in large-scale, real-world environments require semantic understanding that extends beyond pure geometric mapping to perform complex tasks. While volumetric representations like Truncated Signed Distance Functions (TSDFs) excel at geometric integration, they lack an explicit model of scene structure, often decoupling perception from high-level representation. This limitation hinders the modeling of hierarchical and semantic relationships necessary for advanced reasoning.

Furthermore, existing approaches to 3D Semantic Scene Graphs (3DSSGs) typically treat the scene graph as a derivative layer generated post hoc after the mapping process is complete. This separation limits system consistency over extended operations, prevents online interactions, and creates a gap between sub-symbolic sensor data and symbolic reasoning tools like Large Language Models (LLMs) or knowledge graphs. Additionally, many current methods rely on rigid heuristics, fixed ontologies, or purely data-driven bottom-up approaches that struggle with scalability and generalization to open-set vocabularies.

Methodology

The authors propose a mapping architecture where the 3DSSG serves as the foundational backend and primary knowledge representation for the entire mapping lifecycle, rather than a post-processing output. The system is designed for incremental, online integration of high-bandwidth sensor data using a fully GPU-accelerated pipeline.

1. Multi-Layered 3DSSG Backend

The architecture organizes the scene graph into three distinct abstraction layers:

• Frames Layer: Tracks 6D pose information and raw 2D segmentation masks for all processed keyframes.
• Segments Layer: An intermediate representation tracking active 3D object fragments, including raw geometry and extracted feature vectors. It supports lazy, on-demand merging strategies.
• Objects Layer: Tracks persistent, consolidated object instances, maintaining spatio-semantic relations and accumulated geometric/semantic features.

2. Mapping Pipeline

The pipeline operates strictly incrementally, transitioning the 3DSSG from one valid state to the next:

• Segmentation & Feature Extraction: Incoming RGB-D frames are processed using a pre-trained Segment Anything Model (SAM) (specifically FastSAM for speed) to generate segmentation masks. Simultaneously, dense visual features are extracted using DINOv2. To avoid computational bottlenecks, the system utilizes local per-patch features from intermediate layers rather than aggregating features for separate object crops.
• Local Graph Generation: Depth data is projected into 3D point clouds for each segment, filtered via a custom CUDA-based DBSCAN implementation to remove noise, and voxelized. These are instantiated in a local 3DSSG.
• Data Association & Global Integration: A two-stage matching process integrates the local graph into the global graph:
  1. Greedy Association: Merges local segments with global segments based on Intersection over Union (IoU) and DINOv2 feature similarity. Unmatched segments become new nodes.
  2. Active Refinement: Immediately follows greedy merging to resolve over-segmentation. It considers only the active subset of the graph (modified or new nodes), matching them against spatial neighbors using voxel-grid overlap to fuse fragmented segments into coherent instances.
• Vision-Language Integration: To support open-vocabulary queries, CLIP features are integrated using the MaskCLIP paradigm. A gated mechanism modulates local patch features based on a global frame embedding to enhance semantic richness while retaining specific characteristics. View quality scores act as confidence weights during aggregation.

3. Incremental Predicate Prediction (IPP)

To address the lack of deep semantic meaning in graph edges, the authors integrate a prediction module based on a heterogeneous incremental graph model (Renz, Igelbrink, and Atzmueller 2025).

• Structure: The model utilizes a local layer (current sensor frame) and a global layer (mapped scene graph).
• Prediction: A Heterogeneous Graph Neural Network (GraphSage) predicts semantic predicate labels (e.g., "on top of," "attached to") for edges between objects.
• Features: Node features include geometric descriptors (bounding box dimensions, volume, standard deviation) and DINOv2 features. Edges connect touching objects (determined via bounding box intersection with padding) and matched nodes between local and global layers.
• Integration: Predicted predicates are incrementally integrated into the global graph, serving as semantic priors for future steps.

Key Contributions

The paper presents three primary contributions:

1. Modular Mapping Architecture: A design establishing the 3DSSG as the foundational backend, moving beyond rigid hierarchies to support dynamic, modular layers that separate spatial and semantic relations using voxel-based representations for scalability.
2. Real-Time Graph Inference: A method for inferring and updating the graph structure in real-time by integrating bottom-up sensory data with top-down expectations, ensuring topological consistency during exploration.
3. Versatile Integrations: Demonstration of the framework through:
   • Dynamic Open-Vocabulary Queries: Incorporating Vision-Language Foundation Models (VLFMs) like CLIP for natural language interaction.
   • Real-Time Relation Prediction: Integrating a 3DSSG prediction network to dynamically infer structural relationships.
   • Real-World Deployment: Validation on a TIAGo mobile robot using real-world sensor data in large-scale environments.

Results

The authors conducted preliminary experiments to validate the framework:

• ICL Dataset: Qualitative results on the kt1 trajectory showed successful isolation of major structural elements. However, over-segmentation artifacts were observed in areas where the active refinement stage lacked sufficient evidence (e.g., consistent spatial overlap). Semantic accuracy was effective for distinct objects but degraded in highly cluttered areas or with small objects due to resolution constraints in the MaskCLIP approach.
• 3RScan Dataset: The integration of predicted predicates into the 3DSSG was visualized, showing the potential for enriching the graph with semantic relationships.
• Real-World Deployment (TIAGo): Using a mobile robot with an Ouster LiDAR and RGB-D camera, the system successfully integrated the majority of instances into a coherent map. While segmentation was less clean than in synthetic data (due to sensor noise and FastSAM over-segmentation), the system demonstrated robustness.
• Geometric Filtering: An experiment using Rmagine to filter out static building structures (floors, walls) effectively reduced spurious instances, mimicking panoptic segmentation.

Significance and Claims

The paper claims that by establishing the 3DSSG as the central data structure and "single source of truth," the proposed architecture bridges the gap between sub-symbolic raw sensor data and high-level symbolic reasoning. This explicit, spatially grounded representation provides a stable structure that knowledge-driven frameworks (from ontologies to LLMs) can directly exploit.

The authors emphasize that this approach enables agents to operate with enhanced interpretability, trustworthiness, and alignment to human concepts. By maintaining an explicit topology that supports both flat and hierarchical structures, the system facilitates open-set reasoning and the integration of external knowledge priors, which are difficult to achieve with purely implicit or geometric representations.

The authors remain modest regarding current limitations, acknowledging that the Incremental Predicate Prediction module currently relies on supervised learning and extensive ground-truth labeling (e.g., from the 3DSSG dataset), which restricts generalization to novel, unlabelled environments. Future work aims to address this by developing unsupervised geometric heuristics for spatial relation inference.