ExPrIS: Knowledge-Level Expectations as Priors for Object Interpretation from Sensor Data

Technical Summary: ExPrIS – Knowledge-Level Expectations as Priors for Object Interpretation

Problem Statement
While deep learning has significantly advanced robotic object recognition through data-driven approaches, these methods often lack semantic consistency and fail to leverage valuable, pre-existing knowledge about the environment. Purely data-driven models struggle with environmental context, particularly regarding occluded areas or objects not recently observed. Furthermore, existing approaches to constructing open-set 3D Semantic Scene Graphs (3DSSGs) typically rely solely on the power of underlying deep neural networks, ignoring prior observations from sensor data streams or higher-level knowledge. The core research question addressed is how to effectively process expectations derived from a knowledge level as priors within modern scene graph generation methods.

Methodology
The ExPrIS project proposes an architecture based on the incremental construction of a 3D Semantic Scene Graph (3DSSG) to integrate knowledge-level expectations. The methodology is structured around three main components:

1. Dynamic 3DSSG Architecture:

   • The system utilizes a heterogeneous graph organized in a hierarchy of multiple layers with varying node and edge types, ranging from low-level objects to high-level concepts (rooms, buildings).
   • Unlike rigid hierarchies (e.g., fixed object-place-room structures) or simple distance-heuristic connections, the proposed design is modular and extensible. It separates spatial from semantic relations and supports the dynamic inference of composite concepts.
   • For geometric representation, the system employs bounding volumes and sparse voxel grids rather than Truncated Signed Distance Function (TSDF) meshes. This choice prioritizes efficient storage, manipulation, and fast GPU-accelerated processing via PyTorch integration, which is sufficient for semantic reasoning and downstream tasks without the high computational overhead of dense reconstruction.
   • The graph is constructed incrementally from sensor data (RGB-D) in a bottom-up fashion, deriving structure from physical contact and support relationships between objects.

2. Knowledge-Based Expectation Generation:

   • The system defines "expectations" as additional information available prior to an inference step. Two types are integrated:
     • Context: Dynamic information from prior observations, specifically the global 3DSSG constructed from previous frames.
     • Knowledge: Static, general domain information sourced from external knowledge graphs, specifically ConceptNet.
   • To manage the noise and size of ConceptNet, the system extracts relevant subgraphs using a breadth-first search based on specific node classes and relationship types. It also implements graph embedding algorithms to generate feature vectors for message passing, offering alternatives to pre-computed embeddings.

3. Expectation-Biased Machine Learning:

   • The core learning mechanism employs a Heterogeneous Graph Neural Network (GNN), such as GraphSAGE or HGT.
   • The architecture features a global layer (aggregating historical observations and semantic priors) and a local layer (representing the current frame).
   • Cross-layer edges facilitate information flow, allowing the model to refine predictions for partially observed objects by leveraging accumulated context.
   • Global nodes are enriched with prior-based features (e.g., previously predicted labels, CLIP embeddings). These priors are integrated directly into the graph structure rather than encoded as fixed vectors.
   • The training objective focuses on a composite loss over node and edge predictions in the local graph. Crucially, the global graph serves as a biasing mechanism during inference, not as a direct supervision signal for the loss function.

Key Contributions

• Integration of Knowledge Priors: The paper presents a novel approach to embedding both contextual priors (from past sensor data) and semantic knowledge (from external graphs like ConceptNet) directly into the inference process of 3DSSG generation.
• Modular 3DSSG Design: It introduces a flexible, extensible layer design for 3DSSGs that moves beyond rigid hierarchies and simple distance heuristics, supporting dynamic inference and domain-specific abstractions.
• Efficient Geometric Representation: The adoption of bounding volumes and sparse voxel grids over TSDF meshes allows for scalable, efficient processing suitable for real-time robotic perception while maintaining sufficient fidelity for semantic tasks.
• Expectation-Biased GNN: The development of a learning framework where the global graph acts as an active, dynamic biasing mechanism during inference, enhancing robustness and semantic consistency over time.

Results and Evaluation
The paper does not present final experimental results from a deployed robot, as the work is currently in the stage of architectural design and offline analysis.

• Current Status: The 3DSSG implementation is modular. The expectation-biased GNN model has been evaluated in the context of the proposed architecture, with detailed results and code available in a separate publication [15] and on GitHub.
• Planned Evaluation: The authors outline a comprehensive evaluation plan to be conducted on the Mobipick mobile manipulator platform. The system will be tasked with a "tidy-up" scenario in a lab environment. This involves:
  1. Exploring the environment to build a semantic map.
  2. Identifying and relocating objects that are "out of place."
  3. Reasoning about both current and correct semantic contexts to resolve ambiguities.
• Hypothesized Outcomes: The authors hypothesize that the expectation-biased approach will demonstrate clear advantages over context-free, frame-by-frame models, specifically regarding improved semantic consistency and stability over long-term operation, even in the presence of partial observations or perceptual ambiguities.

Significance
The significance of the ExPrIS project lies in its attempt to bridge the gap between purely data-driven deep learning and knowledge-driven reasoning in robotics. By treating knowledge-level expectations as priors, the approach aims to move beyond static, frame-by-frame analysis to create a more robust and semantically consistent understanding of scenes over time. This integration is designed to support long-term autonomous operation, large-scale semantic mapping, and Explainable AI (XAI) by making the influence of knowledge on local predictions traceable. The project works in tandem with the sister project LIEREx, which incorporates vision-language models for open-vocabulary queries, aiming to combine symbolic knowledge with multimodal open-set perception.