💻 computer science

INHerit-SG: Incremental Hierarchical Semantic Scene Graphs with RAG-Style Retrieval

INHerit-SG is a novel framework that redefines semantic scene graphs as structured, RAG-ready knowledge bases with an explicit hierarchical hierarchy and event-triggered updates, enabling interpretable, robust, and scalable retrieval for complex robot navigation tasks through multi-role LLM reasoning.

Original authors: YukTungSamuel Fang, Zhikang Shi, Jiabin Qiu, Zixuan Chen, Jieqi Shi, Hao Xu, Jing Huo, Yang Gao

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: YukTungSamuel Fang, Zhikang Shi, Jiabin Qiu, Zixuan Chen, Jieqi Shi, Hao Xu, Jing Huo, Yang Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate your house.

The Old Way (The "Pixel Pile"):
Most current robots try to build a map by saving every single pixel of every photo they take, turning the world into a massive, dense 3D pile of points. It's like trying to find a specific book in a library where every book is glued together into one giant, solid block of paper. You can see the shape of the block, but you can't read the titles. If you ask the robot, "Find the non-black trash can near the green plant," it gets confused because it only sees a blob of colors, not the meaning of the objects.

The New Way (INHerit-SG): The "Smart Librarian"
The researchers behind INHerit-SG decided to stop building a "pixel pile" and start building a Smart Library.

Here is how their system works, broken down into simple concepts:

1. The Map is a Storybook, Not a Photo Album

Instead of just storing pictures, INHerit-SG writes a story about the room.

The Structure: It organizes the world like a human thinks: Floor → Room → Area → Object.
- Analogy: Think of a filing cabinet. You don't look through every single drawer in the building; you go to the 2nd Floor (Floor), open the Kitchen (Room), look at the Counter (Area), and find the Toaster (Object).
The Content: It doesn't just say "Object #402." It writes a note: "This is a green plant on the table." It uses natural language (English) as the primary way to label things, making it easy for the robot to understand human requests.

2. The "Two-Track" Construction Crew

Building this map while the robot is moving is hard. If the robot stops to think about every detail, it moves too slowly. If it moves too fast, it misses details.

The Fast Track (Geometry): One part of the system is like a fast runner. It just watches the walls and floors to know, "Okay, we are in a room, and we just turned a corner." It keeps the robot moving smoothly.
The Slow Track (Semantics): The other part is like a careful librarian. It waits for a good photo, then uses a powerful AI to analyze it and write the description ("That's a microwave").
The Magic: These two tracks work together asynchronously. The runner keeps the robot moving, while the librarian catches up in the background, adding details to the map only when necessary.

3. The "Event Trigger" (Don't Waste Energy)

Old systems update their maps constantly, even when nothing changes. It's like rewriting your entire diary every time you blink.

INHerit-SG's Approach: It only updates the map when something meaningful happens.
- Analogy: Imagine you are walking through a house. You don't need to update your mental map every time you take a step. You only update it when you walk through a door into a new room, or when you see a new piece of furniture appear. This saves a huge amount of brainpower (computing power).

4. The "Detective" Search (RAG-Style)

When you ask the robot, "Find the non-black trash can near the green plant," old robots just guess based on how similar the words look. They often get it wrong.

The INHerit-SG Detective:
1. Break it down: A special AI (LLM) acts like a detective, breaking your sentence into clues: Location: Floor 1, Target: Trash Can, Constraint: NOT Black, Relation: Near Plant.
2. Filter: It quickly eliminates anything that doesn't fit the clues (e.g., "No, that's a black trash can").
3. The Final Check (Visual Audit): Before the robot goes to the object, it takes a quick "peek" at the best photo it has of that spot. It asks a visual AI, "Does this picture actually show a non-black trash can near a plant?"
4. Result: If the picture matches, the robot goes there. If not, it keeps looking. This "closed-loop" check prevents the robot from going to the wrong place.

Why Does This Matter?

It's Human-Friendly: You can talk to the robot like you talk to a person.
It's Efficient: It doesn't need a supercomputer to remember every pixel; it just needs to remember the "story" of the room.
It's Reliable: By double-checking its answers with a visual audit, it rarely makes mistakes, even with tricky instructions like "Find the thing that is not red."

In summary: INHerit-SG turns a robot's memory from a messy pile of photos into a structured, searchable, and logical storybook that understands human language, allowing robots to navigate complex environments with the same common sense we use every day.

1. Problem Statement

Existing semantic scene graph approaches for robot navigation face three critical limitations when applied to embodied AI tasks:

Misalignment with Human Intent: Most methods rely on offline batch processing or implicit feature embeddings (dense point clouds/voxels). These lack explicit, interpretable natural language descriptions, making it difficult for robots to reason about vague, language-driven instructions (e.g., "find a non-black trash can").
Lack of Scalability and Efficiency: Flat representations (open-vocabulary feature fields) struggle with complex spatial reasoning and scaling in large environments. Conversely, structured hierarchical methods often incur high computational costs and storage redundancy due to heavy geometric reconstruction.
Fragile Retrieval: Current retrieval mechanisms typically use "open-loop" vector similarity matching. This approach is brittle against logical complexities like negation ("not the red one"), chained relations, and ambiguous constraints, often leading to false positives without explicit verification.

The authors argue that for effective embodied intelligence, a mapping system must be Structured (multi-level topology), Semantically Rich (grounded in human concepts), On-the-fly (incremental updates), and Interpretable (verifiable reasoning).

2. Methodology: INHerit-SG Framework

INHerit-SG reframes the map as a Retrieval-Augmented Generation (RAG)-ready knowledge base. Instead of storing dense point clouds, it stores a lightweight, hierarchical scene graph where natural language descriptions serve as explicit semantic anchors.

A. Asynchronous Dual-Process Architecture

The system decouples geometric tracking from semantic reasoning via two parallel streams:

Geometric Stream (Fast): Continuously processes RGB-D data to build a voxel-based occupancy map. It performs room segmentation (using Euclidean Distance Transform and watershed algorithms) and monitors vertical motion to instantiate Floor (L0) and Room (L1) nodes. It operates at ~2Hz.
Semantic Stream (Slow/Asynchronous): Operates on a queue of "keyframes" selected by a visual gating mechanism (based on DINOv3 feature similarity). It uses SAM3 for instance segmentation and VLMs to generate object descriptions. It instantiates Object (L3) nodes.
- Innovation: It uses a "floor-consistent asynchronous query" strategy, querying the latest geometric mask for the specific floor ID to ensure objects are assigned to the correct room even if the geometric map has updated since the frame was captured.

B. Hierarchical Construction & Event-Triggered Updates

The graph follows a Floor $\to$ Room $\to$ Area $\to$ Object hierarchy:

Functional Areas (L2): Generated by clustering objects within rooms and summarizing them via LLMs.
Event-Triggered Updates: Instead of updating the map at fixed time intervals, the system uses a VLM-based supervisor that analyzes a Bird's-Eye View (BEV) map. It triggers re-organization (topology updates, area summarization) only when significant semantic events occur (e.g., entering a new room, loop closure). This ensures long-term consistency with low overhead.
Incremental Association: Uses a two-stage cascade for merging new observations: strict geometric/visual filtering followed by semantic specificity checks (enforcing label consistency for known objects, high visual similarity for open-vocabulary objects).

C. Interpretable Closed-Loop Retrieval

The retrieval pipeline moves beyond simple embedding similarity:

Constraint Decomposition: Multi-role LLMs parse user queries into atomic constraints, extracting negations, spatial relations, and importance weights.
Hierarchical Filtering:
- Hard Filter: Immediate pruning based on Floor ID.
- Soft Filters: Scoring based on semantic similarity, weighted by user intent. Negation constraints are handled by inverting polarity scores ( $p_i \in \{1, -1\}$ ).
Visual Verification (Closed-Loop): Top-ranked candidates undergo a final VLM-based visual audit. The system crops the "Best View" image of the candidate and asks the VLM: "Does this match the query?" This eliminates false positives caused by feature-space ambiguities.

3. Key Contributions

INHerit-SG Framework: A hierarchical scene graph system that organizes maps as a structured, language-indexed knowledge base, bridging the gap between geometric mapping and human reasoning.
Event-Triggered Update Mechanism: A novel approach that decouples geometric tracking from semantic reasoning, updating the graph topology only upon meaningful semantic events, ensuring scalability and efficiency.
Interpretable Closed-Loop Retrieval: A retrieval pipeline that combines LLM-based logical constraint parsing with VLM-based visual verification, significantly improving reliability for complex queries involving negation and chained relations.
HM3DSem-SQR Dataset: A new benchmark dataset containing 6,084 queries across 36 scenes, specifically designed to test high-level reasoning, logical negations, and fine-grained retrieval.

4. Experimental Results

The system was evaluated on the HM3DSem-SQR dataset and in real-world environments (using a Unitree Go1 robot).

Retrieval Accuracy: INHerit-SG achieved State-of-the-Art (SOTA) performance.
- On HM3DSem-SQR, it achieved 37.7% accuracy (within 1m) and 36.3% semantic accuracy, outperforming baselines like ConceptGraphs, Embodied-RAG, and HOV-SG.
- In Human Studies (120 participants), it achieved a 70.6% semantic accuracy on complex queries, significantly higher than baselines (e.g., Embodied-RAG at 20.0%).
Resource Efficiency: By replacing dense point clouds with lightweight references (pointers to best-view images), the system reduced map size to 47.5 MB (34 MB without images), compared to 1.79 GB for HOV-SG and 18.47 GB for ConceptGraphs.
Real-World Validation: The system successfully executed multi-step navigation tasks (e.g., "Find the nearest green plant, then a non-black trash can...") in real office environments, demonstrating robustness to noise and dynamic layouts.
Ablation Studies: Confirmed that removing the hierarchy, VLM verification, or SAM3 features significantly degraded performance, proving the necessity of the full pipeline.

5. Significance

INHerit-SG represents a paradigm shift in robotic mapping from geometric reconstruction to semantic understanding.

Interpretability: By using natural language as the primary semantic anchor, the system's reasoning process becomes transparent and verifiable, addressing the "black box" nature of embedding-based retrieval.
Robustness to Logic: It is the first system to effectively handle complex logical constraints (negation, chaining) in embodied navigation, a critical requirement for real-world human-robot interaction.
Scalability: The event-triggered, incremental architecture allows robots to maintain long-term spatial memory without the computational burden of continuous global optimization or massive storage requirements.

This work paves the way for robots that can truly understand and act upon complex, natural language instructions in dynamic, unstructured environments.