💻 computer science

RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots

This paper introduces a fully visual, hardware-agnostic framework for the active, incremental construction of 3D scene graphs using only RGB inputs, which unifies perception and planning to achieve depth-parity performance and significantly outperform geometric baselines in object detection through semantic-driven viewpoint selection.

Original authors: Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Giorgia Modi, Davide Buoso, Giuseppe Averta, Daniele De Martini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to understand a messy room. Usually, to do this, the robot needs expensive, special "3D eyes" (like LiDAR or depth cameras) that can measure exactly how far away everything is. This paper introduces a new way for robots to build a mental map of a room using only standard video cameras (the kind found on your phone or laptop), without needing those special depth sensors.

Here is the breakdown of their approach using simple analogies:

1. The Problem: The "Blind" Map vs. The "Smart" Map

Old Way: Traditional robots build a map that is like a dense fog. They know exactly where walls and floors are (geometry), but they don't really know what things are or how they relate. It's like knowing a chair is in a spot, but not knowing it's a chair, or that it's next to a table.
The Limitation: Most of these systems also just "walk around" randomly or follow a pre-set path. They don't ask, "Hey, I can't see behind that door; I should go look there!" They just keep collecting data passively.
The Hardware Issue: Because they need special depth sensors, they can't be used on cheap robots or in places where only regular security cameras exist (like a fixed camera on a ceiling).

2. The Solution: The "Detective" Robot

The authors built a system that acts like a detective rather than a surveyor. Instead of just measuring distances, it tries to understand the story of the room.

RGB-Only Vision: The robot uses standard video. It uses advanced AI (like "MapAnything") to guess the 3D shape of the room just by looking at 2D pictures, similar to how a human can guess the depth of a room just by looking at a photo.
The Scene Graph (The Mental Web): Instead of a foggy map, the robot builds a web of connections.
- Nodes: It identifies objects (e.g., "a red chair," "a coffee table").
- Edges: It figures out relationships (e.g., "the chair is next to the table," "the lamp is on the table").
- This is like a family tree for the room, showing not just who is there, but how they are related.

3. Active Exploration: Asking "What's Behind the Door?"

This is the "Active" part of the title.

The Old Way (Geometric): A robot using old methods looks for "frontiers"—places where the map ends. It says, "I see a wall here, so I'll go look at the empty space next to it." It doesn't care if there's a hidden cat behind a curtain.
The New Way (Semantic): The robot uses its "web of connections" to guess what it doesn't see. If it sees a kitchen sink, it might guess, "There's probably a fridge nearby." It then actively moves to check that specific spot.
The Result: In tests, this "smart" robot found more than twice as many objects as the "dumb" geometric robot in the same amount of time. It was like a detective solving a mystery by following clues, rather than just walking around a building hoping to bump into something.

4. The "Eyes in the Sky" (External Cameras)

The paper also tested using fixed cameras (like security cameras on a wall) as extra helpers.

The Analogy: Imagine the robot is a person walking into a dark room. If someone on a balcony above shines a flashlight down, the person can instantly see the layout of the room without having to walk around to find the light switch.
The Result: Even without the robot moving, just adding one fixed camera view helped the robot build a much better initial map. It "bootstrapped" the process, giving the robot a head start.

Summary of Results

Accuracy: Using only video cameras, the robot built a map just as accurate as robots using expensive depth sensors.
Efficiency: By using "logic" (what objects usually go together) to decide where to look next, the robot found objects much faster than robots that just looked for empty space.
Versatility: The system works with any camera, whether it's on the robot's head or fixed on a wall, making it cheaper and easier to deploy in real-world homes or offices.

In short, this paper shows that robots don't need expensive 3D sensors to understand a room. If they are smart enough to use a "web of relationships" to guide their curiosity, standard video cameras are enough to build a complete and useful mental map.

Technical Summary: RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots

Problem Statement

Current approaches to 3D Scene Graph (3DSG) generation face two primary limitations that hinder their deployment in diverse robotic and infrastructure settings:

Hardware Dependency: State-of-the-art methods rely on dedicated depth sensors (LiDAR or RGB-D cameras) for metric 3D reconstruction. This restricts deployment to specialized platforms, excluding scenarios where only commodity RGB cameras are available, such as fixed external surveillance infrastructure, low-cost robots, or handheld devices.
Passive Observation: Existing pipelines typically operate on pre-recorded, passively collected trajectories. They fail to actively select viewpoints based on the partially built scene representation, thereby missing opportunities to exploit the semantic and spatial information encoded within the graph to guide exploration efficiently.

Consequently, there is a need for a framework that can construct 3DSGs using only RGB inputs while actively guiding the robot to maximize information gain.

Methodology

The authors propose a fully visual, hardware-agnostic framework that unifies perception and planning around a shared structured representation. The system operates as an incremental perception–action loop (Fig. 1) consisting of three core components:

1. RGB-only 3DSG Generation

The pipeline processes RGB images to incrementally update a point cloud and a scene graph ( $G_t = \{O_t, E_t\}$ ) without ground-truth depth.

Geometry and Pose Estimation: The system leverages MapAnything [11], a feed-forward model, to infer scene geometry from RGB inputs. Given $N$ images, it predicts per-view ray directions, up-to-scale depth maps, and camera poses relative to the first image, along with a global metric scale factor. These are composed to back-project 2D pixels into a metrically scaled 3D point cloud. Absolute poses are recovered by fixing the reference frame of the first input (e.g., an external camera or initial robot pose).
Open-Vocabulary Semantic Mapping: RGB images, estimated depths, and poses are fed into ConceptGraphs [8].
- Segmentation & Embedding: SAM [22] extracts class-agnostic instance masks, and CLIP [23] embeds these regions into semantic descriptors.
- 3D Association: Masks are projected into 3D using the estimated geometry, and multi-view associations merge detections into 3D object nodes ( $O_t$ ).
- Relation Inference: Unlike the original ConceptGraphs which uses proprietary LLMs for relationships, this work derives spatial edges ( $E_t$ ) via a deterministic, geometry-only procedure. It evaluates fixed-priority predicates (e.g., on top of, supported by, inside, next to) based on oriented 3D bounding boxes, assigning at most one edge per ordered pair to ensure reproducibility.

2. Active Semantic Perception (ASP)

To address the limitation of passive trajectories, the framework implements a Next-Best-View (NBV) selection strategy.

Geometric Baseline: The Surface Edge Explorer (SEE) [17] algorithm is used as a baseline, identifying frontier points based on measurement density to maximize surface coverage.
Semantic-Driven Exploration: The proposed ASP module utilizes the current 3DSG to guide exploration. An LLM samples plausible completions of the unobserved scene from the current graph. For each candidate viewpoint, the system computes the expected information gain (mutual information between the predicted observation and the current graph) using Shannon entropy. The viewpoint maximizing this gain is selected, prioritizing locations likely to resolve semantic ambiguities (e.g., areas behind doors).

3. Multi-View External Camera Integration

Because the framework is strictly visual, it can ingest observations from uncalibrated fixed external RGB cameras alongside onboard robot cameras. These external views are processed identically to egocentric images, allowing them to provide a broad initial estimate of the scene (bootstrapping) and update the graph during exploration without requiring robot motion.

Key Contributions

RGB-only Pipeline: A novel pipeline for 3D scene graph generation that relies solely on RGB inputs, removing the dependency on depth sensors.
Active Semantic Loop: The integration of the 3DSG representation into an active exploration loop, enabling semantic-driven NBV selection that outperforms geometric baselines.
External Camera Utilization: An evaluation demonstrating that fixed external RGB cameras can effectively bootstrap and improve scene graphs, enhancing contextual understanding without additional exploration cost.

Experimental Results

The method was evaluated on the Replica and ReplicaCAD datasets across three settings:

1. Static Pipeline Validation (Replica)

Comparison: The proposed RGB-only variant (CG-RGB) was compared against the original ConceptGraphs (CG) using ground-truth depth.
Result: The RGB-only pipeline achieved F1-score parity (0.500 vs. 0.499) with the ground-truth baseline. While precision slightly decreased due to noisier depth predictions, recall improved, suggesting the system avoids overly aggressive merging of closely situated objects.

2. Active Exploration (ReplicaCAD)

Setup: Experiments were conducted in the Habitat simulator over 30 exploration steps.
Result: The semantic-driven ASP framework significantly outperformed the geometric SEE baseline.
- Object Detection: ASP detected ~110 objects (approaching the ground truth of 124), whereas SEE stalled at ~45.
- Recall: ASP achieved double the recall of SEE (0.54 vs. 0.22).
- Efficiency: ASP matched the quality of hundreds of passive trajectory frames in just 30 intelligent movements.

3. External Camera Integration

Bootstrapping: Initializing the scene graph with a single overhead external camera increased the starting node count for SEE by 125% and ASP by 57%, with corresponding improvements in initial recall.
Standalone Infrastructure: In experiments using only fixed external cameras (1–3 cameras) without a mobile robot:
- In furnished rooms, 3 cameras achieved an F1-score of 0.516, recovering over 55% of ground-truth objects instantly.
- In complex apartments, 3 cameras achieved an F1-score of 0.421.
- While static viewpoints cannot match the recall of active 30-step exploration, they successfully establish the environment's dominant structure.

Significance and Claims

The paper claims that active robot perception benefits more from structured semantic representations than from geometry alone. By combining geometry, semantics, and structure, 3D scene graphs provide a richer perceptual model that is naturally queryable by language models, acting as spatial reasoning engines for partially observed environments.

The removal of depth-sensor dependency offers practical advantages:

Hardware Agnosticism: The system supports heterogeneous visual inputs, enabling deployment on low-cost robots and integration with existing surveillance infrastructure.
Reduced Exploration Cost: External cameras can bootstrap the scene graph, reducing the need for extensive robot exploration while improving scene understanding.
Safety and Planning: Complementary views can reveal obstacles and free space before they are visible to the onboard camera, reducing both geometric and semantic uncertainty for safer motion planning.

Ultimately, the work highlights the potential of semantics-first representations as a foundation for reliable and scalable robot autonomy in real-world deployments where specialized sensors are unavailable.