💻 computer science

Seeing Fast and Slow: Bimodal 3D Scene Graphs for Open-set Tasks

This paper introduces BiMoSG, a bimodal 3D scene graph generation framework that dynamically switches between a fast, coarse mode and a slow, fine-grained open-vocabulary mode to enable efficient, real-time open-set task execution.

Original authors: Marcel Bartholomeus Prasetyo, Shrutika Vishal Thengane, A Manicka Praveen, Yi Loo, Malika Meghjani

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marcel Bartholomeus Prasetyo, Shrutika Vishal Thengane, A Manicka Praveen, Yi Loo, Malika Meghjani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to navigate a messy room to find a specific item, like a lost set of keys or a bag of groceries. If the robot tried to examine every single object in the room with the same level of intense, microscopic detail, it would be like trying to read every word in a library to find one specific book. It would take forever, and the robot would run out of battery long before it finished.

This paper introduces a new system called BiMoSG (Bimodal 3D Scene Graph) that solves this problem by giving the robot a "fast brain" and a "slow brain," mimicking how humans think.

The Two Modes: Fast and Slow

The authors compare the robot's thinking process to the famous "System 1" and "System 2" thinking described in psychology:

The "Fast" Mode (System 1):
- How it works: This is the robot's default setting. It scans the room quickly and creates a rough, "coarse" map. It doesn't worry about the exact brand of a chair or the specific pattern on a rug. Instead, it sees things as simple shapes (like boxes or prisms) and gives them general names like "table," "chair," or "counter."
- The Analogy: Think of this like driving down a highway. You don't need to know the license plate number of every car you pass; you just need to know there's a car, a truck, or a tree. You see the "gist" of the world instantly.
- The Benefit: This mode is incredibly fast and uses very little energy. It allows the robot to move and explore without getting bogged down in details.
The "Slow" Mode (System 2):
- How it works: This mode only kicks in when the "Fast" mode spots something interesting. If the robot is looking for groceries and sees something that might be a refrigerator, it switches to "Slow" mode. It zooms in, uses advanced AI to read labels, and figures out exactly what the object is.
- The Analogy: This is like pulling your car over to the side of the road to read a specific street sign or check a map. You stop the general flow to focus intensely on one specific detail.
- The Benefit: This provides high accuracy for the specific items the robot actually needs to interact with, without wasting time on things that don't matter.

The "Prismatic" Shortcut

To make the "Fast" mode even quicker, the researchers invented a new way to represent 3D objects. Instead of building a detailed 3D model made of thousands of tiny dots (which is computationally heavy), they represent objects as simple prisms (like cardboard boxes).

The Analogy: Imagine trying to pack a suitcase. You don't need to know the exact curve of a sweater or the texture of a shoe. You just need to know that the sweater fits inside a box of a certain size. The robot does the same thing: it turns complex furniture into simple geometric boxes. This makes it much easier to calculate where things are and if they overlap, saving massive amounts of computing power.

How They Work Together

The system is designed so the robot spends 99% of its time in "Fast" mode, just cruising around and building a rough map. It only switches to "Slow" mode when a "trigger" happens—like when the robot's general map says, "Hey, there's a counter here, and the task is to find food."

Once the robot switches to "Slow" mode, it confirms the object is indeed a counter (and not a table), and then the robot can proceed with its task.

What the Results Show

The paper claims that this approach is three times faster than current state-of-the-art methods that try to be detailed all the time.

Speed: In tests, the "Fast" mode could generate a scene map in about 0.1 seconds per frame, while other methods took 0.4 seconds.
Success: The robot was able to complete tasks (like finding a trash can or a kitchen island) much faster than robots using only the "Slow" (detailed) method, while still achieving the same level of success.
Real World: They tested this on a real robot (a Clearpath Jackal) in a mock museum. The robot successfully used the "Fast" mode to scan the room and the "Slow" mode to identify a "backpack" as a suspicious object, proving it works outside of computer simulations.

The Bottom Line

The paper argues that robots don't need to be perfect at seeing everything all the time. By using a bimodal approach—being fast and general most of the time, and slow and detailed only when necessary—robots can navigate complex, unknown environments much more efficiently, saving time and battery life while still getting the job done.

Technical Summary: BiMoSG – Bimodal 3D Scene Graphs for Open-set Tasks

Problem Statement

Open-set task execution requires robots to navigate complex environments containing novel objects without prior information. While open-vocabulary 3D scene graphs (utilizing Large Visual Foundation Models like SAM, CLIP, and LLaVa) offer robust instance segmentation and classification, they face significant barriers to real-time deployment on mobile robots. These barriers include the substantial computational demands of foundation models, limited edge computing resources, and high energy costs, which can restrict operational time-spans.

The paper posits that generating high-granularity, open-vocabulary scene graphs for the entire duration of a task is unnecessary. Instead, robots often require only coarse semantic information to navigate and determine general functionality, reserving fine-grained, open-vocabulary analysis for specific regions likely to contain task-relevant objects. This approach mirrors the human cognitive distinction between fast, intuitive "System 1" thinking and slow, deliberate "System 2" thinking.

Methodology: BiMoSG

The authors propose BiMoSG, a bimodal 3D scene graph generation framework designed to switch seamlessly between a "fast" (coarse) mode and a "slow" (fine) mode based on task context.

1. Dual-Mode Architecture

Fast Mode (Default): Operates using a closed-vocabulary setup to generate a coarse 3D scene graph. It relies on Mask2Former (trained on ADE20K-150) for segmentation and a novel prismatic volumetric representation for objects. This mode avoids large foundation models, drastically reducing computation and energy costs.
Slow Mode (On-Demand): Triggered when the system identifies regions or objects likely relevant to the task. This mode employs Open-Vocabulary techniques, utilizing Vision-Language Models (VLMs) like SmolVLM2 and LangSAM to generate detailed, open-vocabulary descriptions and refine object labels.

2. Prismatic Volumetric Object Representation

To achieve speed in the "fast" mode, BiMoSG introduces a novel geometric abstraction:

Instead of processing dense point clouds (which are computationally expensive and sensitive to viewpoint changes), the system extracts object contours from segmentation masks.
These contours are projected into the world frame, denoised, and converted into polygonal prisms defined by an XY-polygon (convex hull or alpha shape) and a Z-range ( $z_{min}, z_{max}$ ).
This representation significantly reduces computational complexity for merging and deduplication ( $O(n+m)$ vs. $O(n^3)$ for voxelized point clouds) and improves volumetric overlap matching across sparse viewpoints.

3. Vicinity-Based Object Relations

The framework constructs a hierarchical scene graph using a vicinity-based approach:

Visibility Matrix: Tracks how often objects appear together in the same frame.
Hierarchical Partitioning: Objects are ordered by "girth" (cubic root of volume) and relative size to neighbors. A directed acyclic graph (DAG) is formed to establish "vicinities" (local clusters) and "zones" (room-level approximations).
Relation Generation: Natural language relations (e.g., ON, IN, BESIDE) are generated based on the proximity and intersection of object prisms within these vicinities.

4. LLM Planner Integration

A Large Language Model (LLM) Planner (specifically GPT-4o) acts as the cognitive bridge:

Phase 1 (Decomposition): Analyzes high-level tasks to generate "trigger labels" for closed-set objects.
Phase 2 (Monitoring): Monitors the scene graph. If a trigger label matches a detected object, it invokes an InspectAt action.
Phase 3 (Confirmation): The VLM refines the object label, and the LLM confirms the object's relevance to the task before downstream execution. This prevents false positives from visually similar but contextually irrelevant objects.

Two modes of open-set identification are supported:

Hybrid Mode: Uses CLIP embeddings for closed-set masks to implicitly handle open-set semantics.
Switching Mode: Explicitly switches to VLM-based segmentation (LangSAM) when the LLM requests detailed investigation.

Key Contributions

BiMoSG Framework: A novel 3D scene graph generation approach that dynamically switches between closed-set (fast) and open-set (slow) modes to balance efficiency and semantic richness.
Prismatic Volumetric Representation: A new geometric abstraction for 3D objects that enables fast merging and deduplication, reducing computational complexity compared to point-cloud-based methods.
Vicinity-Based Hierarchy: A novel method for establishing object relations and hierarchical partitions (vicinities and zones) based on visibility and relative size, facilitating structured reasoning.
Performance Validation: Empirical evidence showing the "fast" mode is approximately 3x faster than state-of-the-art open-vocabulary approaches (e.g., ConceptGraphs) while maintaining robustness.
Real-World Deployment: Validation of the framework on a Clearpath Jackal robot in a museum environment, demonstrating successful task execution (identifying suspicious objects) without prior mapping.

Experimental Results

Computational Efficiency: On an RTX 4090, BiMoSG's fast mode processes frames in an average of 0.1 seconds, compared to 0.4 seconds for ConceptGraphs and 0.2 seconds for BBQ.
Geometric Accuracy: Evaluation on the Replica dataset shows acceptable geometrical accuracy, with a Mean GT Coverage of ~57% and Linear mIoU of ~61%. Lower precision is attributed to segmentation artifacts rather than the volumetric representation itself.
Task Execution Time: In Habitat Matterport3D simulations, BiMoSG consistently outperformed the Clio baseline in both open-set and closed-set tasks. For example, in open-set tasks, BiMoSG reduced execution time from ~1:45 (Clio) to ~2.13 seconds for finding a keyboard, largely because it avoided full-environment mapping before task completion.
Convergence: In high-level tasks (e.g., "Where should I put ice cream?"), the system achieved high convergence rates (80–90%) with significantly reduced execution times (e.g., 19 seconds for the ice cream task).

Significance and Claims

The paper claims that BiMoSG offers a real-time, energy-efficient solution for open-set task execution by mimicking human cognitive efficiency. By defaulting to a coarse, closed-vocabulary representation and only engaging computationally expensive open-vocabulary models when necessary, the system enables:

Seamless integration of scene graph generation with task execution.
Robust operation in unknown environments without the need for advance mapping.
Significantly shorter task execution times compared to purely open-vocabulary methods, while achieving similar success rates.

The authors conclude that this bimodal design is competitive for real-time robot perception, navigation, and reasoning, effectively addressing the trade-off between the speed required for mobile robotics and the semantic depth required for open-set tasks.