FOUND-IT: Foundation-model-first Task-driven 3D Scene Graphs with Granularity on Demand

Technical Summary: FOUND-IT

Problem Statement
Current 3D scene graph construction methods face two primary limitations: a reliance on calibrated depth sensors (stereo or RGB-D) and a lack of flexibility in semantic granularity. Existing real-time systems often require complex pipelines and fixed sensor setups, limiting deployment. Furthermore, they typically commit to a specific object granularity at mapping time or rely on a predefined, static list of tasks. This rigidity prevents robots from adapting their spatial representation dynamically; for instance, a robot cannot easily switch from navigating to a "stove" (coarse granularity) to manipulating a specific "knob" on that stove (fine granularity) without re-mapping or pre-defining the task. Additionally, many approaches struggle to generalize traversability (places) and region definitions across diverse indoor and outdoor environments without relying on structured geometric priors.

Methodology
The authors propose FOUND-IT (Foundation-model-first Task-driven 3D Scene Graphs with Granularity on Demand), a system that constructs hierarchical, open-set 3D scene graphs using only uncalibrated monocular video in real-time. The system is built upon Geometric Foundation Models (GFMs), specifically leveraging VGGT-SLAM 2.0 as its geometric backbone.

The architecture consists of four main layers:

1. Geometric Mapping: The system processes a stream of monocular images, designating keyframes based on disparity. These are fed into a GFM to generate dense depth maps, poses, and camera intrinsics. Submaps are chained and globally optimized using a factor graph on the SL(4) manifold. To manage memory, the system optionally sparsifies the map via voxelization, keeping the overall map sparse while allowing queried objects to be represented densely.
2. Objects Layer (Visual Memory & Cache): Instead of pre-extracting object instances, FOUND-IT employs a two-stage memory system.
   • Visual Memory: Keyframes are passed through a CLIP-based Perception Encoder to create semantic embeddings. This defers object formation until a query is received.
   • Query-Time Formation: Upon receiving a text query, the system computes cosine similarity between the query embedding and keyframe embeddings. Relevant keyframes are then processed by SAM3 to generate 2D segmentation masks, which are back-projected into 3D using the GFM's depth.
   • Cached Memory: Resolved objects are stored as point clouds with 3D oriented bounding boxes. Subsequent queries check this cache first, enabling incremental, task-driven map building without re-computation.
3. Places Layer (Traversability): To map traversable space without complex geometric clustering, the authors append a ground segmentation head to the GFM. Through empirical analysis, they determined that intermediate layers of the GFM (rather than the final layer) provide the best features for ground segmentation. This head predicts ground masks, which are converted into a sparse graph of square tiles. This graph supports loop closures and dynamic deformation.
4. Regions Layer: Regions (e.g., "kitchen," "parking lot") are not pre-partitioned but are query-driven subsets of the places graph. Each place node summarizes its observations using a von Mises-Fisher (vMF) distribution of semantic embeddings, capturing both a mean direction and a concentration parameter. An open-vocabulary query is scored against these distributions, weighted by observation confidence. A confidence-aware graph label propagation smooths scores across the places graph, and a Gaussian mixture model separates in-region from out-of-region places.

Agentic Integration
The system is designed to be orchestrated by an LLM agent. Unlike prior agentic systems that retrieve from a pre-built, fixed representation, the FOUND-IT agent actively constructs the scene graph incrementally. It queries the visual memory for relevant frames, extracts objects via SAM3, and updates the cache, effectively building the representation "on demand" based on the task.

Key Contributions

1. Task-Driven Granularity: A novel approach to 3D scene graph construction that determines object and region granularity at query time rather than mapping time, supporting dynamic task evolution without a predefined task list.
2. Monocular Foundation-Model-First: The first method to build hierarchical 3D scene graphs using uncalibrated monocular cameras by leveraging GFMs, eliminating the need for depth sensors or complex calibration.
3. Ground Segmentation via Intermediate Layers: A demonstration that intermediate layers of GFMs are optimal for ground segmentation, enabling robust traversability mapping in diverse indoor and outdoor environments.
4. Query-Time Clustering: An efficient algorithm for clustering places into open-set regions using vMF statistics and graph propagation, allowing for flexible region definitions (e.g., a "playroom" within a "living room") based on the specific task.
5. Agentic Construction: An integration where an LLM agent incrementally builds the scene graph representation based on incoming queries, rather than querying a static map.

Experimental Results
The authors evaluate FOUND-IT across several benchmarks and real-world scenarios:

• Clio Dataset (Open-Set Object Extraction): FOUND-IT achieved top performance across all scenes (cubicle, apartment, office) in terms of open-set recall and IoU, outperforming both task-driven baselines (like Clio and Bayesian Fields) and task-agnostic open-set methods.
• ASHiTA SG3D Benchmark (Task Grounding): The system demonstrated a 79% improvement in accuracy over the previous state-of-the-art (ASHiTA) for grounding task descriptions to object locations.
• HOV-SG Benchmark (Region Clustering): The method achieved competitive precision and recall against indoor-specific methods (HOV-SG, Hydra) and exceeded them in semantic accuracy, despite relying less on indoor structural priors.
• Real-Time Performance: On a NVIDIA Jetson Thor mounted on a Spot quadruped robot, the system runs at 4 Hz using a lightweight GFM (Depth Anything). On a desktop GPU (RTX 3090), the full pipeline runs at 6–7 Hz.
• In-the-Wild Generalization: The system successfully constructed scene graphs from casually captured YouTube videos of realtor apartment tours, demonstrating robustness to uncalibrated, variable-quality inputs.

Significance and Claims
The paper claims FOUND-IT represents a fundamental shift toward spatial intelligence in robotics by decoupling map granularity from the mapping process. By utilizing foundation models, the system achieves plug-and-play 3D reconstruction from monocular video while supporting open-set, task-driven reasoning. The authors emphasize that this approach allows robots to handle complex loco-manipulation tasks where the required level of detail evolves dynamically. The work highlights the potential of leveraging intermediate features of foundation models for specific downstream tasks (like ground segmentation) and demonstrates that high-fidelity, hierarchical scene understanding is achievable without expensive sensor suites or rigid, pre-defined task schemas.