💻 computer science

ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Exploration

The paper introduces ObsGraph, a hierarchical observation representation that unifies scene modeling, retrieval, and adaptive multi-scale exploration to enhance robotic performance in complex environments by organizing visual evidence into room-view-object layers and using retrieval outcomes to guide targeted information gathering.

Original authors: Taekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taekbeom Lee, Youngseok Jang, Jeonghwa Heo, Jeongjun Choi, H. Jin Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to solve a mystery in a huge, unfamiliar house. Its job is to find specific clues to answer a question, like "Is the toilet seat open?" or "What's on the kitchen counter?"

The problem is that the house is too big to look at everything at once, and the robot can't remember every single photo it has ever taken. If it tries to remember everything, it gets overwhelmed. If it tries to remember too little, it forgets the important details.

The authors of this paper, ObsGraph, propose a clever new way for robots to organize their memory and decide where to look next. Think of it as giving the robot a smart, three-layered filing system instead of a giant, messy pile of photos.

Here is how it works, broken down into simple parts:

1. The Three-Layer Filing System

Instead of just dumping all photos into one folder, ObsGraph organizes them into three distinct layers, like a set of Russian nesting dolls:

Layer 1: The Rooms (The Big Picture)
Think of this as the Table of Contents. The robot first figures out, "Am I in the kitchen? The bedroom? The bathroom?" This layer doesn't store every detail; it just keeps track of the general types of rooms the robot has seen. It acts like a map that says, "The clues about the bed are in the Bedroom folder."
Layer 2: The Views (The Context)
This is the Photo Album. Inside the "Bedroom" folder, the robot doesn't just store random pictures. It stores specific "views" that show how objects relate to each other. For example, it keeps a photo where you can see both the bed and the window together. This helps the robot understand that "the window is next to the bed." It's smart about picking photos so it doesn't waste space on 50 pictures of the same wall, but it keeps the ones that show unique combinations of things.
Layer 3: The Objects (The Magnifying Glass)
This is the Close-up Detail. If the robot needs to know if a specific item is a pillow or a blanket, it zooms in. This layer stores tight, cropped images of individual objects. It's like having a magnifying glass ready to inspect the fine details that might be blurry in a wide-angle photo.

2. The Detective's Strategy (Retrieval)

When the robot gets a question, it doesn't panic and search through every single photo. Instead, it uses a coarse-to-fine strategy, like a detective narrowing down suspects:

Check the Table of Contents: "The question is about a 'bed.' Let's look at the Bedroom folder first."
Check the Photo Album: "Okay, inside the Bedroom, which photos show the bed clearly?"
Check the Magnifying Glass: "Now, let's zoom in on the grey item on the bed to see if it's a pillow or a blanket."

This saves time because the robot only looks at the relevant "files" instead of the whole library.

3. The "Where to Look Next" Plan (Exploration)

This is the most creative part. If the robot looks through its files and realizes, "I don't have enough clues yet," it doesn't just wander aimlessly. Its filing system tells it exactly what kind of search to do next:

Room Exploration: If the robot realizes it hasn't even entered the "Kitchen" yet, it knows it needs to go find a new room.
View Refinement: If the robot is in the right room but the photo is too far away or at a bad angle, it knows it needs to move closer, step back, or turn around to get a better view of the same spot.
Frontier Exploration: If it has no idea where the clue might be, it goes to the edge of what it knows to explore completely new territory.

Why This Matters

The paper shows that by organizing memory this way, the robot becomes much better at two things:

Answering questions correctly: It finds the right evidence without getting confused by similar-looking things (like mixing up a bedroom window with a living room window).
Being efficient: It doesn't waste energy taking 100 photos when 5 smart ones will do. It knows exactly when to zoom in, when to move rooms, and when to stop looking because it has found the answer.

In short, ObsGraph turns a robot's memory from a messy pile of snapshots into a structured, searchable library that actively guides the robot on where to look next to solve its task.

Technical Summary: ObsGraph

Problem Statement

Autonomous robots operating in complex, unfamiliar environments face a fundamental challenge: they must actively explore to identify and acquire information necessary for task completion. Current approaches struggle with the trade-off between scene representation and task-conditioned access.

Dense 3D representations (e.g., voxel maps) preserve rich geometric and semantic information but lack explicit abstraction, making selective extraction of task-relevant regions difficult and limiting scalability as exploration expands.
Scene graphs offer compact structural abstractions but often rely on premature abstraction before a task is specified, potentially discarding visual evidence crucial for future tasks.
Image-based reasoning preserves rich visual details but lacks explicit structural organization, making it difficult to reason about what level of additional information is needed.

Furthermore, existing methods often fail to adaptively select the exploration scope. Robots need to reason not only about what is missing but also at what level of abstraction (room, view, or object) additional information should be gathered to resolve task gaps efficiently.

Methodology: ObsGraph

The authors propose ObsGraph, a unified framework that tightly integrates observation-centric scene representation, hierarchical retrieval, and adaptive exploration. The core innovation is a hierarchical scene graph that organizes visual observations into three distinct layers: Room, View, and Object.

1. Observation-Centric Scene Graph

Unlike methods that summarize environments into abstract properties, ObsGraph retains the visual observations themselves, organized hierarchically:

Room Layer: Serves as a coarse spatial anchor. It stores inferred room types and organizes views by room. Room assignments are updated via Bayesian inference using VLM predictions of room types and object affiliations, fusing direct predictions with object-level cues for robustness.
View Layer: Preserves object co-visibility as contextual information. Instead of storing all images, the authors propose an informative view selection strategy. This method uses agglomerative clustering based on object co-visibility to select a subset of images that captures diverse object relationships while avoiding visual redundancy. Representative views are assigned to clusters based on CLIP similarity, ensuring a compact yet context-rich layer.
Object Layer: Stores object-centric crops for each detected object. These provide fine-grained visual details that may be lost or ambiguous in the broader view layer.

The graph is updated incrementally as new observations arrive, integrating new objects and refining clusters while maintaining stable higher-level abstractions.

2. Hierarchical Retrieval

Given a task, the system performs a coarse-to-fine retrieval process using a Large Language Model (LLM) with a bounded query budget.

The retrieval traverses the hierarchy (Room $\to$ View $\to$ Object), progressively narrowing the search space.
At each layer, the LLM selects up to $k$ relevant nodes based on task relevance.
The candidate space for the next layer is restricted to nodes structurally connected to the selected nodes, enabling multi-branch reasoning without exhaustive LLM calls.
This process retrieves specific room types, views, and object crops that form the evidence base for the task.

3. Adaptive Exploration Strategy

The retrieval outcomes directly structure the exploration candidate space. If the retrieved evidence is insufficient to complete the task, the system activates exploration options corresponding to the abstraction level where the gap was identified:

Room Exploration: Triggered if a relevant room is identified but unexplored regions exist within it. The agent targets unexplored clusters within that room.
View Refinement: Triggered if a relevant view is identified but the observation is insufficient (e.g., due to scale, occlusion, or angle). The agent selects strategies such as "closer view," "wider context," or "different angle" to acquire richer evidence.
Frontier Exploration: Serves as the default option for gathering broader scene observations when no specific level provides relevant information.

A Vision-Language Model (VLM) reasons over the structured options and the retrieved evidence to select the most appropriate exploration target, decoupling high-level semantic reasoning from low-level coordinate computation.

4. Termination Check

A separate VLM query determines whether the accumulated evidence is sufficient to answer the task, preventing unnecessary exploration.

Key Contributions

Task-Agnostic Observation-Centric Scene Graph: A hierarchical representation (Room-View-Object) that preserves rich visual evidence while structuring it for task-aware reasoning.
Informative View Selection: A clustering-based strategy to construct a compact view layer that preserves diverse object co-visibility patterns while minimizing redundancy.
Scene Graph-Guided Retrieval: A coarse-to-fine retrieval scheme that leverages hierarchical structural and semantic context to identify missing task-relevant information.
Adaptive Exploration: A strategy that dynamically adjusts the exploration scale (room, view, or frontier) based on the abstraction level of the identified information gaps.

Experimental Results

The authors evaluated ObsGraph on three benchmarks: EM-EQA (Episodic Memory), A-EQA (Active Exploration), and GOAT-Bench (Goal-Oriented Navigation).

EM-EQA (Representation): ObsGraph achieved 61.5% LLM-Match accuracy, outperforming the state-of-the-art 3D-Mem (57.2%) and other baselines. It demonstrated that preserving both visual evidence and structural context yields superior performance. Notably, it improved accuracy by 4.3 points over 3D-Mem while only increasing the average number of queried frames by 0.2.
A-EQA (Exploration): ObsGraph achieved the highest LLM-Match (54.5%) and LLM-SPL (45.0), surpassing 3D-Mem (49.6% / 37.0%). This indicates that the framework not only retrieves information better but also guides exploration more efficiently.
GOAT-Bench (Navigation): In goal-oriented navigation, ObsGraph achieved a 72.7% Success Rate (SR) and 51.5 SPL, outperforming all baselines including 3D-Mem (69.1% SR).

Ablation Studies confirmed that each layer (Room, View, Object) and each exploration option (Room, View Refinement, Frontier) contributes significantly to both accuracy and efficiency. Removing the view selection strategy or the hierarchy resulted in notable performance drops.

Significance and Claims

The paper claims that ObsGraph successfully unifies scene representation, retrieval, and exploration into a single task-aware loop. By structuring the exploration candidate space according to the retrieved abstraction level, the system enables targeted information gathering driven by identified evidence gaps.

The authors highlight that their approach addresses the limitations of prior work by:

Avoiding premature abstraction that discards useful visual evidence.
Providing explicit mechanisms to reason about what level of additional information is needed.
Enabling robots to adaptively switch between coarse (room) and fine-grained (view/object) exploration scales.

The paper concludes that while the approach relies on VLM reasoning (which may be sensitive to perception or language errors), it demonstrates that structured scene representation significantly improves the success and efficiency of embodied reasoning and exploration in complex environments. Future work is suggested to investigate more robust geometric reasoning integration and extension to dynamic, long-horizon tasks.