💻 computer science

GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory

This paper introduces HIOcc, a hierarchical benchmark for indoor semantic occupancy, and GEM-Occ, a Gaussian Evidence Memory framework that fuses transient visual geometry into a persistent, hierarchical memory to enable scalable, long-horizon semantic mapping across connected indoor environments.

Original authors: Hu Zhu, Bohan Li, Xianda Guo, Hongsi Liu, Baorui Peng, Mingqi Yuan, Xin Jin, Wenjun Zeng, Chang Wen Chen

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hu Zhu, Bohan Li, Xianda Guo, Hongsi Liu, Baorui Peng, Mingqi Yuan, Xin Jin, Wenjun Zeng, Chang Wen Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a giant, unfamiliar house with your eyes closed, then opening them for a split second, then closing them again. Your goal is to build a perfect mental map of the entire house—knowing where the walls are, where the furniture sits, where the empty air is, and where you simply haven't looked yet.

This paper tackles the problem of teaching robots (embodied agents) to do exactly that: build a long-lasting, reliable memory of indoor spaces as they move through them. The authors introduce two main things: a new "training ground" for robots called HIOcc, and a new "brain" for the robots called GEM-Occ.

Here is a simple breakdown of how it works:

1. The Problem: The "Flashbulb" vs. The "Library"

Current robot vision systems are a bit like people with a bad memory. When a robot looks at a room, it might create a quick, blurry snapshot (a "flashbulb" memory) of what it sees right now.

The Issue: If the robot turns around and looks at the same wall again, old systems often get confused. They might think the wall is in a new place, or they might forget that the space between the robot and the wall is empty air. They struggle to connect small rooms into a whole building, and they often mix up "I haven't seen this yet" with "This space is empty."

2. The New Training Ground: HIOcc

To fix this, the researchers needed a better way to test robots. They created HIOcc (Hierarchical Indoor Occupancy).

The Analogy: Think of previous datasets as a collection of single, isolated photos of different rooms. HIOcc is like a giant, connected 3D blueprint that stitches together thousands of photos from different sources (like ScanNet and Matterport3D).
What it does: It covers everything from a single camera view, to a whole room, to an entire multi-story building. It forces robots to learn not just "what is in this corner," but "how does this corner connect to the next room?"

3. The New Brain: GEM-Occ

The researchers also built a new system called GEM-Occ (Gaussian Evidence Memory). This is the robot's new way of remembering.

The Old Way (Point Maps):
Imagine trying to remember a room by taking millions of tiny, sticky notes and sticking them on every surface you see.

The Flaw: If you walk around the room and look at the table again, you stick another layer of notes on top. Soon, you have a messy, thick pile of notes that doesn't look like a table anymore. You also have no notes for the empty air, so you don't know if you can walk through a space or if it's a wall.

The GEM-Occ Way (Gaussian Evidence):
Instead of sticky notes, imagine the robot uses fuzzy, glowing clouds (called "Gaussians").

Transient Evidence: When the robot sees something, it doesn't save the image forever. It creates a temporary "glow" that says, "I saw a chair here."
The Magic Merge: As the robot moves and sees the chair again, it doesn't just stack new glows. It blends them together. If the robot sees the chair from a new angle, the fuzzy clouds merge into a single, solid, 3D shape.
The "Free Space" Ray: Crucially, GEM-Occ also draws invisible "laser beams" (rays) through the empty air. If the robot looks at a wall, it knows the air before the wall is empty. It saves this as a "free space ray." This prevents the robot from thinking the empty air is a mystery or a wall.
The Library System: The memory is organized like a library.
- Local Cache: A quick note on the desk for what's right in front of you.
- Room Submap: A folder for the whole room.
- Building Graph: A master map connecting all the rooms.

4. Why It Works Better

The paper claims that by using these "fuzzy clouds" instead of sticky notes, and by explicitly marking empty air, the robot:

Stays Consistent: If it walks back into a room it's seen before, the map looks the same, not messy or doubled up.
Knows What's Empty: It clearly distinguishes between "I haven't looked there yet" (Unknown) and "I looked and saw nothing" (Free Space).
Scales Up: It can handle a whole building, not just a single room, without running out of memory.

Summary

In short, the authors built a new, massive test course (HIOcc) and a new, smarter way for robots to remember their surroundings (GEM-Occ). Instead of just stacking up blurry snapshots, the robot now builds a clean, organized, 3D mental model that knows the difference between a wall, a chair, empty air, and places it hasn't visited yet. This allows the robot to explore large, connected buildings without getting lost or confused by its own memories.

Technical Summary: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory

1. Problem Definition

The paper addresses the challenge of long-horizon embodied semantic occupancy mapping in indoor environments. While existing methods effectively handle single-frame local prediction or room-level online perception, they struggle with persistent, scalable mapping across connected indoor spaces (e.g., entire buildings).

Key limitations in current approaches include:

Transient vs. Persistent: Most methods treat visual geometry (e.g., pointmaps) as the map state itself. However, pointmaps are dense in the image plane but sparse and surface-biased in 3D, failing to explicitly encode observed free space, unknown regions, or temporal confidence. Naively accumulating them leads to duplicated surfaces and inconsistent semantics.
Lack of Hierarchical Benchmarks: Existing indoor benchmarks (e.g., Occ-ScanNet, EmbodiedOcc) focus on local volumes or single rooms. They do not jointly support high-fidelity perspective scans, panoramic observations, and connected building-scale environments under a unified hierarchical protocol.
Free Space vs. Unknown: Current systems often fail to distinguish between observed free space and unknown regions, leading to errors in causal updates when revisiting areas.

2. Methodology

The authors propose two main components: a new benchmark (HIOcc) and a novel mapping framework (GEM-Occ).

A. HIOcc: Hierarchical Indoor Occupancy Benchmark

HIOcc unifies three major datasets—ScanNet, ScanNet++, and Matterport3D—into a common sparse semantic occupancy format while preserving their native observation geometries.

Scope: Covers over 2,000 indoor scenes, 300,000+ frames, and 142.6K m².
Three Evaluation Regimes:
1. Local Semantic Occupancy Prediction: Single-view prediction from posed RGB-D frames.
2. Room-Level Online Mapping: Causal fusion of observations within a single room.
3. Building-Level Mapping: Long-horizon mapping across connected panoramic environments (Matterport3D).
Annotation Pipeline: Converts annotated 3D meshes into sparse semantic occupancy targets. It employs visibility-aware filtering (frustum filtering for perspective, first-hit ray filtering for panoramas) to ensure targets reflect only observable geometry, distinguishing occupied, free, and unknown states.

B. GEM-Occ: Gaussian Evidence Memory Framework

GEM-Occ treats local visual geometry predictions not as the map state, but as transient evidence to be fused into a persistent memory.

Evidence Conversion:
- Semantic Gaussian Occupancy Evidence: Local geometry predictions (depth/pointmaps) are converted into anisotropic Gaussian primitives ( $g_i$ ) representing occupied surfaces. These include semantic distributions and confidence weights.
- Free-Space Ray Evidence: Regions along the viewing ray before the first surface hit are recorded as negative evidence (free space), while regions behind the hit remain unknown unless observed later.
Hierarchical Memory Structure:
- The map is organized into local caches, room-level submaps, and a building-level connectivity graph.
- This avoids dense global voxel tensors, enabling scalability.
Causal Fusion:
- Incoming evidence is matched to existing memory Gaussians using spatial or Mahalanobis distance.
- Visibility- and Uncertainty-Aware Updates: The system performs confidence-weighted fusion. Crucially, free-space rays provide negative log-odds evidence to suppress "floaters" in observed free space without erasing valid occluded geometry.
- Pruning and Merging: Low-confidence or redundant Gaussians are periodically pruned or merged to control memory growth.
Query Mechanism: Occupancy is queried via Gaussian-to-occupancy splatting, aggregating nearby Gaussians to compute occupancy probability and semantic distributions at any 3D location.

3. Key Contributions

Formulation: Defined hierarchical semantic occupancy mapping as an embodied memory problem requiring causal maintenance of occupied, free, unknown, and semantic states across local, room, and building scales.
Dataset (HIOcc): Introduced a large-scale benchmark unifying ScanNet, ScanNet++, and Matterport3D with protocols for local, room-level, and building-level evaluation.
Framework (GEM-Occ): Proposed a framework that converts transient visual geometry into persistent semantic Gaussian memory, explicitly modeling free-space ray evidence to distinguish observed empty space from unknown regions.
Fusion Strategy: Designed a visibility- and uncertainty-aware causal fusion rule that improves temporal stability and revisit consistency.

4. Experimental Results

Experiments were conducted on HIOcc across three settings:

Local Prediction: GEM-Occ achieved 61.37 IoU and 57.76 mIoU, outperforming strong baselines like GPOcc (60.69 IoU, 55.08 mIoU) and SplatSSC. The improvement is attributed to the use of semantic Gaussian evidence over direct pointmap voxelization.
Room-Level Online Mapping: GEM-Occ improved IoU to 56.79 and mIoU to 46.20, surpassing GPOcc (52.94 IoU, 44.81 mIoU). The method demonstrated superior stability in sequential observations.
Building-Level Mapping: On Matterport3D, GEM-Occ achieved 53.8 IoU and 46.7 mIoU, with a Revisit Consistency of 86.2%.
- Scalability: The hierarchical design reduced memory usage from 1.36 MB/m² (flat Gaussian memory) to 0.81 MB/m² and query latency from 49.5 ms to 33.8 ms, demonstrating a better accuracy-scalability trade-off.

Ablation Studies confirmed that:

Replacing semantic Gaussians with pointmap voxelization significantly degrades performance.
Explicit free-space ray evidence is critical for online mapping and revisit consistency.
Hierarchical memory is essential for scalability, reducing memory footprint without sacrificing local accuracy.

5. Significance and Claims

The paper claims that GEM-Occ represents a shift from treating occupancy as a per-frame prediction task to a persistent memory problem. By decoupling transient visual evidence from the persistent map state and explicitly modeling free space, the method achieves:

Improved Revisit Consistency: The map remains stable when the agent returns to previously visited areas.
Scalability: The hierarchical structure allows mapping of connected buildings without the computational cost of dense global voxels.
Robustness: The distinction between free and unknown space prevents the accumulation of errors in occluded regions.

The authors note limitations, including reliance on existing dataset annotations (which may contain noise), focus on static scenes, and the coarse 11-class label space. They suggest future work should address dynamic environments and evaluate the impact of these maps on downstream embodied tasks like navigation and manipulation. However, the current results demonstrate that hierarchical Gaussian memory is a viable path toward scalable, long-horizon embodied scene understanding.