💻 computer science

Real-Time RGB-D SLAM with Gaussian Scene Representation for Dynamic Environments

This paper proposes a lightweight, real-time semantic front-end that integrates periodic instance segmentation with a registration-driven Gaussian mapping pipeline to effectively remove dynamic objects and enhance the robustness of RGB-D SLAM in dynamic environments.

Original authors: Chae Min Lee, Hae Min Cho

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chae Min Lee, Hae Min Cho

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, 3D hologram of a room using a camera that sees both color and depth. This is the world of SLAM (Simultaneous Localization and Mapping), a superpower that lets robots and self-driving cars know where they are and what the world looks like around them. For a long time, these systems worked best in quiet, still rooms where nothing moved. But the real world is messy: people walk by, chairs get pushed, and dogs run across the floor. When a robot tries to map a moving object, it gets confused. It thinks the moving person is part of the wall, or that the wall is moving, leading to a shaky, glitchy map.

To fix this, scientists have started using a clever trick called Gaussian Splatting. Instead of building a map out of rigid blocks or wireframes, they build it out of millions of tiny, fuzzy, 3D "clouds" (Gaussians) that can be painted and shaped to look like real objects. This makes the map look incredibly realistic and lets the robot see the world from any angle instantly. However, even with these fancy clouds, if a person walks through the scene, the robot accidentally sprays "person-clouds" all over the walls, creating ghostly artifacts that ruin the map. The big question is: How do we keep the map clean and the robot steady without slowing it down or rebuilding the whole system?

The "Ghost-B-Gone" Filter for Robot Eyes

This paper introduces a clever new way to help robots map dynamic, busy environments without getting tripped up by moving people. The authors, Chae Min Lee and Hae Min Cho from Gachon University, propose a lightweight "front-end" filter that acts like a bouncer at a club, stopping dynamic objects (like people) from entering the robot's map in the first place.

The Problem: The "Ghost" in the Machine
Imagine you are trying to take a photo of a beautiful painting, but a friend keeps walking in front of the lens. If you keep taking pictures and stacking them together to make a 3D model, your friend will appear as a blurry, transparent ghost floating in front of the painting. In the world of robot mapping, this is exactly what happens. When a robot uses Gaussian Splatting to build a map, it tries to match what it sees now with what it saw a moment ago. If a person is moving, the robot gets confused. It tries to match the person's arm to the wall behind them, or it thinks the wall is moving. This creates "ghost" structures in the map and makes the robot's path (trajectory) wobble and drift, like a drunk sailor walking a straight line.

The Solution: The "Do Not Enter" Sign
The authors' solution is simple but effective: Don't let the moving stuff in.

Instead of trying to fix the robot's brain (the complex math that calculates where the robot is), they put a filter at the very beginning, right before the robot even starts building its map. Here is how it works, step-by-step:

The Watchful Eye (Segmentation): The robot uses a fast AI model (called YOLACT) to look at the camera image and instantly draw a mask around any person it sees. It's like the robot puts a "Do Not Enter" sign over every person in the room.
The Eraser (Depth Masking): The robot then looks at its depth sensor (which tells it how far away things are). Wherever the "Do Not Enter" sign is, the robot erases the depth data. It effectively says, "I see a person here, but I'm going to pretend this space is empty."
The Clean Build (Gaussian Mapping): Now, the robot builds its 3D Gaussian map using only the static stuff—the walls, the floor, the furniture. Because the moving people were erased before the map was built, they never become part of the 3D cloud. No ghosts, no glitches.

What They Found
The researchers tested this idea on several standard robot datasets, including tricky sequences where people were walking around desks and sitting in chairs.

Better Stability: In scenes with lots of movement, like a person walking across a room (the fr3_walking_xyz sequence), the new method reduced the robot's path error by a massive 67.0%. The robot stayed on track instead of drifting off course.
Cleaner Maps: When they looked at the 3D reconstructions, the baseline method (without the filter) showed "ghost" blobs of people stuck to the furniture. The new method produced clean, sharp maps where the furniture looked perfect, and the people were gone.
Speed Matters: The team was worried that adding this "bouncer" would slow the robot down. They found that the system still runs in real-time, averaging about 26.91 frames per second (FPS). That's fast enough for a robot to move and think at the same time.

The Catch: Don't Erase Too Much
The paper also points out a small flaw. If the robot is in a room where people are sitting very still (like the fr3_sitting_static sequence), the filter might accidentally erase them too. Since the robot needs some points to figure out where it is, erasing a sitting person can sometimes make the robot's position slightly less accurate than if it had just ignored the person. It's a trade-off: the filter is great for busy, moving scenes but needs to be careful not to be too aggressive when things are calm.

Why This Matters
This work suggests that you don't need to completely redesign a robot's brain to handle moving people. You just need a smart, lightweight filter at the front door. By removing dynamic objects before they mess up the math, the robot can build a stable, high-quality map of the world, even when people are running, walking, or dancing right in front of it. The authors show that this approach is a practical way to make robots more robust in the messy, real world without sacrificing speed.

Technical Summary: Real-Time RGB-D SLAM with Gaussian Scene Representation for Dynamic Environments

Problem Statement
Gaussian Splatting (GS) has emerged as an efficient explicit representation for Simultaneous Localization and Mapping (SLAM), enabling high-fidelity rendering and real-time throughput. However, most existing GS-based SLAM frameworks rely on a static-world assumption. In dynamic environments, independently moving objects introduce inconsistent geometric observations that violate the rigid-body assumption. In registration-driven GS SLAM, where camera pose is estimated via geometric alignment (e.g., G-ICP), dynamic objects generate spurious measurements and unreliable correspondences. These errors corrupt inter-frame registration, accelerate drift accumulation, and, when fused into the map, create persistent "ghost" artifacts that degrade long-term tracking stability and reconstruction quality. Suppressing these dynamic measurements prior to point cloud generation is identified as a practical approach to mitigate these effects without modifying the complex back-end optimizer.

Methodology
The authors propose a lightweight, input-level dynamic object removal framework integrated into a registration-driven Gaussian RGB-D SLAM pipeline. The system retains the original neural explicit SLAM back-end (tracking and mapping modules) and introduces a semantic front-end intervention. The pipeline operates as follows:

Semantic Instance Segmentation: For each incoming RGB frame, a real-time instance segmentation model (YOLACT) generates per-instance masks, semantic labels, and confidence scores. The system defines a set of removable dynamic classes (e.g., "person") and selects instances based on semantic labels and a confidence threshold.
Depth Masking: The selected instance masks are merged to form a dynamic-region mask. This mask is applied to the raw depth image to invalidate depth measurements in dynamic regions (setting them to zero).
Filtered Point Cloud Generation: A filtered point cloud is generated via pinhole back-projection using the masked depth map. Pixels with invalid depth (zero) are excluded, ensuring that only static geometric observations contribute to subsequent processing.
GS-ICP Tracking and Mapping: The filtered point cloud is fed into the standard GS-ICP tracker. The camera pose is estimated by registering the filtered points to the current 3D Gaussian map using a Generalized ICP (G-ICP) objective that leverages anisotropic uncertainty from Gaussian primitives. The mapper then incrementally updates the 3D Gaussian map by fusing only these static observations.

To maintain real-time performance under limited computational resources, the segmentation module is executed periodically (e.g., every $k$ frames), and the most recent mask is reused for intermediate frames, avoiding the latency of per-frame segmentation.

Key Contributions

Input-Level Dynamic Removal Framework: The paper introduces a semantic front-end for Gaussian-based SLAM that integrates real-time instance segmentation into a registration-driven mapping pipeline. This approach invalidates observations of user-specified dynamic classes before point cloud generation.
Analysis of Input-Level Filtering: The work analyzes the specific effects of this filtering on Gaussian-based SLAM, demonstrating that it improves correspondence reliability for G-ICP-based tracking and reduces dynamic-region contamination during Gaussian fusion.
Empirical Validation: Experiments on public dynamic RGB-D benchmarks (TUM RGB-D, Replica) and real-world indoor sequences demonstrate that the method improves trajectory robustness and reduces dynamic artifacts while preserving real-time throughput.

Experimental Results

Tracking Accuracy: On the TUM RGB-D benchmark, the proposed method significantly reduced the Absolute Trajectory Error (ATE) RMSE in highly dynamic sequences (e.g., a 67.0% reduction on fr3_walking_xyz compared to the baseline GS-ICP SLAM). In moderately dynamic scenes, improvements were also observed, though in near-static scenarios (e.g., fr3_sitting_static), over-masking occasionally reduced geometric constraints, leading to slightly higher error than the baseline.
Reconstruction Quality: Qualitative results showed that the proposed method effectively suppressed "ghost" artifacts and local reconstruction errors caused by moving objects, resulting in cleaner mesh reconstructions compared to the baseline.
Runtime Efficiency: The system maintained real-time performance (approximately 23–34 FPS depending on the sequence) despite the added segmentation overhead. An ablation study on the segmentation scheduling interval ( $k$ ) revealed that $k=5$ offered the best trade-off between mask freshness, segmentation overhead, and tracking accuracy.
Map Growth: The method reduced the rate of Gaussian map growth and GPU memory footprint by preventing dynamic measurements from being fused into the map.

Significance and Claims
The paper claims that its primary significance lies in enhancing the robustness of existing registration-driven Gaussian SLAM systems in dynamic environments without requiring modifications to the underlying SLAM optimizer or the introduction of new state variables. By isolating the dynamic object removal to the input level, the method provides a practical, lightweight solution that improves tracking stability and map consistency. The authors position this work as a complementary extension to current GS-ICP pipelines, demonstrating that selective dynamic object removal can yield substantial robustness gains in dynamic scenes while maintaining the computational efficiency required for real-time operation. The paper acknowledges limitations, noting that effectiveness depends on segmentation quality and that over-masking in near-static scenes can weaken geometric constraints, suggesting future directions for temporal mask consistency and adaptive removal criteria.

The "Ghost-B-Gone" Filter for Robot Eyes

Technical Summary: Real-Time RGB-D SLAM with Gaussian Scene Representation for Dynamic Environments

More like this