💻 computer science

Build Once, Monitor Continuously: Persistent Semantic Mapping via Autonomous Exploration and Open-Vocabulary Object Updates

This paper presents a modular two-stage system that separates geometric mapping from semantic updating to enable efficient, persistent monitoring of indoor environments by building a 2D occupancy grid once and subsequently using lightweight open-vocabulary detection to continuously track object changes across multiple robot visits.

Original authors: Sai Haneesh Allu, Itay Kadosh, Tyler Summers, Yu Xiang

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sai Haneesh Allu, Itay Kadosh, Tyler Summers, Yu Xiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot tasked with keeping an eye on a large office building or a warehouse. Your job is to know where everything is, but here's the catch: the building itself (the walls, floors, and corridors) never changes, but the furniture and objects inside (chairs, tables, trash cans) get moved, added, or removed every day.

The problem with most robot mapping systems is that they act like a photographer who takes a brand-new, high-resolution photo of the entire building every single time they walk in. They rebuild the map from scratch, even though the walls are exactly the same. This is slow, wasteful, and inefficient.

This paper proposes a smarter way called "Build Once, Monitor Continuously." Think of it like this:

The Two-Stage Strategy

Stage 1: The One-Time Blueprint (Geometric Mapping)
Imagine you move into a new house. The first thing you do is walk through every room and draw a detailed floor plan of the walls, doors, and hallways. You do this once.

How the robot does it: The robot uses a "frontier-based" explorer. Think of this as a hiker who always walks toward the edge of the known world (the frontier) to find new territory. The paper introduces a "Dynamic Window" trick: if the robot sees lots of unexplored areas nearby, it stays local to finish them quickly. If the area is clear, it looks further ahead. This builds a reliable 2D map of the building's geometry (the walls and floors).

Stage 2: The Daily Check-In (Semantic Updating)
Now, imagine you return to that house the next day. You don't need to redraw the floor plan. You just walk through and update your mental list: "The blue chair moved to the corner," "A new trash can appeared," "The table is gone."

How the robot does it: The robot re-enters the building, uses the old floor plan to know where it is (relocalization), and then uses a "smart eye" (Open-Vocabulary Object Detection) to spot objects. It doesn't just see "a chair"; it understands "a chair" and can even recognize new types of objects it hasn't seen before without needing to be retrained.

The "Object Graph" Notebook

Instead of trying to remember every pixel of every object, the robot keeps a simple notebook (called a Semantic Object Graph).

The Entries: Each object gets a line in the notebook with its name (e.g., "Chair"), its location (3D coordinates), and a confidence score.
The Update Rule: When the robot sees an object, it asks: "Is there already a 'Chair' in my notebook near this spot?"
- Yes: It updates the existing entry (maybe the chair moved slightly).
- No: It adds a new entry to the notebook.
- Missing: If the robot looks at a spot where a "Table" used to be and sees nothing, it crosses that entry out of the notebook.

This system is smart enough to handle changes happening during a single walk (intra-session) and changes that happen between different days (inter-session).

Real-World Testing

The researchers tested this on a robot named "Fetch" in two real places:

A huge office space (8,500 square meters): About the size of a large factory floor.
A smaller lab (117 square meters): About the size of a large living room.

They ran the robot through these spaces multiple times, moving objects around between visits.

The Result: The system successfully tracked the changes. It correctly identified when chairs were removed, when new trash cans were added, and when objects were moved.
The Score: The system achieved high accuracy (F1 scores between 0.75 and 0.87), meaning it was very good at finding objects and not making up fake ones.

Why This Matters (According to the Paper)

The paper argues that this approach is much faster and more efficient than rebuilding the whole map every time.

Speed: In a comparison test, their method could process video data at 30 frames per second and still map objects, while a more complex, state-of-the-art system (Khronos) could only manage 3 frames per second before it got overwhelmed and dropped data.
Flexibility: Because the "geometric map" (walls) and "semantic map" (objects) are separate, you can upgrade the robot's "eyes" (the object detector) without having to rebuild the whole system.

The Limitations (What the Paper Admits)

The system isn't perfect yet.

The "Blind Spot" Problem: If the robot doesn't see a chair, it assumes the chair is gone. It can't always tell the difference between "the chair was moved" and "the robot just missed it because of a shadow or a person blocking the view."
Lighting and Clutter: Shadows, low light, or objects hiding behind others can confuse the system, leading to missed detections or mistakes (like thinking a cardboard box is a trash can).
No Active Planning: The robot follows a pre-planned path. It doesn't currently say, "I think I missed a chair, I should go back and look again." It just keeps moving.

In short, this paper presents a system that builds a house's skeleton once and then efficiently updates a list of its contents every time it visits, making it a practical solution for robots that need to monitor changing environments over long periods.

Technical Summary: Build Once, Monitor Continuously

Problem Statement

Persistent semantic monitoring of indoor environments (e.g., warehouses, hospitals, offices) requires robots to repeatedly traverse the same space and track object changes over time. Existing approaches often run full Simultaneous Localization and Mapping (SLAM) with dense semantic reconstruction from scratch on every visit. This is computationally redundant because the static geometry of the environment remains largely unchanged, while only the objects (their positions, presence, or absence) vary. Furthermore, many current semantic mapping systems build a map once and lack mechanisms to update it efficiently to reflect daily object relocation, removal, or addition.

Methodology

The authors propose a modular, two-stage system that decouples geometric mapping from semantic updating, adhering to a "Build Once, Monitor Continuously" paradigm. The system runs autonomously on a physical robot without human teleoperation.

Stage 1: Autonomous Exploration and Geometric Mapping

Objective: Build a reliable 2D occupancy grid map of the environment geometry.
Algorithm: The system employs a modified Frontier-Based Exploration method called Dynamic Window Frontier Exploration (DWFE).
- Unlike standard frontier methods that may struggle with large spaces due to fixed cost parameters, DWFE utilizes a dynamic search window. It adjusts the search radius between a local and a global region based on the density of available frontiers.
- If the number of frontiers within a local radius drops below a threshold, the search expands to a global radius. This ensures efficient coverage and reduces redundant traversal in large environments.
Output: A saved 2D occupancy grid map and a recorded exploration trajectory.

Stage 2: Autonomous Semantic Mapping and Updating

Objective: Relocalize within the pre-built geometric map and update a semantic object graph to reflect object changes.
Trajectory Planning: The robot replans a traversal path by sampling waypoints from the initial exploration trajectory. These waypoints are ordered using a greedy nearest-neighbor heuristic (solving a Traveling Salesman Problem approximation) to create an efficient traversal path for revisiting the environment.
Perception Stack:
- Detection: Uses an open-vocabulary object detector (GroundingDINO) to identify object categories and bounding boxes without retraining.
- Segmentation: Uses a promptable segmentation model (MobileSAM) to generate object masks within the bounding boxes.
- 3D Localization: Depth data is back-projected and transformed into the map frame to compute the 3D centroid of each object.
Semantic Representation: The semantic map is a hybrid representation:
- Lower Level: The 2D occupancy grid for geometry.
- Higher Level: A Semantic Object Graph where nodes represent unique objects with attributes: unique ID, category, 3D position, and confidence score.
Update Mechanism (Object Association):
- The system associates new detections with existing graph nodes based on category and centroid distance (threshold $\delta$ ).
- Node Removal: If an expected node (based on the graph) is not detected within the robot's field of view (FoV) and distance threshold, it is removed (indicating the object was moved or removed). Nodes are only evaluated for removal when in FoV to handle occlusions robustly.
- Node Addition: If a detected object has no matching existing node within the distance threshold, a new node is added.
- Change Types: This mechanism captures intra-session changes (objects moving during a single traversal) and inter-session changes (changes across multiple revisits).

Key Contributions

Integrated Autonomous System: A fully autonomous pipeline combining exploration, relocalization, open-vocabulary semantic mapping, and map updating on a physical robot (Fetch). It utilizes a lightweight object graph and category/distance-based association rules to manage object states across visits.
Decoupled Two-Stage Design: The system explicitly trades dense, repeated geometric reconstruction for a fast, repeatable semantic update process. By building the geometric map once and only re-running the lightweight semantic stage, the system scales efficiently with the number of revisits.
Real-World Validation: The system was deployed and evaluated in two distinct real-world indoor environments (approx. 8,500 m² and 117 m²). The authors report precision, recall, and F1 scores across multiple update iterations, demonstrating the system's ability to handle dynamic object changes.

Experimental Results

Exploration Efficiency: In a simulated large-scale environment (1,440 m²), the DWFE method covered approximately 7.8% more area while traveling 12% less distance compared to a conventional frontier baseline.
Semantic Mapping Performance:
- Environment A (8,500 m²): Mapped tables, chairs, and doors. F1 scores ranged from 0.75 to 0.87. Performance dips in chair detection were attributed to human occlusion and path deviations rather than detection failures.
- Environment B (117 m²): Mapped diverse objects (trash bins, umbrellas, cabinets, etc.). The system successfully tracked removals and additions across iterations.
Throughput Comparison: Compared against Khronos (a state-of-the-art spatio-temporal semantic SLAM system) on a mezzanine dataset:
- At 21 FPS, both methods mapped a comparable number of objects (~86-87).
- At 30 FPS, the proposed method mapped 79 objects, whereas Khronos mapped only 3. The authors attribute this to Khronos's dense reconstruction and loop-closure optimization dropping frames to maintain consistency, whereas the proposed lightweight pipeline maintains throughput.

Significance and Claims

The paper positions this work as a systems contribution rather than a novel single algorithm. Its primary significance lies in addressing the persistent monitoring setting, where the efficiency and reliability of deployment over many revisits are prioritized over dense, single-pass reconstruction.

Scalability: By separating geometry from semantics, the system avoids the computational cost of re-mapping static environments, making it suitable for frequent revisits in dynamic indoor spaces.
Robustness: The use of centroid-based association (rather than IoU or appearance features) provides stability across revisits where object poses or viewpoints may change.
Limitations Acknowledged: The authors note that the distance-threshold association cannot distinguish between a missed detection and a true removal (absence of evidence vs. evidence of absence). They also acknowledge that the two-stage design prevents the robot from using semantic goals to guide the initial exploration or actively re-plan toward changing objects during a pass. Future work is suggested to include interactive perception and adaptive traversal planning.