🤖 AI

Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Models

This paper introduces UPPM, a training-free framework that leverages foundation models to generate dynamic descriptors and fuse them within a multi-resolution TSDF map, thereby resolving label fragmentation in open-vocabulary panoptic mapping to achieve high-quality, persistent, and promptable scene understanding.

Original authors: Mohamad Al Mdfaa, Raghad Salameh, Geesara Kulathunga, Sergey Zagoruyko, Gonzalo Ferrer

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohamad Al Mdfaa, Raghad Salameh, Geesara Kulathunga, Sergey Zagoruyko, Gonzalo Ferrer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to build a 3D map of a room while walking through it. To do this well, the robot needs two things: a precise understanding of where things are (geometry) and what things are (semantics).

For a long time, robots were like students who only knew a fixed list of vocabulary words. If they saw a "sofa," they knew it. But if they saw a "couch," a "loveseat," or a "big comfy seat," they might get confused, treat them as three different objects, or just call them "furniture." This made their mental maps messy and inconsistent.

This paper introduces a new system called UPPM (Unified Promptable Panoptic Mapping) that solves this problem using "foundation models" (super-smart AI models trained on the entire internet). Here is how it works, using simple analogies:

1. The Problem: The "Confused Translator"

Imagine a robot taking photos of a room. Every time it sees a table, a fancy AI model (the "translator") might describe it differently based on the angle or lighting:

Frame 1: "A round wooden table."
Frame 2: "A dining table."
Frame 3: "A brown table."

In older systems, the robot would treat these as three separate objects. It would build three different 3D shapes for the same table, making the map look glitchy and fragmented.

2. The Solution: The "Dynamic ID Card"

UPPM introduces a clever trick called a Dynamic Descriptor. Think of this as a special ID card that every object in the room gets.

Instead of forcing the robot to pick just one name immediately, UPPM does three things:

Gathers the Clues: It collects all the different descriptions the AI gave the object over time ("round," "wooden," "dining," "brown").
Finds the "Real" Name: It uses a smart search to find the one standard category that fits best (e.g., "Table"). It also assigns a standard size to it (e.g., "Medium").
Keeps the Details: Even though it knows the object is a "Table," it keeps a note of all the fancy descriptions it heard ("round," "wooden," etc.) on the ID card.

3. How It Builds the Map

The robot builds a 3D map made of tiny blocks (like a giant 3D Lego structure).

The "Unified" Part: When the robot sees the "round wooden table" and later the "dining table," it realizes they are the same block in the 3D Lego structure. It merges them into one solid object.
The "Promptable" Part: Because the ID card holds all those extra descriptions, you can ask the robot, "Show me the round wooden table," or "Show me the dining table," and it will point to the exact same object. It understands that these different words refer to the same thing.

4. Cleaning Up the Mess

The paper also mentions a few "housekeeping" tricks to make the map better:

The "Blurry Photo" Filter: If the robot's camera is shaking or the image is blurry, the system skips that frame. It's like a photographer ignoring a blurry photo so they don't ruin the final album.
The "Duplicate Detector": Sometimes the AI gets excited and draws two boxes around the same chair. The system has a special rule (Custom NMS) to spot these near-identical duplicates and delete the extra one, ensuring the robot doesn't think there are two chairs where there is only one.

The Results

The authors tested this system on three different datasets (simulated and real-world rooms). They found that:

Better Maps: The 3D maps were more accurate and complete than previous methods.
Better Understanding: The robot could correctly identify objects even when the AI gave it weird or varied names.
No Extra Training: The system works "out of the box" using existing AI models; it doesn't need to be retrained for every new room.

In short: UPPM is like giving a robot a smart assistant that can listen to many different ways of describing an object, figure out what the object actually is, and keep a record of all the interesting details, all while building a perfect 3D map of the world.

Technical Summary: Unified Promptable Panoptic Mapping with Dynamic Labeling Using Foundation Models

Problem Statement
Panoptic mapping is essential for enabling robots to reason about both geometry and semantics in structured environments. However, existing systems face a critical trade-off: traditional methods relying on predefined category sets (closed-vocabulary) lack semantic expressiveness, while recent open-vocabulary approaches often produce inconsistent labels. Open-vocabulary models frequently generate closely related or synonymous labels (e.g., "dining table," "wooden table," "kitchen desk") for the same object. When these inconsistent labels are fused into a 3D map, they fragment panoptic entities, degrade volumetric consistency, and fail to resolve how to reconcile open-vocabulary predictions with volumetric priors or object scale. Current solutions either require per-scene offline training, rely on graph-based reasoning, or fail to unify synonyms into a consistent categorical structure while maintaining geometric accuracy.

Methodology
The proposed Unified Promptable Panoptic Mapping (UPPM) framework addresses these challenges by introducing a Dynamic Descriptor mechanism that bridges open-vocabulary perception with consistent 3D panoptic mapping. The pipeline operates without per-scene training or neural radiance fields, utilizing off-the-shelf foundation models in a multi-stage process:

Visual-Linguistic Feature Extraction++ (VLFE++): The system processes RGB inputs using Tag2Text to generate image captions and object-level tags. These are parsed using Part-of-Speech (POS) tagging and lemmatization to normalize morphological variants (e.g., "chairs" to "chair"), creating a set of open-vocabulary candidate labels.
Semantic Retrieval (SR): To resolve label fragmentation, SR maps the open-vocabulary labels to a fixed set of unified semantic categories (based on COCO-Stuff classes) using cosine similarity with MPNet embeddings. Crucially, each unified category carries an inherited volumetric size prior (Small, Medium, or Large). This step produces Elementary Descriptors ( $e_i = \langle o_i, \hat{c}_i, \hat{\nu}_i \rangle$ ), which contain the original open-vocabulary label, the unified category, and the size prior.
Open-Vocabulary Panoptic Segmentation: Grounding-DINO detects objects using the open-vocabulary labels, and the Segment Anything Model (SAM) generates segmentation masks. A Custom Non-Maximum Suppression (NMS) strategy is employed to handle duplicates arising from semantically similar prompts. Unlike traditional IoU-based NMS, this method uses a coordinate-based proximity criterion to suppress near-identical bounding boxes and containment-based suppression for nested boxes, significantly reducing redundant associations.
Unified Panoptic Fusion (UPF): The system fuses geometry and semantics into a multi-resolution multi-TSDF map. During fusion, elementary descriptors associated with the same 3D submap (object) are aggregated into a single Dynamic Descriptor. This descriptor accumulates all observed open-vocabulary labels over time while maintaining a single, stable unified category and size prior. Association is determined by Intersection-over-Union (IoU) between predicted and rendered masks, with a minimum threshold to avoid spurious links.

Key Contributions

Panoptic Dynamic Descriptor Mechanism: UPPM introduces a novel data structure that aggregates open-vocabulary labels at the object level during fusion rather than at detection time. This allows the system to preserve rich, open-vocabulary expressiveness (enabling queries like "light-brown table" or "eating table") while enforcing categorical consistency (unifying them under "table") and geometric size priors.
Training-Free Pipeline: The system constructs 3D panoptic maps that are geometrically accurate and semantically consistent without requiring per-scene training, graph-based reasoning, or neural radiance fields, making it suitable for real-time robotic applications.
Comprehensive Evaluation and Ablation: The authors evaluate UPPM across three regimes (Segmentation-to-Map, Map-to-Map, and Segmentation-to-Segmentation) on ScanNet v2, RIO, and Flat datasets. Ablation studies confirm that the unified semantics, custom NMS, and blurry-frame filtering components are critical to the system's performance.

Results
Experimental results demonstrate that UPPM achieves superior overall performance compared to state-of-the-art closed-set and open-vocabulary baselines:

Geometric Reconstruction: On the Flat dataset, UPPM achieved an accuracy of 0.61 cm and completeness of 1.10 cm, outperforming competitors like Kimera and PanMap. On the noisy, real-world RIO dataset, UPPM showed robustness to motion blur, achieving a Chamfer distance of 5.65 cm and a Hausdorff distance of 10.05 cm, significantly better than other open-vocabulary methods.
Panoptic Segmentation Quality: UPPM attained the highest Panoptic Quality (PQ: 0.414) and Recognition Quality (RQ: 0.475) on the Flat dataset.
Ablation Insights: The removal of unified semantics (PPM) led to ambiguous labeling and lower reconstruction accuracy. The custom NMS improved the completion ratio by 8.27% and reduced false negatives by 16.9% in open-vocabulary settings. Blurry frame filtering notably improved performance on the noisy RIO dataset.
Efficiency: While UPPM operates at 2.28 FPS (slower than closed-set baselines due to vision-language processing), it trades frame rate for open-vocabulary capability and improved panoptic quality without retraining.

Significance
The paper claims that UPPM represents a paradigm shift from detection-time labeling to fusion-time semantic aggregation. By reconciling the long-tailed semantics of foundation models with the consistency requirements of panoptic mapping, UPPM enables robots to maintain persistent, queryable 3D maps. The system preserves the interpretability of open-vocabulary prompts—allowing language-conditioned object retrieval using synonyms and paraphrases—while delivering strong geometric accuracy and preventing the fragmentation of instances that plagues existing open-vocabulary mapping approaches. The authors position this as a step toward robust, real-time scene understanding that does not rely on heavy per-scene optimization.