💻 computer science

POMA-3D: The Point Map Way to 3D Scene Understanding

This paper introduces POMA-3D, the first self-supervised 3D representation model that leverages point maps to transfer 2D foundation priors into 3D understanding via a view-to-scene alignment and joint embedding-predictive strategy, demonstrating strong performance across diverse 3D tasks using only geometric inputs.

Original authors: Ye Mao, Weixun Luo, Ranran Huang, Junpeng Jing, Krystian Mikolajczyk

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ye Mao, Weixun Luo, Ranran Huang, Junpeng Jing, Krystian Mikolajczyk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world around it. Usually, we teach robots by showing them 3D point clouds (like a swarm of digital dust) or depth maps (like a topographic map). But the researchers at Imperial College London argue that there's a better way: Point Maps.

Think of a Point Map like a "3D photograph." Instead of just storing colors in a grid (like a normal photo), this grid stores 3D coordinates (x, y, z) for every single pixel. It's like taking a flat photo but labeling every dot with its exact distance and position in the room. This format is special because it looks like a 2D image to a computer, which makes it easy to use the massive knowledge already built into 2D image AI models.

Here is the breakdown of their new system, POMA-3D, using simple analogies:

1. The Problem: The "Language Barrier"

Current 3D AI models are like students who only speak "3D." They struggle to learn from the billions of 2D images and text descriptions available on the internet because the formats don't match. It's like trying to teach a student who only knows French using a textbook written in Chinese.

2. The Solution: The "Universal Translator" (POMA-3D)

The authors created POMA-3D, the first model that learns 3D understanding directly from these "3D photographs" (Point Maps). Because Point Maps look like 2D images, POMA-3D can "speak" the same language as powerful 2D AI models (like CLIP). This allows it to instantly borrow the vast knowledge of the 2D world and apply it to 3D spaces.

3. The Training Data: "The ScenePoint Library"

To teach this new model, they built a massive library called ScenePoint.

The Real Rooms: They took 6,500 real-world room scans (from datasets like ScanNet) and turned them into Point Maps.
The Imaginary Rooms: They took 1 million regular 2D images from the internet and used a clever trick (a model called VGGT) to turn them into fake 3D Point Maps.
The Descriptions: Every scene has a "caption" written by a Large Language Model (LLM), describing what's in the room (e.g., "There are six windows and two tables").

4. How It Learns: Two Special Exercises

The model learns through two main methods, like a student doing homework:

Exercise A: The "View-to-Scene" Match (Alignment)
Imagine showing the robot a picture of a living room and a sentence saying "This is a cozy living room with a red sofa." The robot has to learn to match the 3D Point Map of that room with the text and the 2D photo. It learns that "sofa" in text equals "sofa" in the 3D grid.
Exercise B: The "Puzzle" (POMA-JEPA)
This is a game of "fill in the blanks." The robot is shown a 3D room where some parts are hidden (masked). It has to guess what the hidden parts look like based on the visible parts.
- The Twist: Because 3D space is messy, the hidden pieces might be in a different order than the visible ones. So, instead of forcing a perfect one-to-one match, the model uses a "fuzzy" matching rule (Chamfer Distance) that says, "As long as the hidden pieces are somewhere in the right area, you're doing well." This teaches the robot to understand the overall shape and geometry of the room, not just pixel-perfect details.

5. What Can It Do? (The Results)

After this training, POMA-3D becomes a super-teacher for other robots. It acts as a strong backbone that helps them do four main things, even without knowing the color of objects (using only geometry):

3D Question Answering: If you ask, "How many chairs are around the coffee table?" it can count them correctly.
Embodied Navigation: If a robot says, "I'm sitting on the sofa and want to go to the bed," POMA-3D can tell it, "Move forward."
Scene Retrieval: If you describe a room ("A room with a round table and two bookshelves"), the model can find that specific room in a database of thousands.
Coarse Localization: If a robot says, "I am standing between two blackboards," the model can pinpoint exactly where that robot is in the 3D space.

The Bottom Line

The paper claims that by using Point Maps as a bridge, they can finally transfer the massive intelligence of 2D image AI into the 3D world. Their model, POMA-3D, outperforms previous methods in understanding 3D scenes, proving that you don't need to reinvent the wheel for 3D; you just need to translate the 2D knowledge into a format 3D can understand.

Note: The authors explicitly state that their current model only uses geometric coordinates (shapes and positions), not colors. They suggest that future work could combine this with color to make it even better, but that is not part of the current results.

Technical Summary: POMA-3D

Problem Statement

Understanding 3D scenes is fundamental for embodied agents and AR systems, yet the field faces a significant bottleneck: the scarcity of large-scale, pretrained 3D priors compared to the billion-scale data available for 2D vision-language models. Existing 3D vision-language learning (VLL) methods typically rely on point clouds, depth maps, or 3D Gaussian splats. However, these modalities differ substantially from the input formats of powerful 2D foundation models (e.g., CLIP), hindering effective knowledge transfer. Furthermore, current approaches often struggle to achieve a "CLIP moment" in 3D understanding, where a single model can generalize across diverse tasks like question answering, navigation, and retrieval without task-specific architectural changes.

Methodology

Core Concept: Point Maps

The paper proposes POMA-3D, the first self-supervised 3D representation model learned from point maps. A point map encodes explicit 3D coordinates $(x, y, z)$ on a structured 2D grid, preserving global 3D geometry while maintaining compatibility with the input format of 2D foundation models. This modality bridges the gap between 2D priors and 3D structure.

Dataset: ScenePoint

To facilitate large-scale pretraining, the authors introduce ScenePoint, a dataset comprising:

6,562 Room-Level Scenes: Constructed from RGB-D datasets (ScanNet, 3RScan, ARKitScenes). Each scene includes multi-view point maps paired with LLM-generated view-level and scene-level captions.
1 Million Single-View Scenes: Generated from the ConceptualCaptions dataset. Using the VGGT model, 2D images are converted into single-view point maps with predicted depth and camera parameters.
Annotations: Captions are generated using InternVL3-14B, with a selection process based on cosine similarity to FG-CLIP embeddings to ensure quality.

Model Architecture and Training

POMA-3D employs a two-stage pretraining paradigm using a view-to-scene vision-language alignment strategy and a novel POMA-JEPA objective.

View-to-Scene Vision-Language Alignment:
- The model aligns point map embeddings with frozen image and text embeddings from FG-CLIP.
- View-level alignment: Encourages point map, image, and caption embeddings of the same view to align while contrasting against other views in the batch.
- Scene-level alignment: Aggregates multi-view embeddings into a scene representation and aligns it with the scene-level caption.
- This strategy transfers rich 2D semantic priors into the 3D domain.
POMA-JEPA (Joint Embedding-Predictive Architecture):
- To enforce geometric consistency across multiple views, the authors introduce POMA-JEPA.
- Mechanism: A context encoder processes visible patches of multi-view point maps, while a target encoder (updated via Exponential Moving Average) processes the full point maps. A predictor reconstructs the target embeddings of masked patches.
- Innovation: Unlike standard JEPA which assumes fixed 2D grid ordering, POMA-JEPA accounts for the order-agnostic nature of point clouds in world coordinates. It utilizes Chamfer Distance as the loss function rather than MSE, allowing the model to handle permutation-invariant embedding predictions without mode collapse.
Training Stages:
- Warm-up: Pretraining on single-view point maps derived from 2D images using only the view-level alignment loss ( $L_{view}$ ).
- Main Stage: Joint optimization on multi-view room scenes using alignment losses ( $L_{view} + L_{scene}$ ) and the POMA-JEPA loss ( $L_{pjepa}$ ).

Key Contributions

POMA-3D: The first self-supervised 3D model that learns representations directly from point maps, enabling effective transfer of 2D foundation model priors to 3D.
ScenePoint Dataset: A large-scale dataset combining 6.5K real-world room-level scenes and 1M synthetic single-view scenes, specifically curated for point map pretraining.
Novel Training Objectives: The introduction of view-to-scene alignment and POMA-JEPA, which explicitly handles the geometric consistency and order-agnostic nature of multi-view 3D data.
Modality Efficiency: Demonstration that point maps serve as a superior intermediate modality for 3D VLL, outperforming depth maps and point clouds in transferring 2D priors.

Experimental Results

POMA-3D was evaluated as a backbone for both specialist (BERT-based) and generalist (LLM-based) models across four tasks:

3D Question Answering (ScanQA, SQA3D, Hypo3D):
- The specialist model (POMA-3Dspec) outperformed state-of-the-art point cloud-based models (e.g., SceneVerse) on SQA3D (+6.2% EM@1) and Hypo3D (+4.5% EM@1), despite using only uncolored geometric inputs.
- The generalist model (POMA-3Dllm) surpassed existing 2D LLMs and 3D LLMs on situational reasoning tasks.
Embodied Navigation (MSNN):
- POMA-3Dspec achieved the highest performance in 4-directional navigation, while POMA-3Dllm excelled in 8-directional navigation, indicating strong capabilities in complex spatial reasoning.
Scene Retrieval:
- POMA-3D significantly outperformed 3D VLL methods and even the RGB-based FG-CLIP baseline in zero-shot retrieval, successfully retrieving scenes based on complex textual descriptions (e.g., "round footstool next to two bookshelves").
Coarse Localization:
- The model demonstrated the ability to retrieve specific point map views matching situational text queries (e.g., "I am standing between two blackboards"), effectively localizing an agent's active region in a zero-shot setting.

Ablation Studies confirmed that:

The 2D warm-up stage is crucial for learning robust features.
Room-level scene alignment improves performance across all benchmarks.
POMA-JEPA provides consistent gains, particularly in reasoning-heavy tasks.
Point maps outperform depth maps as an input modality for 3D VLL.

Significance and Claims

The paper claims that POMA-3D addresses the long-standing data scarcity challenge in 3D representation learning by leveraging the vast availability of 2D vision-language data. By introducing the point map as a superior intermediate modality, the work demonstrates that 3D understanding can be scaled without relying solely on expensive 3D scanning data.

The authors assert that POMA-3D provides a strong, generalizable backbone that benefits both lightweight specialist models and large generalist LLMs. The results suggest that aligning 3D geometry with 2D priors via point maps is a promising direction for building scalable foundation models for 3D understanding. The paper concludes that while current results are strong using only geometric inputs, future work should explore fusing point maps with color features to create unified scene representations.