Uni3R: Unified 3D Reconstruction and Semantic Understanding via Generalizable Gaussian Splatting from Unposed Multi-View Images

1. Problem Statement

Reconstructing and semantically interpreting 3D scenes from sparse, unposed 2D images is a fundamental challenge in computer vision. Existing methods face three primary limitations:

• Decoupling: Most approaches separate geometric reconstruction from semantic understanding, requiring separate pipelines.
• Scalability: Traditional methods (e.g., NeRF, standard 3DGS) rely on time-consuming, per-scene optimization, making them unsuitable for real-time or zero-shot applications.
• Input Constraints: Recent generalizable methods often require known camera poses or are limited to two-view inputs (e.g., based on DUSt3R), leading to fragmented reconstructions when handling arbitrary multi-view sequences or videos.

2. Methodology: Uni3R

Uni3R is a novel feed-forward framework that jointly reconstructs a unified 3D scene representation enriched with open-vocabulary semantics directly from unposed multi-view images. It operates in a single forward pass without per-scene optimization.

Core Architecture

• Intrinsic Embedding: To resolve scale ambiguity in monocular reconstruction, the model encodes camera focal lengths and principal points, concatenating them with image features before tokenization.
• Cross-View Transformer Encoder:
  • Inspired by VGGT, this module processes input views augmented with intrinsic embeddings.
  • It utilizes a Cross-View Transformer that alternates between intra-frame (local context) and cross-frame (global correspondence) attention mechanisms.
  • This allows the model to fuse information from an arbitrary number of views into a globally consistent, view-agnostic latent representation.
• Gaussian Decoding:
  • A Dense Prediction Transformer (DPT) refines the latent features into dense per-pixel maps.
  • Dedicated MLP heads predict the parameters for a set of 3D Gaussian primitives. Each primitive $G_j$ includes:
    • Geometry: Center ($\mu$), Scale ($s$), Rotation ($r$).
    • Appearance: Opacity ($\alpha$), Color ($c$).
    • Semantics: High-dimensional semantic feature vector ($f^{sem}$).
• Rendering with Open-Vocabulary Semantics:
  • The Gaussians are rendered using a differentiable rasterizer.
  • Semantic features are compressed via an autoencoder to reduce memory costs during rendering.
  • Inference: Semantic segmentation is performed by computing cosine similarity between rendered pixel features and text-derived prototypes (generated by a CLIP text encoder), enabling zero-shot open-vocabulary segmentation.

Training Objectives

The model is trained with a weighted sum of three losses:

1. Photometric Loss ($\mathcal{L}_{rgb}$): Combines pixel-wise L1 loss and LPIPS to ensure rendered images match input views.
2. Semantic Loss ($\mathcal{L}_{sem}$): Distills knowledge from a frozen 2D vision-language model (LSeg). It aligns rendered semantic features with 2D CLIP-based features using cosine similarity, lifting 2D semantics into 3D without 3D annotations.
3. Geometry Loss ($\mathcal{L}_{geo}$): A novel point-map-guided regularization.
   • Uses a frozen VGGT to generate dense point maps with confidence scores.
   • Applies a confidence-based mask (selecting top-k confident pixels) to create a soft geometric prior.
   • Computes a single-directional Chamfer distance between the predicted point cloud and the VGGT-guided map. This stabilizes training, prevents local minima, and improves structural consistency (especially at object boundaries).

3. Key Contributions

1. Unified Feed-Forward Framework: Uni3R is the first architecture to unify 3D reconstruction (geometry/appearance) and open-vocabulary semantic understanding in a single pass from unposed multi-view inputs, eliminating the need for per-scene optimization.
2. Generalizable Multi-View Fusion: By leveraging a Cross-View Transformer, the model handles arbitrary numbers of input views (from pairs to video sequences) and produces globally consistent 3D representations, overcoming the pairwise limitations of previous DUSt3R-based methods.
3. Geometry-Guided Training: The introduction of a point-map-guided geometric loss significantly enhances depth accuracy and training stability, allowing the model to learn robust 3D structures using only RGB supervision.
4. Zero-Shot Semantic Capability: The embedded semantic features enable zero-shot 3D segmentation via text prompts, removing the dependency on expensive 3D semantic ground truth.

4. Experimental Results

Uni3R was evaluated on RE10K, ScanNet, ACID, and the out-of-domain Mip-NeRF360 datasets.

• Novel View Synthesis:
  • Outperforms pose-dependent methods (PixelSplat, MVSplat) by a clear margin (~1.7 dB on RE10K).
  • Surpasses pose-free baselines (NoPoSplat, VicaSplat) in PSNR, SSIM, and LPIPS, demonstrating superior detail preservation and structural coherence.
  • Achieves state-of-the-art (SOTA) performance on 4-view and 8-view settings.
• Semantic Segmentation:
  • Sets new SOTA on ScanNet for open-vocabulary 3D semantic segmentation (mIoU: 55.84% vs. 50.78% for LSM).
  • Effectively resolves 2D view-dependent ambiguities by aggregating features across multiple views into a unified 3D space.
• Depth Estimation:
  • Achieves lower depth errors (AbsRel) compared to baselines, validating the effectiveness of the geometry-guided loss.
• Efficiency:
  • Speed: Reconstruction takes approximately 0.15–0.36 seconds per scene.
  • Comparison: Significantly faster than per-scene optimized methods (e.g., Feature-3DGS takes ~18–60 minutes) while offering better generalization.

5. Significance

Uni3R represents a paradigm shift in 3D scene understanding by moving away from per-scene optimization toward generalizable, feed-forward learning. Its ability to simultaneously reconstruct high-fidelity geometry, render novel views, and perform zero-shot semantic segmentation from unposed images makes it highly applicable to real-world scenarios such as:

• Autonomous Driving & Robotics: Real-time 3D perception and navigation in dynamic environments.
• Augmented Reality: Instant scene understanding and interaction without prior calibration.
• Digital Twins: Scalable reconstruction of large-scale environments from sparse video data.

By unifying geometry, appearance, and semantics, Uni3R provides a robust, scalable foundation for next-generation 3D AI systems.