GSVisLoc: Generalizable Visual Localization for Gaussian Splatting Scene Representations

Technical Summary: GSVisLoc

Problem Statement
Visual localization involves estimating the camera's position and orientation (pose) within a 3D environment given a single query image. While classical structure-based methods achieve high accuracy, they often require substantial storage for 3D descriptors and an initial image retrieval step. Conversely, end-to-end learned approaches (e.g., Absolute Pose Regression) lack the generalization and accuracy of structure-based methods. Recently, 3D Gaussian Splatting (3DGS) has emerged as a high-quality, fast-training scene representation for novel view synthesis. However, existing methods for visual localization using 3DGS are limited: some only refine an initial pose, others exhibit lower accuracy compared to other representations, and some require optimizing a specific 3DGS model with additional features, preventing the use of "vanilla" 3DGS models.

Methodology
The authors propose GSVisLoc, a deep learning-based method designed to perform visual localization directly on standard 3DGS scene representations without requiring retraining, modifications to the 3DGS model, or additional reference images. The approach establishes 3D-2D correspondences between the 3D Gaussian points and 2D image patches through a three-stage pipeline:

1. Feature Encoding:

   • 3D Encoding: The 3D Gaussians are first filtered (removing low-opacity Gaussians) and downsampled. A KPConv-based encoder (KPFCNN) processes the remaining Gaussians, incorporating information from neighboring points to create robust 3D region features.
   • 2D Encoding: A 2D image encoder extracts features from the query image at two scales: coarse features (downsampled by 8x) and fine features (downsampled by 2x).
   • Alignment: The network employs interleaved self-attention and cross-attention layers to refine representations within each domain and align the 3D and 2D feature spaces.

2. Matching Strategy:

   • Coarse Matching: The system computes pairwise cosine similarities between coarse 3D and 2D features. Matches are selected based on a score threshold and a mutual nearest-neighbor (MNN) criterion to filter outliers.
   • Fine Matching: For each coarse match, the system refines the correspondence to the pixel level. It generates a heatmap representing the probability distribution of alignment for a local window around the coarse match location. The final fine-grained match is determined by taking the expectation over this distribution.

3. Pose Estimation and Refinement:

   • The established 3D-2D correspondences are fed into a Perspective-n-Point (PnP) solver with RANSAC to estimate an initial camera pose.
   • A Pose Refinement module (adapted from GS-CPR) further improves accuracy. It renders an image and depth map from the estimated pose using the 3DGS model, establishes dense 2D-2D correspondences between the query and rendered images using MASt3R, lifts these to 3D, and solves for a refined pose using PnP+RANSAC.

Training Regimes
The model is trained using ground-truth matches derived from projecting 3DGS points onto query images using known poses. The training is evaluated under three scenarios:

• Single-scene: Training and testing on the same scene.
• Multi-scene: Training on a collection of scenes and testing on the same set.
• Cross-scene Generalization: Training on a collection of scenes and testing on novel, unseen scenes.

Key Contributions

1. GSVisLoc Network: A deep neural network specifically designed for visual localization on 3D Gaussian Splatting representations.
2. Zero-Modification Deployment: The method operates on vanilla 3DGS models without requiring retraining of the scene representation or the addition of learned features to the Gaussians.
3. Retrieval-Free Inference: Unlike many structure-based or NeRF-based methods, GSVisLoc does not require an image retrieval step during inference, allowing reference images to be discarded after training.
4. Generalization: The model demonstrates the ability to generalize to novel, unseen scenes through learned 3D-2D matching, maintaining accuracy comparable to single-scene trained models.
5. Performance: It achieves state-of-the-art performance among 3DGS-based methods and competitive results against state-of-the-art baselines on standard benchmarks.

Experimental Results
The authors evaluated GSVisLoc on the 7-Scenes (indoor) and Cambridge Landmarks (outdoor) datasets, as well as the ScanNet++ V2 dataset for cross-scene generalization.

• 7-Scenes: GSVisLoc outperformed other 3DGS-based methods (6DGS, GSplatLoc) and surpassed APR and NeRF-based methods in both translation and rotation accuracy. It achieved performance comparable to Scene Coordinate Regression (SCR) methods, though it remained slightly behind the top structure-based method (DVLAD+R2D2).
• Cambridge Landmarks: The method surpassed GSplatLoc in rotation accuracy and achieved comparable translation accuracy. It significantly outperformed APR methods but trailed the state-of-the-art hierarchical method HLoc, particularly in translation accuracy. The authors attribute this gap partly to the lower rendering quality of 3DGS on this specific outdoor dataset.
• Generalization: In cross-scene experiments on ScanNet++ V2, the model generalized effectively to unseen scenes without retraining, retaining high accuracy.

Significance and Claims
The paper claims that GSVisLoc successfully leverages the explicit 3DGS scene representation for visual localization, bridging the gap between the high-quality rendering of 3DGS and the need for accurate pose estimation. By eliminating the need for image retrieval and avoiding the optimization of specific 3DGS models for localization, the method simplifies deployment. The authors emphasize that their approach generalizes to novel scenes, making it a practical solution for applications requiring robust localization in dynamic or previously unseen environments using standard 3DGS reconstructions.