Technical Summary: Disambiguating 2D-3D Correspondences in Gaussian Splatting-based Feature Fields for Visual Localization

1. Problem Statement

While Gaussian Splatting-based Feature Fields (GSFFs) have demonstrated potential for visual localization, the authors identify a fundamental structural mismatch between photometrically optimized GSFFs and the requirements of Perspective-n-Points (PnP) algorithms used for pose estimation.

Two primary issues hinder the direct application of standard GSFFs to 2D-3D matching:

1. Volumetric Ambiguity (Many-to-One Mappings): Each 3D Gaussian in a standard GSFF covers a volumetric region. When approximating a Gaussian as a single 3D point (e.g., its center) for PnP input, multiple 2D pixels from a query image may map to this single point. This "many-to-one" correspondence introduces geometric inconsistency and accumulates reprojection errors, destabilizing PnP convergence and leading to suboptimal pose estimates.
2. Redundancy and Lack of Multi-View Consistency: Photometric optimization often spawns numerous redundant Gaussians to capture view-dependent effects. These Gaussians lack multi-view consistency, rendering them ineffective for 2D-3D matching under novel viewpoints. Furthermore, existing methods like STDLoc require hours of per-scene optimization and additional query-side detection or dense 2D-2D refinement to achieve high accuracy, limiting their practicality.

2. Methodology

The paper proposes SplitGS-Loc, a framework designed to construct localization-specialized GSFFs without per-scene training or iterative pose refinement. The methodology consists of two core components: PlugGS-Loc (for feature lifting and sampling) and SplitGS-Loc (for correspondence disambiguation).

2.1 PlugGS-Loc: Feature Lifting and Sampling

This module addresses the issue of redundant Gaussians and feature blurring by leveraging composition weights derived from the rasterization process.

• Weight Aggregation: Instead of storing all Gaussians contributing to a pixel, the method identifies Gaussians primarily rasterized to a pixel based on a composition weight threshold ($\tau$). It aggregates the maximum contribution weight ($\tilde{w}$) for each Gaussian across all training views to form an informative weight set ($W_n$).
• Gaussian Sampling: To prune superfluous Gaussians, the scene is divided into local regions. Within each region, Gaussians are scored based on their aggregated weights ($\psi(g_n)$), which reflect their significance across multiple views. Only the highest-scoring Gaussian per region is retained, ensuring a compact map of multi-view consistent Gaussians.
• Feature Registration: Instead of optimizing features via rendering loss (which causes blurring), features are directly lifted. For each selected Gaussian, a distinctive feature vector is computed as a weighted sum of pixel-level features from the training views, where weights are derived from the strong pixel-Gaussian associations identified in the previous step. This enforces multi-view consistency and prevents feature blurring.

2.2 SplitGS-Loc: Disambiguating Correspondences

This module addresses the structural mismatch between volumetric Gaussians and point-based PnP.

• Mixture-of-Gaussians Splitting: The core innovation is decomposing each 3D Gaussian along its major axis (the direction of highest ambiguity). A single Gaussian is split into three components: a central component and two side components positioned symmetrically along the major axis.
• Moment Matching: The splitting parameters are derived by matching the first four moments of the original distribution to the mixture distribution. This ensures the split components collectively represent the original volume while extending the spatial extent along the major axis.
• Integration: The splitting strategy is integrated with PlugGS-Loc. During the sampling phase, child Gaussians are temporarily aggregated to score their parent, ensuring spatial coverage is maintained. However, during feature registration, each child Gaussian is paired with a distinct pixel feature. This transforms the "many-to-one" mapping into precise "one-to-one" correspondences, as each split Gaussian now represents a finer-grained region with a unique feature.

3. Key Contributions

• Identification of Structural Mismatch: The paper highlights that photometrically optimized GSFFs are inherently ill-suited for direct 2D-3D matching due to volumetric ambiguity and redundant Gaussians.
• SplitGS-Loc Framework: A novel construction framework that disambiguates 2D-3D correspondences without per-scene training. It combines:
  • Mixture-of-Gaussians-based Splitting: A deterministic, feature-independent decomposition that replaces ambiguous many-to-one mappings with precise one-to-one correspondences.
  • Composition Weight-based Feature Lifting: A pipeline that selects salient, multi-view consistent Gaussians and assigns them discriminative features via strong pixel-Gaussian associations.
• Efficiency and Performance: The method achieves state-of-the-art performance on localization benchmarks (Cambridge Landmarks and 7Scenes) with a mapping time of approximately 7 minutes (including the initial 3DGS training) and without requiring iterative pose refinement.

4. Experimental Results

The authors evaluate their method on the Cambridge Landmarks (outdoor) and 7Scenes (indoor) datasets, comparing against classical methods, deep learning-based regression (SCR, APR), NeRF-based methods, and other GS-based approaches.

• Accuracy: SplitGS-Loc achieves the best median translation and rotation errors among training-free methods. For instance, on the Cambridge Landmarks dataset, it achieves an average error of 8.7 cm / 0.15°, outperforming STDLoc (11.1 cm / 0.19°) and other GSFF-based methods.
• Stability: Analysis of PnP convergence shows that SplitGS-Loc significantly reduces the number of "many-to-one" inputs (from 313 to 199 in one test case) and increases the number of valid inliers. This leads to stable convergence and robustness even when using efficient RANSAC settings (fewer iterations), where other methods suffer accuracy drops.
• Scalability: Unlike methods where performance saturates with increased map size, SplitGS-Loc shows sustained accuracy improvements as the map size increases, validating the efficacy of the splitting strategy in providing finer detail.
• Ablation Studies: Experiments confirm that:
  • The weight threshold ($\tau$) balances feature blurring and spatial coverage.
  • The splitting factor ($\beta$) monotonically improves accuracy within feasible bounds by increasing the separation of child Gaussians.
  • Replacing Gaussian attributes (composition weights, volumetric splitting) with geometry-only alternatives significantly degrades performance, highlighting the importance of exploiting GS attributes.

5. Significance and Claims

The paper claims to reframe GSFFs from exhaustive 3D maps to "deployable" ones, restoring the balance between performance and practicality. The primary significance lies in:

• Eliminating Per-Scene Training: The framework constructs the feature field directly from a pre-trained RGB 3DGS and training views in minutes, removing the need for the hours-long multi-stage optimization required by prior GSFF methods.
• Direct 2D-3D Matching: By resolving the many-to-one ambiguity, the method enables stable PnP-based pose estimation without relying on query-side detection or dense 2D-2D refinement.
• Practicality: The approach offers a streamlined solution for real-world visual localization that is smaller, faster, and more precise than previous pipelines, making GS-based representations viable for applications requiring rapid deployment.

The authors modestly note that the framework relies on the quality of the original photometrically optimized Gaussians; thus, scenes with severe illumination changes or sparse trajectories where GS fails to converge remain challenging. However, they position their work as a significant step toward making neural scene representations practical for efficient and accurate visual localization.