Technical Summary: LocusGS

Problem Statement

Recent feed-forward 3D Gaussian Splatting (3DGS) methods have shifted from costly per-scene optimization to direct prediction from input views. Among these, query-based approaches (e.g., TokenGS) represent a scene using a fixed set of learnable queries (tokens). Each token aggregates multi-view evidence via a Transformer decoder and is decoded into a group of 3D Gaussians.

However, the authors observe a critical limitation in existing query-based methods: weak spatial coherence. While ideally, different queries should specialize in distinct, coherent local regions of the scene, the Gaussians decoded from a single latent query often scatter across distant, unrelated scene regions. This results in a spatially diffuse distribution that poorly aligns with the underlying scene structure. The authors attribute this failure to the purely latent representation of existing Gaussian queries, which lack an explicit notion of where in 3D space a query operates or how large a region it covers. Consequently, the multi-view evidence aggregated by a query and the resulting Gaussians are not constrained to a coherent local region.

Methodology: LocusGS

To address this, the authors propose LocusGS (Spatially Grounded Tokens), which augments each Gaussian query with an explicit 3D anchor state. This state consists of a 3D center ($\mu$) and a support radius ($r$).

Key Components

1. Explicit 3D Anchor States:
   Instead of representing a query solely as a latent feature vector, LocusGS associates each query $i$ at decoder layer $l$ with an anchor state $a^l_i = (\mu^l_i, r^l_i)$.

   • Center ($\mu$): Specifies the current 3D reference location of the query.
   • Radius ($r$): Defines the extent of the query's local support region. The radius is parameterized via an unconstrained variable $\rho$ passed through a softplus function to ensure positivity.
   • These states are learnable, shared across scenes, and progressively refined throughout the decoder layers.

2. Anchor-Guided Decoder:
   The decoder utilizes the anchor states to guide both self-attention and cross-attention mechanisms:

   • Anchor-Aware Self-Attention: A positional embedding derived from the anchor center $\mu$ is injected into the token features. This allows interactions between Gaussian tokens to depend on their relative 3D locations, not just latent feature similarity.
   • Anchor-to-Ray Geometric Bias (Cross-Attention): To aggregate multi-view evidence, the method introduces a geometric bias based on the distance between the query's anchor center $\mu^l_i$ and the camera rays $\ell_j$ associated with image tokens.
     • The bias is defined as $b^l_{ij} = -\frac{1}{2} \left( \frac{D(\mu^l_i, \ell_j)}{\sigma_0 r^l_i} \right)^2$, where $D$ is the Euclidean distance from the point to the ray.
     • This bias penalizes image tokens whose rays are far from the anchor, effectively guiding the query to focus on geometrically relevant observations. The radius $r^l_i$ controls the bandwidth of this spatial preference.

3. Dynamic Anchor Refinement:
   After each decoder layer, the token features predict residual updates ($\Delta \mu, \Delta \rho$) for the anchor center and radius. This allows the anchor state to adapt its 3D location and support scale to the specific input scene as the decoder progresses.

4. Anchor-Centered Gaussian Decoding:
   In the final decoding stage, Gaussians are not predicted in a fully unconstrained manner. Instead, the model predicts local offsets ($\delta$) relative to the final refined anchor center. The final Gaussian center is computed as:
   $$\mu^G_{i,k} = \mu^L_i + r^L_i \delta_{i,k}$$
   Here, the anchor center provides the global reference, and the radius scales the local offsets. This formulation forces the Gaussians decoded from a single token to form a compact local group around the anchor, rather than scattering freely.

5. Multi-layer Rendering Supervision:
   The training objective includes rendering supervision at intermediate decoder layers (in addition to the final layer). The loss combines image reconstruction (MSE + SSIM) with visibility regularization applied to both the decoded Gaussian centers and the anchor centers, encouraging coherent spatial organization throughout the decoding process.

Key Contributions

• Spatial Grounding of Queries: The introduction of explicit 3D anchor states (center and radius) to query-based 3DGS, moving beyond purely latent representations.
• Anchor-Aware Mechanisms: The design of geometric biases for cross-attention (anchor-to-ray) and positional embeddings for self-attention that utilize the anchor state to enforce spatial coherence.
• Structured Gaussian Generation: A decoding strategy that predicts local offsets scaled by the anchor radius, ensuring that tokens generate spatially compact Gaussian groups.
• Progressive Refinement: A mechanism to iteratively refine anchor locations and support radii across decoder layers, allowing the model to adapt to scene geometry.

Experimental Results

The authors evaluate LocusGS on novel view synthesis benchmarks, specifically RealEstate10K (RE10K) and DL3DV, comparing against baselines like TokenGS, MVSplat, and DepthSplat.

• Rendering Quality: Under identical token and Gaussian budgets, LocusGS consistently outperforms query-based baselines (TokenGS) in PSNR, SSIM, and LPIPS. For instance, on DL3DV with 4 input views, LocusGS (4096 tokens) achieves a PSNR of 24.80 compared to TokenGS's 23.44.
• Spatial Organization: Qualitative and quantitative analyses show that LocusGS produces significantly more structured Gaussian distributions.
  • Token-Level Compactness: The paper introduces a metric $C_{centroid}$ to measure the dispersion of Gaussians within a token. LocusGS drastically reduces this dispersion (e.g., from 5.11 to 0.19 on RE10K), indicating that Gaussians from a single token are tightly clustered.
  • Anchor Layouts: The learned anchors form a coarse, coherent spatial scaffold over the scene, covering main structures and adapting their radii to local complexity (larger radii in sparse regions, smaller in detailed areas).
• Generalization: The model maintains a fixed Gaussian budget independent of the number of input views, unlike dense pixel-aligned methods. It demonstrates robust performance when tested with varying numbers of context views (2, 4, and 6 views) on DL3DV.
• Efficiency: While LocusGS has a slightly higher parameter count (241.5M vs. 222.0M for TokenGS) and forward latency, it achieves superior reconstruction quality without increasing the number of predicted Gaussians.

Significance and Claims

The paper claims that explicit 3D anchor states provide an effective spatial prior for query-based feed-forward 3DGS. By grounding queries in 3D space, LocusGS transforms Gaussian tokens from implicit latent embeddings into interpretable spatial reconstruction queries.

The authors argue that this approach does not merely improve rendering accuracy but fundamentally enhances the spatial organization and interpretability of the representation. The results demonstrate that explicit spatial states encourage queries to specialize in coherent local regions, leading to more structured Gaussian distributions that better align with the scene geometry. The work suggests that incorporating geometric states into the query mechanism is a crucial step toward more robust and structured 3D scene reconstruction.