Learning Unified Representation of 3D Gaussian Splatting

Technical Summary: Learning Unified Representation of 3D Gaussian Splatting

Problem Statement

While 3D Gaussian Splatting (3DGS) has established itself as a powerful technique for high-fidelity, real-time 3D reconstruction and rendering, its native parameterization ($\theta = \{\mu, q, s, c, o\}$) presents fundamental challenges when used as a learning space for neural networks. The authors identify three critical misalignments between standard 3DGS parameters and the inductive biases of neural architectures:

1. Non-Uniqueness (Ambiguity): The mapping from parameters to rendered output is not one-to-one. Issues such as quaternion sign duality (where $q$ and $-q$ represent the same rotation) and symmetry-induced variances create "embedding collisions." Distinct parameter inputs producing identical visual outputs generate conflicting supervision signals, leading to training instability and poor convergence.
2. Numerical Heterogeneity: Parameter components span vastly different magnitudes and distributions (e.g., spatial positions in $\mathbb{R}^3$ vs. unit-normalized quaternions in $SO(3)$). This violates the homogeneous feature distribution assumption required for effective gradient flow in standard neural networks.
3. Manifold Mismatch: Parameters reside on distinct mathematical manifolds (Euclidean space for positions, Lie groups for rotations, spherical harmonics for color). Forcing these non-Euclidean variables into standard Euclidean feature spaces breaks their intrinsic geometric structure, making the representation difficult to compress, regularize, or generalize effectively.

These limitations result in unstable encoders, geometric "jitters" during latent interpolation, and poor robustness to noise, hindering downstream tasks like generation, compression, and editing.

Methodology

To address these issues, the paper proposes a Submanifold Field Representation that encapsulates the intrinsic information of Gaussian primitives through a continuous field defined on an iso-probability surface.

1. Submanifold Field Representation ($E$):
Instead of using raw parameters, each Gaussian primitive is represented as a pair $(M, F)$:

• Iso-Probability Surface ($M$): A 2D submanifold (an ellipsoid) defined by the equation $(x - \mu)^\top \Sigma^{-1} (x - \mu) = r^2$, where $r$ is a fixed radius. This surface captures the geometric shape and orientation.
• Color Field ($F$): A function defined on $M$ that maps points on the surface to view-dependent colors based on the original spherical harmonic coefficients.
  This formulation establishes a provably unique one-to-one correspondence between the Gaussian's radiance field and its representation, eliminating parameter ambiguities.

2. Submanifold Field Variational Auto-Encoder (SF-VAE):
To learn embeddings from this continuous field, the authors design a VAE framework:

• Discretization: The continuous field is discretized by uniformly sampling $P$ points from the iso-probability surface to form a colored point cloud.
• Encoder: A PointNet-based network encodes the point cloud into a latent vector $z$.
• Decoder: Two implicit neural networks reconstruct the original field:
  • A coordinate transform network $g_c$ that maps latent vectors and unit sphere coordinates to 3D positions.
  • A color field network $g_f$ that predicts colors for these positions.
• Parameter Recovery: The original Gaussian parameters ($\theta$) are recovered from the decoded point cloud using Principal Component Analysis (PCA) for the covariance matrix and spherical harmonic fitting for color coefficients.

3. Manifold Distance (M-Dist):
The authors introduce a novel loss function based on the Wasserstein-2 distance (Optimal Transport) to measure similarity between submanifold fields. This metric jointly considers spatial distance and color difference, offering a better correlation with perceptual quality than standard parameter-space distances (e.g., L1).

Key Contributions

• Formal Characterization of Limitations: The paper formally identifies and proves the non-uniqueness and numerical heterogeneity of native 3DGS parameters as fundamental barriers to neural learning.
• Unified Representation: It proposes a submanifold field representation that provides a unique, geometrically consistent, and numerically stable encoding of Gaussian primitives.
• Learning Framework: It develops an SF-VAE framework incorporating the Manifold Distance metric, demonstrating that learning in this unified space yields superior results compared to parametric baselines.

Experimental Results

The authors evaluate the method using a self-implemented VAE framework to ensure a controlled comparison, training on a randomly generated dataset of Gaussian primitives to ensure domain invariance.

• Reconstruction Quality: On both object-level (ShapeSplat) and scene-level (Mip-NeRF 360) datasets, the Submanifold Field representation significantly outperforms parametric baselines.
  • On ShapeSplat, SF-VAE achieves a PSNR of 63.408 compared to 37.512 for the parametric baseline.
  • On Mip-NeRF 360, SF-VAE achieves a PSNR of 29.833 versus 18.818 for the parametric baseline.
  • Perceptual metrics (SSIM, LPIPS) and the proposed M-Dist also show substantial improvements.
• Zero-Shot Generalization: Models trained on synthetic random data generalize effectively to real-world scenes, outperforming models trained on specific real-world domains and transferred to other domains. This suggests the representation strips away domain-specific priors.
• Robustness: The submanifold embedding space is significantly more robust to noise. Adding noise to the latent space results in much smaller reconstruction errors (measured by M-Dist) compared to parametric embeddings.
• Latent Space Regularity: Linear interpolation in the SF-VAE latent space produces smooth geometric transitions, whereas parametric interpolation results in unnatural "jitters" in rotation and scale.
• Downstream Applicability:
  • Unsupervised Clustering: SF embeddings yield clearer semantic separation in graph clustering tasks compared to raw parameters.
  • Gaussian Neural Fields (GNF): Using SF embeddings as targets for neural fields results in higher visual fidelity (e.g., PSNR 58.6 vs. 51.6 on ShapeSplat) compared to regressing raw parameters directly.

Significance and Claims

The paper claims that the proposed submanifold field representation resolves the fundamental theoretical misalignments between 3DGS primitives and neural learning. By enforcing a unique mapping and respecting the geometric structure of the data, the method enables:

1. Stable Training: Elimination of embedding collisions leads to better convergence.
2. High-Fidelity Learning: The representation preserves intrinsic geometric and photometric structures, allowing for superior reconstruction and generalization.
3. Unified Framework: A single model trained on synthetic data can generalize across diverse domains (objects and scenes), suggesting a path toward a truly unified representation for 3DGS.

The authors conclude that this approach provides a principled foundation for future learning-based tasks in 3DGS, including generative modeling, compression, and semantic understanding, while acknowledging current limitations regarding single-primitive modeling and the need for future work on scene-level encoders.