FACT-GS: Frequency-Aligned Complexity-Aware Texture Reparameterization for 2D Gaussian Splatting

1. Problem Statement

While 2D Gaussian Splatting (2DGS) has revolutionized real-time, high-quality novel view synthesis, its ability to model complex scene appearances is limited.

• Limitation of Standard 2DGS: It relies on view-dependent spherical harmonics, which provide a single color per primitive, lacking spatial color variation within a single Gaussian.
• Limitation of Existing Textured Gaussians: Recent methods (e.g., Textured GS) attach learnable texture maps to each Gaussian to enable spatial variation. However, these methods use a uniform sampling grid for texture parameterization.
• The Core Issue: Uniform allocation ignores local visual complexity.
  • High-frequency regions (edges, fine patterns, text) receive insufficient sampling density, leading to blurring and detail loss.
  • Low-frequency regions (smooth surfaces) waste texture capacity on redundant information.
  • Simply increasing texture resolution is inefficient, causing quadratic growth in memory and bandwidth with only marginal visual gains.

2. Methodology: FACT-GS

The authors propose FACT-GS (Frequency-Aligned Complexity-Aware Texture Gaussian Splatting), which reformulates texture parameterization as an adaptive sampling-density allocation problem. Instead of a fixed uniform grid, the method dynamically redistributes sampling density based on local signal frequency.

Key Technical Components:

1. Frequency-Aligned Reparameterization:

   • The method introduces a learnable deformation field ($D_i$) defined in the texture domain for each Gaussian primitive.
   • This field predicts continuous per-texel displacements, creating a continuous warp function $\Phi_i(u, v)$.
   • Mechanism: The warp expands high-frequency regions (allocating more texels) and compresses low-frequency regions (allocating fewer texels).

2. Jacobian-Based Density Modulation:

   • The local sampling density is controlled by the Jacobian determinant ($|\det J_{\Phi}|$) of the deformation field.
   • According to adaptive sampling theory, the target sampling density $\rho^*$ should be proportional to the gradient magnitude of the ground-truth color function ($\|\nabla C\|$).
   • During training, the photometric loss implicitly drives the optimization of the deformation field to shape the Jacobian determinant such that the induced sampling density aligns with the local visual complexity.

3. Differentiable Pipeline:

   • The warped coordinates $\Phi_i(u, v)$ are used to sample from the fixed-resolution RGB and opacity texture maps via bilinear interpolation.
   • The entire process is fully differentiable, allowing end-to-end training using standard photometric losses (L1, SSIM) without modifying the core Gaussian splatting rasterization pipeline.

4. Optimization Strategy:

   • Stage 1: Train 2D Gaussian primitives (geometry, rotation, scale) using standard losses.
   • Stage 2: Optimize the per-Gaussian textures ($T$) and deformation fields ($D$) while fine-tuning Gaussian parameters. No pruning or densification occurs in this stage.

3. Key Contributions

• FACT-GS Framework: A novel frequency-aligned texture reparameterization that reallocates sampling density based on local signal frequency under a fixed texture budget.
• Theoretical Reformulation: Recasting the texturing problem as a sampling-density allocation problem, solved via a learnable deformation field where the Jacobian controls local texture space distortion.
• Efficiency & Quality: Demonstrates that adaptive sampling yields sharper high-frequency details and higher perceptual fidelity compared to uniform sampling, without sacrificing real-time rendering performance.

4. Experimental Results

The authors evaluated FACT-GS on standard benchmarks (NeRF Synthetic, MipNeRF 360, DTU, Tanks & Temples, LLFF) against 2DGS, Textured GS, and other state-of-the-art methods.

• Quantitative Performance:

  • FACT-GS consistently achieves higher PSNR and SSIM, and lower LPIPS across all datasets.
  • Low-Budget Regime: The improvements are most significant when the primitive budget is restricted (e.g., 10% or 1% of default). Under these conditions, FACT-GS improves PSNR by up to +1.0 dB and reduces LPIPS by 22% compared to Textured GS.
  • Parameter Efficiency: FACT-GS achieves the same reconstruction quality as Textured GS using only 1/16th of the texture parameters.

• Qualitative Performance:

  • Detail Preservation: In low-budget scenarios, FACT-GS preserves sharp edges, printed characters, and fine material details that are blurred or lost in Textured GS and 2DGS.
  • High-Capacity Regime: Even with full primitive budgets, FACT-GS outperforms Textured GS by preventing the blurring of high-frequency patterns caused by uniform sampling.

• Efficiency (Inference Speed):

  • FACT-GS maintains real-time rendering performance (approx. 119–216 FPS on an RTX A6000), comparable to Textured GS. The deformation field adds negligible overhead as it is computed in parallel via a lightweight CUDA kernel.

5. Significance

• Paradigm Shift: The work shifts appearance modeling from uniform parameterization to signal-driven adaptive parameterization. It assigns representational capacity based on information density rather than spatial uniformity.
• Generalizability: While implemented on Gaussian textures, the principle of frequency-aligned sampling is applicable to other spatially parameterized representations (e.g., feature planes, neural textures).
• Practical Impact: It offers a solution to the "sampling-complexity mismatch" in 3D reconstruction, enabling higher visual fidelity in resource-constrained environments (e.g., mobile AR/VR) without increasing memory footprint or computational cost.