Fourier Splatting: Generalized Fourier encoded primitives for scalable radiance fields

The paper introduces Fourier Splatting, a novel radiance field rendering method that utilizes Fourier-encoded planar surfels to enable a single model to be rendered at varying levels of detail through coefficient truncation, thereby achieving scalable high-fidelity visualization without relying on primitive pruning.

The Gist

Imagine you are trying to build a realistic 3D world inside a computer, like a video game or a virtual museum. In the past, to make these worlds look good, developers had to use millions of tiny, simple building blocks (like smooth, round marbles or flat discs). If you wanted a sharp corner on a table or a delicate leaf on a tree, you had to use thousands of these tiny blocks to approximate the shape.

This works, but it's heavy. If you want to send this 3D world over the internet to a phone with a slow connection, you have to delete thousands of those blocks to save space. The problem? When you delete them, the image gets blocky and blurry. It's like trying to fix a blurry photo by throwing away half the pixels; you just get a worse picture.

Fourier Splatting is a new way to build these 3D worlds that solves this problem. Here is how it works, using some simple analogies:

1. The Old Way: The "Lego" Problem

Think of current 3D methods (like 3D Gaussian Splatting) as building a sculpture out of Lego bricks.

• If you want a smooth curve, you have to use hundreds of tiny bricks.
• If you want to make the sculpture smaller to fit in a small box (like a phone screen), you have to smash away whole bricks.
• Result: The sculpture loses its shape. A smooth curve becomes a jagged staircase. You can't make it "less detailed" without making it look bad.

2. The New Way: The "Magic Clay" (Fourier Splatting)

The authors of this paper invented a new kind of building block. Instead of a rigid Lego brick, imagine a piece of magic, stretchy clay.

• The Shape: This clay can be a perfect circle, but it can also stretch, squish, and twist into any shape you want (a star, a leaf, a jagged rock).
• The Secret Sauce (Fourier): How do you control the shape? You use a "recipe" made of Fourier coefficients. Think of these as knobs on a sound mixer.
  • Knob 1 (Low Frequency): Turns the clay into a simple circle.
  • Knob 2 (Medium Frequency): Adds a little bump or a wave to the side.
  • Knob 3 (High Frequency): Adds tiny, sharp details like the edge of a leaf.

3. The Superpower: "Volume Control" for Detail

This is the magic part. In the old Lego method, to reduce detail, you had to throw away bricks. In this new method, you just turn down the volume on the high-frequency knobs.

• High Quality (Full Volume): You turn all the knobs up. The clay shape is complex, sharp, and detailed.
• Low Quality (Muted): You turn off the high-frequency knobs. The clay automatically smooths out into a simple, round shape.
• The Result: You didn't delete any building blocks! You just changed the complexity of the existing ones. The object is still there, but it's now a "low-poly" version of itself.

4. Why This Matters (The "Bandwidth" Analogy)

Imagine you are streaming a movie.

• Old Method: To save data, the server deletes the actors' faces. You see a blob where a face should be.
• Fourier Splatting: The server keeps all the actors, but it sends a "blurry" version of their faces for slow connections. When you get a fast connection, it instantly sharpens them up. The data size changes, but the structure of the scene stays perfect.

5. The "Smart Assistant" (HYDRA)

There was one tricky problem: If you have a very complex shape (like a crumpled piece of paper), a single piece of magic clay might struggle to hold that shape perfectly.
The authors created a smart assistant called HYDRA.

• If a piece of clay is trying to be too complicated, HYDRA gently splits it into two simpler pieces that work together to hold the shape.
• It's like a teacher noticing a student is struggling with a giant math problem and breaking it down into two smaller, easier problems for them to solve.

Summary

Fourier Splatting is like upgrading from a box of rigid Legos to a set of smart, shape-shifting clay blobs.

• Old way: To save space, you throw away pieces, and the picture breaks.
• New way: You just "smooth out" the pieces. You can send a high-definition 3D world to a supercomputer, and the exact same world can be sent to a smartphone, where it automatically simplifies itself to look good without losing its structure.

It's a way to make 3D worlds that are scalable by nature, allowing us to see beautiful, high-fidelity scenes on any device, from a VR headset to a low-end phone, without the image falling apart.

Technical Summary

1. Problem Statement

Novel view synthesis has been revolutionized by 3D Gaussian Splatting (3DGS), which enables real-time rendering through explicit primitive rasterization. However, existing methods suffer from a fundamental scalability limitation:

• Static Primitives: Current approaches (3DGS, 2DGS, Triangle Splatting) rely on primitives with fixed geometric shapes (e.g., ellipsoids, discs, triangles).
• Fidelity vs. Count: Visual fidelity is strictly tied to the number of primitives. To improve quality, more primitives must be added; to reduce size (for bandwidth or memory), primitives must be pruned.
• Inefficient Downscaling: Existing scalable methods (e.g., Octree-GS) achieve Level-of-Detail (LoD) by pruning or filtering primitives. This often leads to "graceful degradation" issues where objects vanish or become blocky when resolution is reduced, rather than simply losing fine texture or shape detail.

The authors propose a solution where scalability is embedded within the primitive itself, allowing a single trained model to render at varying levels of detail without retraining or changing the primitive count.

2. Methodology: Fourier Splatting

The core innovation is the Fourier Splat, a planar primitive whose boundary is parameterized by a Fourier series rather than a fixed shape.

A. Primitive Parameterization

Instead of a fixed disc or ellipse, each surfel (surface element) $s_i$ is defined by:

• Standard Attributes: Center position $p_i$, rotation quaternion $q_i$, opacity $o_i$, and spherical harmonics for color.
• Fourier Boundary: The boundary of the primitive in its local tangent plane is defined by a closed curve $r_i(\theta)$:
  $$r_i(\theta) = R_i \cdot \left| \sum_{k=0}^{K-1} c_{i,k} e^{i k \theta} \right|$$
  Where $R_i$ is a circumradius, and $c_{i,k}$ are complex Fourier coefficients (amplitude and phase).
  • $K=1$: Reduces to a standard circular disc.
  • $K>1$: Allows the primitive to approximate arbitrary closed shapes (e.g., stars, polygons, complex organic shapes) by adding higher-frequency components.
• Normalization: Amplitudes are normalized via squared-$\ell_1$ to ensure the boundary never exceeds the circumradius $R_i$.

B. Differentiable Rendering

• Forward Pass: Uses a tile-based rasterizer. For each pixel, the signed distance to the Fourier boundary is calculated. A "power window" function determines opacity, decaying from the center to the boundary.
• Backward Pass (Straight-Through Estimator): A critical challenge is that the hard cutoff at the boundary creates a zero-gradient region outside the primitive, causing optimization to stall (primitives shrink rather than learn complex shapes).
  • Solution: The authors employ a Straight-Through Estimator (STE). In the backward pass, the hard window is replaced by a smooth surrogate function that provides non-zero gradients even for pixels outside the boundary. This allows the Fourier coefficients to "learn" to expand or contract the shape effectively.

C. Optimization and Densification (HYDRA)

The method adopts the MCMC (Markov Chain Monte Carlo) framework used in 3DGS-MCMC but adapts it for planar primitives.

• HYDRA (Hybrid Decomposition for Rendering-Aware Preservation): A novel densification strategy to handle complex geometries.
  • Scale-Preserving Split: For small primitives, they are split into two smaller copies.
  • Learned Lobe Decomposition: For large primitives with complex, multi-lobed boundaries, a geometric split is insufficient. A pre-trained MLP analyzes the boundary's "valleys" and splits the primitive into multiple child primitives, each representing a specific lobe. This preserves the MCMC probability distribution while increasing geometric expressiveness.

3. Key Contributions

1. Arbitrary-Shape Planar Primitives: The first splatting primitive that can assume arbitrary closed shapes via Fourier encoding, enabling higher fidelity representation of complex geometry compared to 2D/3D Gaussians.
2. Inherent Scalability: The ability to adjust the Level-of-Detail (LoD) at runtime simply by truncating Fourier coefficients.
   • Mechanism: Reducing $K$ (the number of active frequencies) smooths the primitive shape without removing the primitive itself.
   • Benefit: This provides a continuous rate-distortion trade-off where quality degrades gracefully (loss of fine shape detail) rather than abruptly (loss of entire objects).
3. MCMC-Compatible Densification: Introduction of HYDRA and the STE to enable stable training of these complex, non-convex primitives within a fixed budget.
4. State-of-the-Art Performance: Achieves superior rendering quality among planar methods and competitive results against volumetric methods.

4. Experimental Results

The method was evaluated on standard benchmarks: Mip-NeRF 360 and Tanks and Temples.

• Quantitative Performance:
  • Planar Methods: Fourier Splatting achieves State-of-the-Art (SOTA) results among all planar primitive methods (outperforming 2DGS, Triangle Splatting, and BBSplat).
    • Example: +0.67 dB PSNR improvement over Triangle Splatting on Mip-NeRF 360 average.
  • Volumetric Comparison: It competes favorably with volumetric methods (3DGS, DBS, GES), often surpassing them in LPIPS (perceptual similarity) and SSIM, despite using purely planar primitives.
• Qualitative Analysis:
  • Recovers fine details (thin structures, sharp silhouettes) better than 3DGS, which tends to blur them.
  • Avoids the "blocky" artifacts seen in Triangle Splatting when the underlying primitive shape is visible.
• Scalability Analysis:
  • Demonstrates that truncating coefficients results in a monotonic, graceful degradation of image quality.
  • Unlike Octree-GS (which removes primitives and causes objects to vanish), Fourier Splatting maintains object presence while simplifying shape complexity.

5. Significance and Impact

Fourier Splatting represents a paradigm shift in radiance field representations:

• Decoupling Fidelity from Density: It breaks the traditional link where higher quality requires exponentially more primitives. Instead, quality is controlled by the complexity of each primitive.
• Bandwidth Efficiency: This is particularly significant for bandwidth-constrained environments (e.g., VR/AR, mobile streaming). A single model can serve multiple devices: high-end devices receive full Fourier coefficients, while low-end devices receive truncated versions, all without storing multiple models.
• Geometric Fidelity: By preserving the planar surfel nature of 2DGS while adding shape flexibility, it offers a robust solution for surface reconstruction that is both geometrically accurate and computationally efficient.

In conclusion, Fourier Splatting provides a versatile, scalable, and high-fidelity framework for real-time radiance field rendering, solving the long-standing issue of how to efficiently downscale 3D scene representations without sacrificing structural integrity.