MEGS$^{2}$: Memory-Efficient Gaussian Splatting via Spherical Gaussians and Unified Pruning

MEGS$^{2}$ is a novel memory-efficient framework for 3D Gaussian Splatting that achieves significant VRAM reduction by replacing spherical harmonics with lightweight spherical Gaussian lobes and employing a unified soft pruning strategy to jointly optimize the number of primitives and color lobes.

The Gist

Imagine you have a massive, incredibly detailed 3D model of a city. It's so realistic that you can walk through it, look at the shiny reflections on a car, and see the light filtering through leaves. This is what 3D Gaussian Splatting (3DGS) does. It's currently the best way to create these hyper-realistic 3D worlds.

But there's a huge problem: It's too heavy.

Think of this 3D city like a giant library. To show it on a computer screen, your computer has to carry the entire library in its backpack (its memory/VRAM).

• On a powerful desktop computer, the backpack is big enough, but it's still heavy.
• On a mobile phone (like an iPhone or Android), the backpack is tiny. The phone tries to put the whole library in, the backpack rips, and the app crashes. You can't even see the city.

The paper you shared, MEGS2, is a brilliant new way to shrink that library so it fits in a phone's backpack without losing any of the magic.

Here is how they did it, explained with simple analogies:

1. The Problem: The "Over-Engineered" Paintbrush

In the old way (3DGS), every little dot of light in the 3D world was painted using a complex mathematical tool called Spherical Harmonics (SH).

• The Analogy: Imagine you are painting a sunset. The old method forces you to use a giant, multi-bristled brush with 48 different colors mixed together to paint a single tiny speck of light. It's overkill. You need all those bristles to get the right shade, but it makes the paint bucket (memory) huge.
• The Result: Even if you delete half the dots (pruning), the remaining dots are still so heavy that your phone's memory fills up.

2. The Solution: The "Smart, Shape-Shifting" Paintbrush

The authors, MEGS2, decided to swap that giant brush for something smarter: Spherical Gaussians (SG).

• The Analogy: Instead of a giant 48-bristle brush, they use a tiny, flexible paint blob that can stretch and shrink.
  • If the light is simple (like a dull wall), the blob stays small (1 or 2 "lobes").
  • If the light is complex (like a shiny car reflection), the blob stretches out to catch the detail.
• The Benefit: You don't need 48 numbers to describe the color anymore; you might only need 3 or 4. This instantly makes the "paint bucket" much lighter.

3. The Secret Sauce: The "Unified Pruning" Chef

Just changing the paintbrush wasn't enough. They also needed to throw away the unnecessary stuff.

• The Old Way: Most previous methods tried to cut the problem in two steps. First, they would chop off 50% of the dots (pruning primitives). Then, they would try to shrink the paint on the remaining dots.
  • The Flaw: This is like a chef who first throws away half the ingredients, then tries to cook a smaller meal. Sometimes, you throw away the wrong ingredients, and the soup tastes bad.
• The MEGS2 Way: They use a Unified Soft Pruning framework.
  • The Analogy: Imagine a master chef who looks at the entire recipe at once. Instead of just chopping off ingredients, they ask: "If I reduce the number of dots and shrink the paint on each dot simultaneously, how can I get the best taste with the smallest pot?"
  • They treat the whole problem as one math equation. They gently nudge the system to remove the "lazy" dots and the "useless" paint blobs all at the same time.

4. The "Magic Trick": Color Compensation

When you throw away a paint blob (a "lobe"), the color might look a little dull.

• The Fix: MEGS2 has a clever trick. When it deletes a blob, it calculates exactly how much color that blob was contributing and adds that amount to the "base color" of the dot.
• The Analogy: It's like if you remove a specific spice from a stew, but you instantly adjust the salt and pepper to make sure the flavor stays perfect. The viewer never notices the ingredient is gone.

The Result: From "Crash" to "Smooth"

The paper shows some amazing results:

• Desktop: A scene that used to run at 27 frames per second (a bit choppy) now runs at 117 FPS (super smooth).
• Mobile: A scene that used to crash the phone (because it ran out of memory) now runs smoothly at 91 FPS.
• Memory: They reduced the memory needed by 8 times for storage and 6 times for rendering compared to the original method.

In a Nutshell

MEGS2 is like taking a heavy, bulky winter coat (the original 3DGS) and turning it into a high-tech, lightweight down jacket. It keeps you just as warm (the image looks just as good), but now you can wear it while running a marathon (on a mobile phone) without passing out from the weight.

It solves the "memory bottleneck" by using smarter paint (Spherical Gaussians) and a smarter way to cut the fat (Unified Pruning), making high-quality 3D worlds accessible to everyone, everywhere.

Technical Summary

1. Problem Statement

3D Gaussian Splatting (3DGS) has become the dominant paradigm for real-time novel view synthesis due to its high rendering speed and visual fidelity. However, its widespread adoption on edge devices (e.g., mobile phones, tablets) is severely limited by high memory consumption.

Existing compression methods primarily focus on storage compression (reducing file size) using techniques like vector quantization, neural compression, or hash grids. These methods often fail to address the critical rendering memory (VRAM) bottleneck because:

• They require decoding full Gaussian parameters into memory before rendering, often resulting in a runtime memory footprint larger than the original uncompressed data.
• They do not sufficiently reduce the number of primitives or the parameters per primitive required for the rendering pipeline.

The core challenge is to reduce the rendering VRAM without sacrificing visual quality, enabling high-quality 3DGS on resource-constrained hardware.

2. Methodology: MEGS2

The authors propose MEGS2, a framework that jointly optimizes two key factors determining memory usage: the total number of primitives and the parameters per primitive. The method consists of three main components:

A. Color Representation: Spherical Gaussians (SG)

The paper replaces the standard Spherical Harmonics (SH) used in 3DGS with Spherical Gaussians (SG).

• Limitations of SH: SH functions are global and require many high-order coefficients to model localized, high-frequency details (like sharp specular highlights), leading to low parameter efficiency.
• Advantages of SG: SG lobes are local and arbitrarily oriented. They can model view-dependent signals with significantly fewer parameters.
• Arbitrary Orientation: Unlike previous SG attempts that used fixed orthogonal axes, MEGS2 allows arbitrarily oriented lobes. This flexibility provides higher representation capability and better fits complex lighting.
• Prunability: The number of SG lobes per primitive is a tunable parameter, making it ideal for compression.

B. Unified Soft Pruning Framework

Instead of pruning primitives and parameters in separate stages, MEGS2 formulates a single constrained optimization problem to minimize the total memory budget.

• Objective: Minimize reconstruction loss subject to a memory constraint:
  $$\rho_o \|o\|_0 + \rho_s \|s\|_0 \leq \kappa$$
  Where $\|o\|_0$ is the count of active primitives (opacity), $\|s\|_0$ is the count of active SG lobes (sharpness), and $\kappa$ is the total parameter budget.
• Optimization Algorithm: Since the $L_0$ norm is non-differentiable, the authors employ an ADMM-inspired (Alternating Direction Method of Multipliers) approach.
  • They introduce proxy variables for opacity and sharpness.
  • The problem is split into a gradient descent step (updating parameters) and a proximal step (enforcing sparsity constraints).
  • This unified approach finds a better trade-off between reducing the number of Gaussians and reducing the complexity of each Gaussian compared to sequential pruning methods.

C. Post-Processing and Color Compensation

After optimization, primitives and lobes with near-zero values are hard-pruned. To prevent quality degradation from removing lobes:

• Color Compensation: The energy of removed SG lobes is mathematically integrated and added back to the primitive's diffuse color term ($c_0$).
• Fine-tuning: A brief fine-tuning phase recovers any remaining rendering quality.

3. Key Contributions

1. Fully Standalone SG Representation: MEGS2 completely replaces Spherical Harmonics with arbitrarily-oriented, prunable Spherical Gaussians. This reduces the per-primitive parameter count significantly while maintaining high-frequency detail.
2. Unified Soft Pruning: A novel framework that models primitive-count pruning and lobe-count pruning as a single memory-constrained optimization problem, outperforming staged or hard-pruning baselines.
3. Unprecedented Memory Compression: The method achieves massive reductions in both static and rendering VRAM while maintaining or improving rendering quality compared to State-of-the-Art (SOTA) lightweight methods.

4. Experimental Results

The authors evaluated MEGS2 on standard datasets (Mip-NeRF 360, Tanks & Temples, Deep Blending) and compared it against vanilla 3DGS, SG-Splatting, and various compression/pruning baselines (e.g., GaussianSpa, CompactGaussian, LightGaussian).

• VRAM Reduction:
  • Static VRAM: Achieved ~8x compression compared to vanilla 3DGS and ~2x compression compared to the SOTA method GaussianSpa.
  • Rendering VRAM: Achieved ~6x compression vs. vanilla 3DGS and ~40% reduction vs. GaussianSpa.
  • Absolute Numbers: On the Mip-NeRF 360 dataset, rendering VRAM dropped from 1.7GB (Vanilla 3DGS) to **265MB** (MEGS2 HQ).
• Rendering Quality:
  • MEGS2 maintains comparable or slightly superior PSNR, SSIM, and LPIPS scores compared to existing lightweight methods.
  • Qualitative results show superior handling of specular highlights and high-frequency details compared to methods using SH or fixed-axis SG.
• Real-World Performance (Edge Devices):
  • Desktop (RTX 3060): Achieved 117.4 FPS (vs. 27 FPS for 3DGS with SH).
  • Mobile (Dimensity 9400+): Achieved 91.0 FPS (vs. 6.6 FPS for 3DGS with SH).
  • Mobile (Snapdragon 865): Successfully ran at 34.9 FPS, whereas 3DGS with SH failed to render (N/A).
  • The method enables interactive 3D viewing on mobile devices where standard 3DGS crashes or runs unacceptably slowly.

5. Significance

MEGS2 represents a paradigm shift in 3DGS compression by prioritizing rendering memory efficiency over mere storage size. By addressing the root causes of VRAM bloat (excessive primitives and heavy color representations), it unlocks the potential for real-time, high-quality 3D rendering on consumer mobile devices. This makes applications like mobile AR/VR, real-time 3D scanning previews, and on-device gaming feasible, which were previously restricted by the memory constraints of current 3DGS implementations.