ProGS: Towards Progressive Coding for 3D Gaussian Splatting

ProGS introduces a novel streaming-friendly codec that organizes 3D Gaussian Splatting data into an octree structure with mutual information enhancement, achieving a 45-fold reduction in file size and over 10% visual improvement while enabling progressive coding for varying bandwidth conditions.

The Gist

Here is an explanation of the ProGS paper, translated into simple, everyday language with creative analogies.

🌟 The Big Idea: "Streaming 3D Worlds Like a Netflix Movie"

Imagine you have a massive, incredibly detailed 3D model of a city. It's so detailed that it contains millions of tiny, glowing balloons (called 3D Gaussians) floating in the air to represent buildings, trees, and cars.

The Problem:
This model is huge. It's like trying to download a 4K movie file before you can even start watching it. If you have a slow internet connection, you can't see anything until the whole thing downloads. Also, storing this on your phone would fill up your memory instantly.

The Solution (ProGS):
The researchers created ProGS, a new way to pack and send these 3D worlds. Instead of sending the whole giant file at once, ProGS breaks it down into layers, like a Russian Nesting Doll or a Netflix video stream.

• Layer 1 (The Sketch): You get a blurry, low-quality version of the scene immediately. It's enough to know you're looking at a city.
• Layer 2 (The Outline): A bit more detail arrives. You can see the shapes of buildings.
• Layer 3+ (The HD): As your internet speed allows, more and more details (textures, colors, tiny cracks) are added until the image is crystal clear.

This means you can start "watching" the 3D scene instantly, even on a slow connection, and it gets better the longer you wait.

---

🏗️ How It Works: The "Tree" and the "Smart Foreman"

To make this magic happen, ProGS uses three clever tricks:

1. The Octree (The Family Tree of the City)

Usually, 3D models are just a messy pile of millions of balloons with no order. ProGS organizes them into a Tree Structure (specifically an Octree).

• The Analogy: Imagine a family tree.
  • The Grandparent (Root) represents the whole city.
  • The Parents represent neighborhoods.
  • The Children represent individual houses.
  • The Grandchildren represent the windows and doors.
• Why it helps: When you want to see the whole city, you only need the "Grandparent" data. If you zoom in, you ask for the "Parent" and "Child" data. This structure allows the computer to send only what you need right now.

2. The Smart Foreman (Adaptive Anchor Adjustment)

In the old methods, the "balloons" were just placed randomly. ProGS uses a Smart Foreman who constantly checks the construction site.

• The Analogy: Imagine building a house. If a room is empty, the Foreman removes the scaffolding there. If a room is complex (like a kitchen with many gadgets), the Foreman adds more scaffolding and workers.
• What ProGS does: It looks at the 3D scene and decides: "This part of the city is boring; let's use fewer balloons. This part is complex; let's add more balloons." It dynamically grows or shrinks the tree branches to save space without losing important details.

3. The "Telepathy" Trick (Mutual Information Enhancement)

This is the paper's biggest innovation.

• The Problem: When you only have the "Grandparent" (low detail), the image looks blurry. The "Children" (high detail) know the secrets of the house, but the "Grandparent" doesn't.
• The Solution: ProGS teaches the "Grandparent" to telepathically learn from the "Children."
• The Analogy: Imagine a student (Low Detail) and a professor (High Detail). Usually, the student only knows the basics. But ProGS uses a special training method (called InfoNCE) where the student is forced to understand the professor's deep knowledge before the professor even speaks.
• The Result: Even the blurry, low-quality version of the scene looks surprisingly good because the "blurry" parts have been trained to guess the details correctly based on the "sharp" parts.

---

🚀 Why This Matters (The Results)

The paper tested ProGS against the best existing methods, and the results were impressive:

1. Massive Space Savings: ProGS shrinks the file size by 45 times compared to the original format.
   • Analogy: It's like compressing a 100-page book into a 2-page summary that still tells the whole story.
2. Better Quality: Despite being smaller, the images look 10% sharper than other compressed methods.
3. Real-Time Streaming: Because it works in layers, you don't have to wait. You can start exploring a 3D world instantly, and it gets sharper as you watch, just like buffering a video but in 3D.

🏁 In a Nutshell

ProGS is like a smart, adaptive delivery service for 3D worlds.

• Instead of shipping a giant, heavy crate (the whole file) that takes forever to open, it sends a small box with a sketch.
• As you open the box, it automatically sends more layers of detail.
• It uses a family tree to organize the data, a smart foreman to remove unnecessary clutter, and telepathy to make the blurry parts look sharp.

This makes it possible to stream high-quality 3D scenes over the internet in real-time, even if your connection isn't perfect. It's a huge step forward for virtual reality, gaming, and digital twins.

Technical Summary

Here is a detailed technical summary of the paper "ProGS: Towards Progressive Coding for 3D Gaussian Splatting".

1. Problem Statement

3D Gaussian Splatting (3DGS) has emerged as a state-of-the-art technique for novel view synthesis, offering real-time rendering and high visual fidelity by representing scenes as collections of learnable 3D Gaussians. However, this representation suffers from massive data volume (often millions of Gaussians per scene), leading to significant storage and transmission challenges.

While existing compression methods (e.g., pruning, quantization, anchor-based approaches like Scaffold-GS and HAC) have reduced file sizes, they generally produce static files that must be decoded entirely. They lack progressive coding capabilities, which are essential for streaming applications where network bandwidth fluctuates. Current methods cannot adaptively transmit coarse data first and refine it with finer details as bandwidth allows, limiting their utility in real-time, variable-network environments.

2. Methodology: ProGS Framework

The authors propose ProGS, a streaming-friendly codec that organizes 3DGS data into a hierarchical octree structure to enable progressive coding. The methodology consists of four core components:

A. Hierarchical Octree Construction & Adaptive Anchor Adjustment

• Structure: Instead of a flat list of Gaussians, ProGS constructs an octree where nodes represent "anchors." Each anchor generates a local cluster of neural 3D Gaussians via Multi-Layer Perceptrons (MLPs).
• Levels of Detail (LoD): The scene is divided into multiple LoDs. LoD $l$ contains all anchors from level 1 to $l$. Higher LoDs provide finer spatial resolution.
• Dynamic Adjustment: During training, ProGS adaptively grows and prunes octree branches based on optimization gradients:
  • Growing: Significant Gaussians (identified by high gradients) trigger the creation of new anchor candidates at appropriate levels.
  • Pruning: Leaf nodes with low accumulated opacity (indicating low contribution to the scene) are removed to eliminate redundancy.
  • Parent-Child Verification: Ensures strict octree properties where every anchor has a valid parent, maintaining structural integrity for decoding.

B. Mutual Information (MI) Enhancement

A key challenge in progressive coding is that lower LoDs represent incomplete scenes, leading to poor visual quality. ProGS addresses this by maximizing the Mutual Information between parent and child nodes in the octree, ensuring lower-level anchors inherit informative content from higher levels.

• InfoNCE Loss ($L_{NCE}$): A contrastive learning objective that treats parent-child anchor pairs as positive samples and unrelated anchors as negatives. This forces the attributes of lower-level anchors to correlate strongly with their parents, improving their ability to represent the full scene.
• Coarse-to-Fine Optimization ($L_{c2f}$): A loss function that supervises lower-level anchors using the ground truth images directly. This encourages coarse-grained anchors to approximate the entire scene's distribution, bridging the gap between low-bitrate and high-fidelity rendering.

C. Context-Based Entropy Coding

To achieve high compression ratios, ProGS employs arithmetic coding (AE) with a context model:

• Hash-Grid Context: A mixed 3D-2D hash grid ($H$) is queried based on an anchor's position and its parent's position to generate context features.
• Adaptive Quantization: Small MLPs predict the mean and scale of the attribute distribution and an adaptive quantization step size based on the context.
• Bitstream Structure: The bitstream is organized as a Header (containing the hash grid and MLP parameters, transmitted once) followed by Chunks for each LoD. This allows for incremental decoding.

D. Training Pipeline

The training follows a staged strategy:

1. Initialization: Construct the octree from SfM point clouds.
2. Stabilization: Activate anchor adjustment and switch from single-level photometric loss to multi-level $L_{c2f}$ loss.
3. Quantization-Aware Training: Inject noise to simulate quantization effects.
4. Entropy Regularization: Add the entropy loss ($L_e$) to minimize bit cost while maintaining fidelity.

3. Key Contributions

1. First Progressive Codec for 3DGS: ProGS is the first framework to enable scalable, progressive decoding for 3D Gaussian Splatting, allowing adaptive bitrate transmission.
2. MI Enhancement Mechanisms: The introduction of InfoNCE loss and Coarse-to-Fine optimization significantly improves the visual quality of low LoDs, mitigating the degradation typically seen in incomplete scene representations.
3. Adaptive Octree Structure: A novel anchor adjustment strategy that dynamically grows and prunes the octree during training, optimizing the trade-off between storage and rendering quality.
4. Efficient Bitstream Design: A header-chunk pipeline that amortizes context model costs, enabling low-latency startup and incremental refinement.

4. Experimental Results

The authors evaluated ProGS on three real-world datasets: Mip-NeRF360, BungeeNeRF, and Tanks and Temples.

• Compression Efficiency: ProGS achieves a 45× reduction in file storage compared to the original 3DGS format. Compared to state-of-the-art (SOTA) anchor-based methods (like HAC), ProGS reduces the number of anchors by up to 67% while maintaining nearly 80% of the visual performance at the lowest bitrate.
• Visual Fidelity: At high bitrates (LoD 5), ProGS matches or exceeds the PSNR/SSIM of SOTA methods (e.g., HAC, CompGS).
• Progressive Performance:
  • LoD 1: Even at the coarsest level, ProGS provides usable visual quality with a file size reduction of over 144× compared to vanilla 3DGS.
  • Rate-Distortion (RD) Curves: ProGS demonstrates monotonic quality improvement as more LoDs are added, outperforming static compression methods in variable bandwidth scenarios.
• Performance Metrics:
  • Storage: On the Tanks and Temples dataset, ProGS-LR (Low Rate) achieves ~2.99 MB vs. 431 MB for original 3DGS.
  • Speed: Decoding is real-time. While training is ~3× slower than Scaffold-GS due to the complex optimization, the encoding/decoding time per level is lightweight (e.g., ~0.36s for decoding LoD 1 on a GPU), supporting real-time streaming.

5. Significance

ProGS represents a paradigm shift in 3D scene representation for streaming applications. By solving the "all-or-nothing" decoding limitation of current 3DGS compression methods, it enables:

• Adaptive Streaming: Seamless adaptation to fluctuating network conditions (e.g., mobile networks).
• Low-Latency Interaction: Users can start viewing a scene immediately with low-resolution data and refine it as data arrives, crucial for VR/AR and remote collaboration.
• Scalability: The hierarchical octree approach allows for efficient storage and transmission of massive 3D datasets without sacrificing the high-fidelity rendering capabilities that make 3DGS popular.

In conclusion, ProGS bridges the gap between high-fidelity 3D rendering and practical data transmission constraints, offering a robust solution for the future of volumetric scene streaming.