Proxy-GS: Unified Occlusion Priors for Training and Inference in Structured 3D Gaussian Splatting

Proxy-GS introduces a fast, sub-millisecond proxy system to provide occlusion awareness that simultaneously accelerates rendering through efficient culling and improves training quality by guiding densification toward visible surfaces, thereby outperforming existing methods in both speed and fidelity for large-scale, occluded scenes.

The Gist

Imagine you are trying to build a hyper-realistic, 3D virtual world of a busy city street for a video game or a VR tour. You want it to look so real that you can see the texture of the bricks and the reflection in a window.

The Problem: The "Over-Enthusiastic" Builder
Current technology (called 3D Gaussian Splatting) builds these worlds by placing millions of tiny, fuzzy "clouds" (Gaussians) everywhere.

• The Issue: The builder is too enthusiastic. It places clouds in empty air, behind walls, and inside other buildings just to make sure every single photo looks perfect.
• The Result: Your computer has to process millions of useless clouds that the player will never see because they are hidden behind a building. It's like a chef cooking a massive feast for 1,000 people, but only 10 people are actually coming to dinner. The kitchen (your computer) gets overwhelmed, and the game runs slowly.

The Solution: The "Smart Proxy" (Proxy-GS)
The authors of this paper, Proxy-GS, introduced a clever two-step trick using a "Proxy." Think of a Proxy as a rough, low-poly sketch of the city. It's not the final beautiful building; it's just a simple cardboard cutout that shows where the walls and floors are.

Here is how they use this sketch to fix the problem:

1. The "Security Guard" (Occlusion Culling)

Before the computer tries to render the final, beautiful image, it first looks at the Proxy sketch.

• How it works: The computer asks the sketch, "If I stand here, what can I see?" The sketch instantly tells it, "You can't see the building behind that wall."
• The Magic: Because the sketch is so simple, the computer can answer this question in less than a millisecond (faster than a blink).
• The Result: The computer immediately tells the "enthusiastic builder," "Don't bother rendering the clouds behind that wall!" It cuts out the useless work instantly. This is like a security guard checking a guest list before letting people into a party, so the bouncer doesn't waste time trying to stop people who aren't even on the list.

2. The "Architect's Compass" (Training Guidance)

When the computer is learning how to build the city (training), it usually just tries to match the photos. Sometimes, it gets confused and builds clouds in weird, invisible places.

• The Fix: The Proxy acts like a compass. It tells the builder, "Hey, the ground is here, and the wall is there."
• The Result: The builder stops wasting time trying to fill invisible spaces. Instead, it focuses its energy only on the surfaces that actually matter. This makes the final building look sharper and more accurate.

The Analogy: Painting a Mural

Imagine you are painting a giant mural on a wall, but there are people standing in front of it.

• Old Way: You paint the whole wall, including the parts hidden behind the people. Then, you try to erase the parts you can't see. It takes forever, and you waste a lot of paint.
• Proxy-GS Way: Before you pick up a brush, you hold up a simple cardboard cutout of the people. You look at the cutout and realize, "Oh, I don't need to paint the part of the wall behind the guy in the red shirt." You only paint the visible parts.
  • Speed: You finish 2.5 times faster.
  • Quality: Because you aren't distracted by painting invisible spots, you can focus on making the visible parts look amazing.

Why This Matters

• Speed: On complex city scenes, this method runs 2.5 to 3 times faster than the previous best methods.
• Quality: The images look better because the computer isn't confused by "ghost" clouds in empty space.
• Real-World Use: This works even on regular gaming computers (like the RTX 4090), not just massive supercomputers. This means we can soon have ultra-realistic VR tours of entire cities that run smoothly on consumer hardware.

In short: Proxy-GS uses a simple, fast "sketch" of the world to tell the computer exactly what to ignore and what to focus on, resulting in a faster, sharper, and more efficient 3D experience.

Technical Summary

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has revolutionized real-time photorealistic rendering, standard implementations often generate excessive redundancy, particularly in large-scale scenes with complex occlusions (e.g., city streets, indoor environments).

• The Limitation of MLP-based Variants: Recent structured approaches (e.g., Scaffold-GS, Octree-GS) use Multi-Layer Perceptrons (MLPs) to decode Gaussian attributes from anchor points, improving visual fidelity. However, they suffer from high decoding overhead during inference.
• The Occlusion Blindness: Existing pruning and Level-of-Detail (LOD) strategies often fail to account for occlusion. Consequently, the system spends significant computational resources decoding and rendering Gaussians that are hidden behind other objects, leading to unnecessary latency without contributing to the final image quality.
• Hardware Mismatch: Consumer-grade GPUs (common in AR/VR) possess dedicated hardware rasterization units, but many 3DGS pipelines do not fully leverage these for efficient occlusion culling.

2. Methodology: Proxy-GS

The authors propose Proxy-GS, a unified framework that introduces occlusion awareness into both the training and inference stages of MLP-based 3DGS using a lightweight proxy mesh.

A. The Proxy System

• Construction: A coarse, lightweight proxy mesh is constructed for the scene. For outdoor scenes, this can be derived from dense point clouds (e.g., COLMAP). For indoor scenes, large-scale reconstruction models (like MapAnything) are used to generate dense point clouds, which are then converted to meshes.
• Efficiency: The mesh is simplified to retain only coarse geometric structures. It is partitioned into clusters to enable hierarchical visibility checks (Hierarchical Z-buffer).
• Hardware Acceleration: The system leverages hardware rasterization (GPU fixed-function units) to render a depth map from the proxy mesh. This process is extremely fast, generating a $1000 \times 1000$ resolution depth map in under 1 ms.

B. Inference: Proxy-Guided Culling

During rendering, the proxy system acts as a filter before the main Gaussian rendering pipeline:

1. Depth Acquisition: The proxy mesh is rendered to a depth buffer on the GPU.
2. Anchor Culling: For each anchor point intended for rendering, the system projects it to the image plane.
3. Visibility Check: The anchor's depth is compared against the proxy's depth map at the corresponding pixel. If the anchor lies behind the proxy surface (occluded), it is culled immediately.
4. Integration: This culling is fused with standard frustum culling in a single CUDA kernel, avoiding CPU-GPU round-trips. This significantly reduces the number of anchors passed to the MLP decoder and rasterizer.

C. Training: Proxy-Guided Densification

The proxy also guides the training process to prevent the growth of redundant anchors in occluded regions:

• Problem: Standard densification adds new anchors around Gaussians with high gradients. However, in occluded areas, these gradients may be artifacts, leading to anchors growing behind surfaces where they will never be seen.
• Solution: The method identifies patches with high rendering errors. Instead of blindly densifying, it back-projects these error regions onto the proxy mesh surface to determine the new anchor position.
• Constraint: New anchors are constrained to lie on or near the proxy geometry, ensuring they represent visible surfaces and improving structural consistency.

3. Key Contributions

1. Unified Occlusion-Aware Pipeline: A novel framework that integrates lightweight proxy meshes into both the training (densification) and inference (culling) stages of structured 3DGS.
2. Ultra-Fast Depth Acquisition: A hardware-rasterization-based approach to generate high-resolution depth maps in <1 ms, enabling real-time occlusion culling with negligible overhead.
3. Structured Densification: A proxy-guided densification strategy that prevents anchors from growing in occluded regions, leading to more geometrically meaningful representations.
4. Performance Gains: Demonstrates that occlusion awareness is critical for large-scale scenes, achieving significant speedups without sacrificing (and often improving) rendering quality.

4. Experimental Results

The method was evaluated on large-scale urban datasets (MatrixCity), indoor scenes (ZipNeRF), and real-world street/aerial scenes.

• Rendering Speed (FPS):
  • On the MatrixCity dataset (highly occluded urban streets), Proxy-GS achieves a 2.5× to 3× speedup over the state-of-the-art Octree-GS baseline.
  • Specifically, FPS increased from 32 (Octree-GS) to 150 (Proxy-GS) in dense street scenes, while maintaining or improving quality.
• Rendering Quality:
  • Proxy-GS consistently outperforms baselines in PSNR, SSIM, and LPIPS.
  • In MatrixCity Block 5, it achieved 21.68 PSNR (vs. 21.41 for Octree-GS) with 151 FPS (vs. 48 FPS).
  • Qualitative results show better preservation of fine details (e.g., windows, crosswalks) and fewer artifacts in occluded regions.
• Ablation Studies:
  • Training vs. Inference: Using the proxy only for inference (culling) boosts speed but drops quality due to training-inference mismatch. Using it for both training and inference yields the best balance of speed and quality.
  • Proxy Fidelity: The method is robust to coarse proxy resolutions (down to 1% of original) and minor vertex noise (<5%), as urban scenes are dominated by planar surfaces where coarse geometry suffices for occlusion logic.

5. Significance

• Practicality for AR/VR: By leveraging consumer-grade GPU hardware rasterization, Proxy-GS makes high-fidelity, MLP-based 3DGS viable for real-time applications on standard hardware, overcoming the latency bottlenecks of previous methods.
• Scalability: It addresses the specific challenge of large-scale, occlusion-rich environments (like city walkthroughs), where traditional pruning fails.
• New Paradigm: The work establishes that explicit geometric priors (proxies) can effectively guide neural rendering pipelines, bridging the gap between explicit geometry efficiency and neural representation fidelity.

In summary, Proxy-GS solves the "redundant decoding" problem in structured 3DGS by using a fast, hardware-accelerated proxy to filter out invisible content, resulting in a system that is both faster and higher quality than current state-of-the-art methods.