💻 computer science

ImprovedVBGS: Real-time Continual Variational Bayes Gaussian Splatting

The paper introduces ImprovedVBGS, an accelerated framework for real-time continual Variational Bayes Gaussian Splatting that achieves a 1680x speed-up in per-frame latency through spatially truncated variational inference and optimized reassignment, enabling on-the-fly reconstruction without sacrificing quality.

Original authors: Damani Mguni-Coker

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Damani Mguni-Coker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to see the world and build a 3D map of it while it walks around, rather than showing it a giant photo album of the whole place first. This is the challenge of "on-the-fly" reconstruction, a super-important skill for self-driving cars and robots that need to navigate new places without crashing. To do this, scientists use a clever trick called "Gaussian Splatting." Think of this like painting a 3D scene not with solid blocks, but with millions of tiny, fuzzy, colored clouds (Gaussians). Each cloud has a position, a size, a rotation, and a color. By stacking enough of these fuzzy clouds, the computer can create a picture so realistic you can look at it from any angle.

However, there's a catch. Usually, to make these clouds perfect, the computer needs to look at every single photo of the scene at the same time and tweak the clouds over and over again. But for a robot walking down a street, it can't wait to see the whole future; it only sees one frame at a time. If it tries to learn from just the new frame, it often forgets everything it learned about the previous frames—a problem called "catastrophic forgetting." A previous method called VBGS (Variational Bayes Gaussian Splatting) solved the forgetting problem by using a smart math trick that lets the robot learn from new photos without needing to store old ones. But there was a big downside: it was incredibly slow. It was like trying to paint a masterpiece by checking every single pixel in the entire universe for every new brushstroke. It took over a minute just to process one frame, which is useless for a robot moving in real-time.

This paper introduces ImprovedVBGS, a new framework that speeds up this process so much it becomes practical for many applications, though it still faces a gap to full real-time video. The researchers took the slow, careful method of VBGS and gave it a massive makeover to make it lightning-fast.

First, they realized that when the robot looks at a new spot, it doesn't need to check every single fuzzy cloud in the entire 3D world to see which one fits best. Most clouds are far away and irrelevant. So, they added a "spatially truncated" step. Imagine you are looking for your lost keys in a messy room. Instead of checking every drawer in the entire house, you only check the drawers in the room you are currently standing in. ImprovedVBGS does exactly this: it builds a quick map (a KD-tree) and only checks the nearest neighbors for each new piece of data. This cuts down the work from checking millions of possibilities to just a handful.

Second, they fixed a clumsy part of the process called "reassignment." In the old method, the computer would constantly stop, erase, and rewrite its own code (a process called dynamic recompilation) every time the number of "bad" clouds changed. This is like a chef stopping to rewrite their entire recipe book every time they decide to add a pinch of salt. The new method uses "static tensor padding," which is like keeping a fixed-size recipe book where empty slots are just left blank. This stops the computer from wasting time rewriting its own instructions. They also found a way to "forward" information, meaning they reuse calculations they just made instead of doing them twice, trading a tiny, almost invisible drop in picture quality for a huge gain in speed.

The results are staggering. On a standard gaming graphics card (an RTX 3070 Ti), the original method took 84.0 seconds to process a single frame. ImprovedVBGS reduced the mean latency across all scenes to 0.133 seconds (about 7.5 frames per second). While the paper notes that a specific configuration without reassignment could reach 0.050 seconds, the full system that maintains high-quality reconstruction and handles the reassignment step runs at roughly 7.5 fps. This is a massive 1,680 times speed-up compared to the original. While the old method was too slow for a robot to use while moving, the new method is significantly faster, bringing it much closer to real-time capabilities, though it may still lag slightly behind standard 30fps video. The authors show that despite these massive speed tricks, the quality of the 3D map remains high, with the reconstructed images looking just as good as the slower, original version. They tested this on a standard set of 3D scenes (the NeRF Synthetic dataset) and found that the new system could handle the workload without needing to store a massive library of past images, making it perfect for robots and devices with limited memory.

Technical Summary: ImprovedVBGS

Problem Statement

Real-time on-the-fly 3D reconstruction is a critical requirement for robotics and autonomous navigation. While 3D Gaussian Splatting (3DGS) has established itself as a standard for high-quality novel-view synthesis, its standard gradient-based optimization assumes simultaneous access to all training views. In continual learning scenarios where data arrives sequentially, gradient-based methods suffer from catastrophic forgetting unless they employ replay buffers, which incur prohibitive memory and compute costs.

Variational Bayes Gaussian Splatting (VBGS) offers a solution by formulating scene reconstruction as variational inference over a probabilistic mixture model. This approach is inherently immune to catastrophic forgetting because it accumulates sufficient statistics without revisiting past data or requiring replay buffers. However, the original VBGS implementation is too slow for real-time applications. Its per-frame iteration requires evaluating the responsibility of every component for all observed points ($O(nK)$), leading to latencies of approximately 84 seconds per frame on consumer hardware. Furthermore, the original implementation requires depth input, uses significantly more parameters than standard 3DGS, and lacks view-dependent color modeling via spherical harmonics.

Methodology

The paper presents ImprovedVBGS, an accelerated framework designed to enable real-time continual reconstruction on consumer-grade hardware (specifically an RTX 3070 Ti). The methodology builds upon prior work that introduced kernel fusion and mixed-precision search, adding three primary optimizations:

1. Spatially Truncated Variational E-Step

The core bottleneck in VBGS is the Variational E-step, which computes the expectation over component assignments for every data point. The authors observe that spatial likelihood concentrates mass on nearby means. To address this, they implement a KD-tree nearest-neighbour candidate pruning strategy:

A KD-tree is built on the spatial means $\{\mu_{s,k}\}$ of the components.
For each observed point, only the $C$ nearest-neighbour component indices are queried.
The log-scores and soft responsibilities are computed exclusively on this truncated subset rather than the full set of $K$ components.
This reduces the computational complexity from linear in the number of components to a function of the nearest-neighbour count, significantly accelerating the E-step.

2. Improved Reassignment with Forwarding

In continual learning, "reassignment" is necessary to relocate unused mixture components to poorly modelled regions of the scene.

Mechanism: Components with the lowest Dirichlet posterior weights (indicating they are unused) are identified and moved to points with the lowest Evidence Lower Bound (ELBO) scores.
Optimization (Forwarding): The paper notes that ELBO values are already computed during the truncated E-step. Instead of performing a dense ELBO rescan for reassignment, the method forwards these pre-computed values directly.
Trade-off: This reordering of steps (running reassignment before the variational EM step) allows relocated components to receive same-frame evidence. While this results in a minor drop in Peak Signal-to-Noise Ratio (PSNR), it drastically reduces latency by eliminating redundant computations.

3. Static Tensor Padding

The original implementation suffered from frequent JAX recompilation because the number of components to reassign ( $n_{reassign}$ ) varied dynamically per frame, changing tensor shapes.

Solution: The authors fix the reassignment array size across frames by padding indices to a static compile-time shape ( $n_{max} = \lfloor f \cdot N \rfloor$ ).
Impact: This eliminates the overhead of dynamic graph recompilation, ensuring consistent and low-latency execution.

Key Contributions

The paper's primary contributions are the acceleration of VBGS to make it viable for edge devices and consumer graphics cards without sacrificing the "replay-free" nature of the algorithm:

Spatially Truncated Variational Inference: A KD-tree based pruning strategy that reduces the E-step complexity while maintaining reconstruction quality.
Efficient Reassignment: A pipeline that reuses truncated E-step outputs (forwarding) and employs static tensor padding to eliminate dynamic recompilation and redundant ELBO calculations.
Implementation Baseline: A reproduction of the fused-statistics and mixed-precision techniques from prior work [11], integrated with the new optimizations to demonstrate performance on an RTX 3070 Ti (8 GB VRAM), a more constrained environment than the A5000 (24 GB) used in previous studies.

Results

Experiments were conducted on the NeRF Synthetic dataset (8 scenes) using an RTX 3070 Ti. The results demonstrate a massive reduction in latency while maintaining reconstruction quality comparable to the baseline VBGS.

Latency Reduction: The mean per-frame latency was reduced from 84.0 seconds (Baseline VBGS) to 0.050 seconds (ImprovedVBGS with large batch and truncation). This represents a 1,680× speed-up.
Quality: The method maintains high reconstruction quality. For example, on the "Lego" scene, the PSNR remained stable around 20.64 dB even with aggressive truncation and large batch sizes. When reassignment is included, PSNR improves to ~21.5 dB, comparable to the baseline's 20.65 dB (with reassignment) and the original VBGS [10] results.
Ablation Insights:
- The Truncated E-step alone reduced latency from 41.0s to 3.39s.
- Increasing the batch size from 100 to 250,000 (made possible by memory optimizations) further reduced latency to 0.050s.
- Reassignment Forwarding provided a small speed-up (0.107s vs 0.131s) at the cost of a slight PSNR decrease (21.37 vs 21.57), demonstrating a tunable trade-off between speed and accuracy.
- Static Tensor Padding was critical for stabilizing latency, reducing it from 0.373s to 0.131s in the reassignment-heavy configuration.

Significance and Claims

The paper claims that ImprovedVBGS makes continual, replay-free 3D reconstruction practical on consumer hardware. By reducing per-frame latency from minutes to milliseconds, the framework addresses the strict memory and latency requirements of real-time applications in robotics and autonomous navigation.

The authors emphasize that unlike gradient-based continual learning methods that rely on replay buffers or local optimization (which increase storage or compute costs), ImprovedVBGS eliminates catastrophic forgetting by design through variational inference. The work demonstrates that with spatial truncation and efficient reassignment strategies, the computational overhead of VBGS can be reduced by orders of magnitude, enabling "on-the-fly" reconstruction without the need for expensive high-end GPUs or large memory footprints.