💻 computer science

3D Gaussian Accelerated Ray Tracing: Fast training through particle-based backward propagation

This paper introduces 3DGART, a practical training framework that accelerates 3D Gaussian ray tracing by reorganizing backward propagation from a pixel-centric to a primitive-centric approach, thereby eliminating atomic contention and achieving a 3–4× speedup while enabling high-quality, fully ray-traced rendering.

Original authors: Laurent Vit, Oliver Batchelor, Richard Green

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Laurent Vit, Oliver Batchelor, Richard Green

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can walk through a photograph, looking around a corner or stepping closer to a tree, and the scene shifts with perfect realism, just as it would in real life. For years, computer scientists have chased this dream of "novel view synthesis," trying to teach machines to understand the three-dimensional structure of a scene from a flat set of images. One of the most successful recent approaches involves representing a scene not as a solid object or a complex neural network, but as a cloud of millions of tiny, fuzzy blobs. These blobs, shaped like ellipses, carry information about color and transparency. By projecting these blobs onto a screen and blending them together, computers can create images so sharp and fast that they can be viewed in real time. This method, known as Gaussian Splatting, has revolutionized how we reconstruct 3D environments, making it possible to turn a simple video into a navigable 3D space almost instantly.

However, this speed comes with a compromise. The standard method relies on a technique borrowed from video games, where the computer approximates how these blobs look from the camera's perspective on a flat screen. While incredibly fast, this screen-based approach struggles with the complex physics of light. It cannot easily handle reflections bouncing off a shiny surface, light bending through glass, or the way shadows fall when an object blocks a ray of light. To solve this, researchers have tried a different path: instead of approximating the blobs on a screen, they let virtual rays of light travel through the scene and hit the blobs directly. This "ray tracing" method is physically accurate and handles reflections and shadows beautifully, but it has a fatal flaw: it is painfully slow to train. Creating a scene this way takes hours or even days, rendering it useless for many practical applications. The question remained: could we keep the physical accuracy of ray tracing without sacrificing the speed that made the original method so popular?

A team of researchers from the University of Canterbury in New Zealand has found a way to bridge this gap. They developed a new system called 3D Gaussian Accelerated Ray Tracing, or 3DGART, which makes training these accurate, ray-traced scenes fast enough to be practical. The breakthrough did not come from making the rays travel faster or finding a better way to bounce them off objects. Instead, the team discovered that the bottleneck was not in how the computer looked at the scene, but in how it learned from its mistakes. In the standard approach, when the computer tries to improve the image, it looks at the final picture pixel by pixel. If a single fuzzy blob contributes to the color of thousands of different pixels, thousands of computer processors try to update that same blob's settings at the exact same time. This causes a massive traffic jam in the computer's memory, where the processors have to wait in line to make their changes, slowing everything down to a crawl.

The researchers realized that the solution was to flip the process on its head. Instead of having thousands of processors fighting over the same blob, they organized the work so that each processor takes full responsibility for one specific blob and all the pixels it touches within a small section of the image. They built a temporary storage system that groups the necessary data by these blobs rather than by the pixels on the screen. This change transforms the chaotic traffic jam into a smooth, organized assembly line. Each processor gathers the information it needs for its assigned blob, calculates the necessary adjustments, and updates the settings without ever having to wait for others. It is a shift from a scattered, competitive process to a structured, cooperative one.

The results of this reorganization are striking. When tested on complex outdoor scenes, such as a bicycle parked in a garden, the new method trained the scene roughly four times faster than the previous best ray-tracing approach. More importantly, it did not just get faster; it produced higher quality images. Because the system no longer needed to use shortcuts or compromises to speed things up, it could preserve the full, accurate physics of light. The resulting scenes showed sharper details and more realistic lighting than the older, faster methods that relied on screen approximations. The researchers also found that while the new method uses more computer memory during the training process, this memory cost is a temporary trade-off that disappears once the scene is finished. At the end of the training, the final model is just as efficient to run as the original fast methods, but it carries the superior visual fidelity of a fully ray-traced world.

This work suggests that the future of 3D reconstruction does not require choosing between speed and accuracy. By rethinking how computers organize their learning process, the team demonstrated that we can have both. The method opens the door to creating highly realistic, interactive 3D environments that include complex effects like reflections and shadows, all while training in a timeframe that makes sense for real-world use. It turns a process that was once too slow to be practical into one that is ready for the next generation of virtual reality, digital twins, and immersive media, proving that sometimes the fastest way forward is not to push harder, but to organize better.

Technical Summary: 3D Gaussian Accelerated Ray Tracing (3DGART)

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has established itself as a dominant paradigm for real-time novel view synthesis, its reliance on screen-space rasterization introduces fundamental limitations. These include projection inaccuracies, view-dependent sorting artifacts, aliasing, and restricted support for secondary ray effects (reflections, refractions, shadows).

Gaussian Ray Tracing (e.g., 3DGRT) addresses these by evaluating explicit ray-primitive intersections, offering a conceptually cleaner formulation that supports perspective-correct visibility and complex light transport. However, ray-traced Gaussian methods have not displaced rasterization in practice due to prohibitive training times.

The authors identify that the primary bottleneck is not ray traversal itself, but the structure of the backward propagation. Existing differentiable ray-tracing pipelines utilize a pixel-centric formulation:

Each thread processes a ray and accumulates gradients into the parameters of intersected primitives.
Since a single primitive is often visible across many pixels, thousands of threads concurrently attempt to update the same memory locations.
This results in severe atomic contention and thread serialization, causing the backward pass to become the dominant computational cost, absorbing any gains made by efficient ray traversal.

2. Methodology

The paper presents 3DGART (3D Gaussian Accelerated Ray Tracing), a framework that reorganizes the backward pass from a pixel-centric scatter operation to a particle-centric (primitive-centric) gather-like computation.

Core Concept: Particle-Centric Backpropagation

Instead of assigning gradient accumulation ownership to pixels (which scatter updates to shared primitives), 3DGART assigns ownership to primitives over local regions of the image (tiles).

Forward Pass: Remains a standard hardware-accelerated ray-traced process.
Backward Pass: Each thread is responsible for a single primitive within a specific screen-space tile. The thread iterates over the pixels covered by that primitive within the tile, gathers the necessary intermediate values, accumulates local gradients in registers, and performs a single atomic update to the global primitive parameters.

Key Technical Components

A. Perspective-Correct Screen-Space Bounds

To efficiently determine which pixels are affected by each primitive without re-tracing rays, the system computes conservative, perspective-correct screen-space Axis-Aligned Bounding Boxes (AABB) for each Gaussian.

These bounds are derived using the affine transform mapping the normalized Gaussian space to world space and the camera projection matrix.
This defines the set of pixels that may interact with the primitive, ensuring consistency with the ray tracer's affine transform.

B. Compact Intermediate Buffer (LT Buffer)

A compact intermediate buffer stores the accumulated color ( $L \in \mathbb{R}^3$ ) and transmittance ( $T \in \mathbb{R}$ ) values required for backpropagation.

Memory Layout: The buffer uses a primitive-major layout. For each primitive, entries are grouped by tile and then by local pixel.
Access Pattern: This layout ensures that during the backward pass, reads for a specific primitive-tile pair are contiguous in memory, while the forward pass maintains collision-free writes.
Addressing: A global address is computed via a prefix sum over projected footprints, decomposing the address into a primitive base offset, a tile-local offset, and a pixel-local offset.

C. Tile-Primitive Mapping

To facilitate the backward pass, a mapping is constructed during preprocessing:

Tile-Primitive Pairs: Each primitive emits a pair $(tileID, primitiveID)$ for every tile its AABB footprint overlaps.
Radix Sort: These pairs are sorted by $tileID$, grouping all contributing primitives for each tile.
Tile Offset Table: A prefix sum over the number of primitives per tile allows the backward pass to quickly retrieve the list of primitives for any given tile.

D. Execution Strategy

The backward pass launches one CUDA block per screen-space tile (e.g., $8 \times 8$ tiles).
Threads within a block iterate over the primitive list for that tile.
Each thread processes one primitive, accumulates gradients over its covered pixels, and performs a final atomic update.
Hybrid Approach: To reduce VRAM usage, a hybrid scheme is proposed where color gradients (the primary source of atomic contention, especially for Spherical Harmonics) use the particle-centric formulation, while remaining terms retain the pixel-centric approach. This reduces memory consumption by a factor of 4 at the cost of a slight performance trade-off.

3. Key Contributions

Bottleneck Identification: The paper identifies atomic contention in the pixel-centric backward pass as the primary obstacle preventing practical ray-traced Gaussian training.
Intermediate Buffer Design: A compact intermediate buffer layout based on perspective-correct projected bounds, enabling efficient storage and retrieval of differentiation values.
Particle-Centric Backpropagation: A novel backward propagation scheme based on tile-primitive mapping that transforms gradient accumulation from a contention-heavy scatter operation into a structured gather-like process.
Performance Validation: Demonstration that 3DGART achieves significant speedups (up to $4\times$ ) over existing ray-traced baselines while improving reconstruction quality, bringing ray-traced training runtimes into the range of rasterization-based methods.

4. Experimental Results

Experiments were conducted on the Mip-NeRF 360, Tanks&Temples, and Deep Blending datasets using an NVIDIA RTX 4090.

Training Speed:
- On Mip-NeRF 360, 3DGART achieves a $3.51\times$ speedup over a per-pixel baseline with $1.0$ million primitives and $3.10\times$ with $2.0$ million primitives.
- Compared to 3DGRT (the state-of-the-art fully ray-traced approach), 3DGART is approximately $4\times$ faster (e.g., reducing training time from ~62 minutes to ~12 minutes on the Bicycle scene).
- The backward pass, which previously consumed 81–86% of iteration time in pixel-centric approaches, is reduced to a duration comparable to or faster than the forward pass.
Reconstruction Quality:
- 3DGART achieves higher PSNR, SSIM, and lower LPIPS scores compared to 3DGRT.
- Unlike 3DGRT, which relies on compromises (shorter Gaussian kernels, stricter opacity cutoffs, capped ray intersections) to mitigate contention, 3DGART preserves the full ray-traced Gaussian formulation without these heuristics.
Memory Scalability:
- The LT buffer memory usage scales linearly with the number of primitives and total pixels.
- For standard resolutions (up to ~1.5 megapixels), the additional memory overhead fits within modern GPU capacities.
- The hybrid approach allows training at higher resolutions (e.g., 4K) by reducing VRAM consumption by a factor of 4, albeit with a slight reduction in speedup.

5. Significance and Claims

The paper claims that 3DGART makes fully ray-traced Gaussian training practical. By restructuring the backward pass to be primitive-centric, the method eliminates the severe atomic contention that previously forced ray-traced methods to rely on accuracy-reducing heuristics.

The significance lies in:

Unification: It enables a unified representation for primary visibility, reflections, refractions, and shadows without sacrificing training efficiency.
Quality-Speed Trade-off: It achieves training speeds competitive with rasterization-based pipelines while preserving the superior visual fidelity and physical correctness of ray tracing.
Scalability: It demonstrates that ray-traced Gaussian fields can scale to large scenes and high primitive counts without the backward pass becoming a computational bottleneck.

The authors acknowledge limitations regarding non-perspective camera models (e.g., fisheye) due to the reliance on perspective-correct projection and the memory trade-off required for high-resolution training, suggesting these as areas for future work.