🤖 machine learning

A Query-Efficient Stochastic Volume Rendering Framework for Time-Varying Implicit Neural Volumes

This paper presents a query-efficient stochastic volume rendering framework based on delta tracking that leverages heterogeneous GPU parallelism and adaptive query reduction strategies to enable high-fidelity, interactive rendering of time-varying implicit neural volumes at 30–40 FPS on consumer hardware.

Original authors: Alper Sahistan, Haichao Miao, Zhimin Li, Peer-Timo Bremer, Joshua A Levine, Valerio Pascucci

Published 2026-07-31

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Alper Sahistan, Haichao Miao, Zhimin Li, Peer-Timo Bremer, Joshua A Levine, Valerio Pascucci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a scientist trying to watch a movie of a complex event, like a swirling storm or a beating heart, but the movie isn't made of frames. Instead, it's a magical, continuous recipe that can describe the scene at any moment in time, even the tiny fractions of a second between the frames. This is the world of Implicit Neural Representations (INRs). Think of them not as a stack of photos, but as a super-smart, compact neural network that knows the shape and color of an object at any coordinate $(x, y, z)$ and any time $(t)$ . The catch? To see what the object looks like, you have to ask this neural network a question: "What is the color here?" The network then has to do a lot of heavy mental math to answer. In the past, asking this question for every single pixel on a screen was so slow and expensive that watching these movies in real-time was impossible. Scientists were stuck with blurry, low-quality views or had to wait minutes for a single frame to appear.

This paper introduces a clever new way to watch these "neural movies" in real-time, achieving a smooth 30 to 40 frames per second on a powerful gaming computer. The authors, Alper Sahistan and their team, realized that the old way of looking at these volumes—checking every single point along a ray of light like a slow, methodical walker—is too wasteful when the "walker" has to stop and ask a neural network for help at every step. Instead, they built a query-efficient stochastic volume rendering framework. They treat the rendering process like a game of "guess and check" using a technique called delta tracking. Imagine you are walking through a foggy forest where the density of the fog changes. Instead of taking tiny, equal steps and checking the fog at every inch, you take big, random leaps. If you land in a thick patch, you stop and paint it; if you land in a thin patch, you keep leaping. By doing this, they ask the neural network far fewer questions.

To make this even faster, they split the work between two different types of computer brains on the graphics card. The "traversal" part (deciding where to leap) uses specialized ray tracing cores, while the "evaluation" part (asking the neural network) uses tensor cores designed for heavy math. They also added smart strategies to skip questions entirely in areas that look the same (homogeneity pruning) and to only re-check pixels that are changing (adaptive ray budgeting). The result is a system that can render a deforming, time-varying object at 1024² resolution with ray-traced shadows, updating the time step in just 1 to 2 milliseconds. The paper suggests that this approach allows scientists to explore continuous time directly, without needing to retrain the neural network or store massive caches of data for every single moment, making the invisible visible in a way that feels truly interactive.

The Magic of the "Leaping" Ray

To understand how this works, let's picture a ray of light as a curious explorer trying to find the color of a mysterious, invisible object. In the old days, this explorer would take tiny, baby steps along a straight line, stopping at every single step to ask a librarian (the neural network), "What color is it here?" If the object is huge and the steps are tiny, the explorer asks the librarian millions of times. Since the librarian is busy doing complex math, the explorer gets stuck waiting, and the movie freezes.

The authors' new method, delta tracking, changes the explorer's strategy. Instead of baby steps, the explorer takes big, random jumps. They have a rule: "I know the fog can't be thicker than this maximum limit." So, they jump a distance that would be safe if the fog were at that maximum limit. When they land, they ask the librarian, "Is the fog actually this thick here?"

If the answer is "Yes, it's thick," they stop and paint the color.
If the answer is "No, it's actually thin," they ignore the landing spot and take another big jump.

This is like playing a game where you only stop to check the score if you land on a "special" tile. Most of the time, you just keep flying through the air. Because the neural network is the slow part, skipping the questions where the answer is "nothing special" saves a massive amount of time.

The Two-Brain Team

The paper highlights a crucial insight: the way computers handle "walking" (traversal) and "math" (neural inference) is very different. Walking is like a single person checking a map step-by-step, while doing complex math is like a choir singing in perfect harmony. If you try to make the choir sing one note at a time, it's inefficient.

The authors built a four-stage pipeline that acts like a well-organized factory:

Ray Initialization: The factory sets up the explorers (rays) for the camera.
Traversal (The RT Cores): The "walking" experts (Ray Tracing cores) take the explorers and make them leap through the virtual space. They find the landing spots but don't ask the librarian yet. They just write down the addresses.
Evaluation (The Tensor Cores): The "math" experts (Tensor cores) take a huge list of addresses at once and ask the librarian for all the answers simultaneously. This is where the magic happens; the neural network is fed a batch of questions and answers them all in one go, using its full power.
Finalization: The results are brought back, and the explorers decide if they stop or jump again.

This "wavefront" approach ensures that the computer's brain is never idle. While one group is walking, the other is doing math. The paper shows that this method is about 125 times faster than a naive approach that tries to do everything one by one, and more than 2 times faster than using older methods that don't use the special ray-tracing hardware.

Smart Skipping: Not Asking Every Question

Even with the leap-and-check method, asking the librarian is still expensive. The authors added two "smart skip" features to make it even faster:

The "Boring Zone" Skip (Homogeneity-Based Query Pruning):
Imagine the explorer lands in a room that looks perfectly white and empty. If the room is known to be uniform (all the same color), why ask the librarian? The system checks if the area is "boring" (uniform). If it is, it just guesses the average color and keeps moving. This saves a question. However, the paper notes that if you guess too aggressively, you might miss small details, so they use a "stochastic falloff" to be careful near the edges of these boring zones.
The "Only Change What Moves" Skip (Adaptive Ray Budgeting):
Imagine you are watching a video. If the background is a still wall, you don't need to redraw it every second. You only need to redraw the parts where a bird is flying. The system looks at the previous frame and asks, "Did this pixel change?"
- If the pixel is stable (the wall), it skips drawing it and just uses the old color.
- If the pixel is changing (the bird), it spends its "budget" to draw it again.
  They tested three ways to decide what to draw: one that looks at how much the color changed (Residual-Histogram), one that uses a random but pretty pattern (STBN), and a mix of both. They found that mixing the random pattern with the "change detection" gave the best results, keeping the image looking smooth and natural even when skipping many pixels.

The Results: Smooth, Real-Time Science

The team tested their system on six different datasets, including X-ray scans of deforming objects and simulations of fluid flow. They ran everything on an NVIDIA RTX 4090 GPU at a resolution of 1024².

Speed: The system achieves 30 to 40 frames per second (FPS) for standard rendering and about 30 FPS even when adding complex shadows. This is fast enough for a smooth, interactive experience.
Responsiveness: When the user changes the time (the "t" in the recipe), the system updates in about 1 to 2 milliseconds. This means you can drag a slider and see the object deform instantly, without waiting.
Memory: The system is surprisingly efficient. The neural network itself is tiny (only 1–2 MB), and even with all the extra structures needed for the rendering, the total memory usage stays around 600 MB, which is much lower than traditional scientific visualization systems.

The paper explicitly rules out the idea that you need to retrain the neural network or convert the data into a grid of voxels (tiny 3D pixels) to make it renderable. They show that you can render the network directly as it was trained. They also argue that while caching (saving answers for later) works for static images, it breaks down for time-varying data because the answers become wrong as soon as time moves forward. Their method avoids this by calculating the answers on the fly but efficiently.

Why This Matters

For scientists studying things like how a heart beats, how a storm swirls, or how materials deform under stress, being able to see the data in real-time is a game-changer. Before this, they might have to wait minutes for a single frame or settle for a low-quality view. Now, they can interact with the data, spin it around, and watch it evolve continuously. The authors suggest that this framework opens the door to using these compact neural representations for interactive scientific visualization, allowing researchers to explore the "continuous time" of their data without getting bogged down by the heavy math required to see it.

The paper concludes that while there are still some limitations—like the need for careful setup of the "boring zone" thresholds or the fact that the system currently relies on specific NVIDIA hardware—the approach successfully bridges the gap between the compactness of neural representations and the need for interactive, high-quality visualization. It proves that you don't need to sacrifice the "neural" nature of the data to make it fast; you just need to ask the right questions, in the right order, and skip the ones you don't need.

Technical Summary: A Query-Efficient Stochastic Volume Rendering Framework for Time-Varying Implicit Neural Volumes

Problem Statement
Time-varying implicit neural representations (INRs) offer a compact, continuous encoding of scientific volumes, particularly for dynamic modalities like X-ray computed tomography (CT) of deforming objects. Unlike discrete meshes or voxel grids, INRs allow for arbitrary temporal navigation without loading separate timesteps. However, interactive rendering of these representations is hindered by the computational cost of neural inference. Conventional volume rendering methods, such as ray marching with dense sampling, require evaluating the neural network at every sample point, leading to prohibitive runtimes. Existing mitigation strategies—such as resampling onto grids, caching, or retraining—often compromise accuracy, introduce approximation errors, or fail to support continuous temporal exploration due to cache invalidation during time navigation. The core challenge is to render time-varying INRs directly, without modification or preprocessing, while achieving interactive frame rates and low-latency temporal updates.

Methodology
The authors propose a query-efficient stochastic volume rendering framework based on delta tracking (Woodcock tracking), reinterpreted specifically to reduce the number of expensive neural field evaluations. The system operates on a four-stage pipeline that decouples ray traversal from neural inference to maximize heterogeneous GPU utilization:

Ray Initialization: Rays are generated and intersected with the volume bounding box.
Traversal (RT Cores): Using NVIDIA OptiX, rays traverse a sparse acceleration structure (Geometry Acceleration Structure, GAS) composed of Axis-Aligned Bounding Boxes (AABBs) representing macrocells. Within each macrocell, delta tracking samples candidate collision points based on a precomputed local majorant (upper bound of extinction).
INR Evaluation (Tensor Cores): Candidate collision points are batched and evaluated on the GPU's tensor cores using a fused kernel. This stage leverages large, uniform batches to saturate tensor cores, contrasting with the serial nature of per-ray traversal.
Finalization: Results are accumulated, and statistics are updated for adaptive budgeting.

To handle the continuous nature of time-varying INRs, the framework constructs a temporal macrocell grid at startup. For intermediate timesteps, it linearly interpolates between precomputed grids. To address potential underestimation of extrema (which causes visual artifacts), the system employs "ghost passes"—rendering with sweeping transfer functions to probabilistically visit and refine macrocell bounds.

The framework further introduces two primary strategies for query reduction:

Adaptive Ray Budgeting: Three schemes (Residual-Histogram Thresholding, Spatiotemporal Blue-Noise (STBN) Thresholding, and Residual-Weighted STBN) dynamically allocate ray budgets. They prioritize rendering pixels in temporally or perceptually changing regions while reusing accumulated estimates for stable areas.
Homogeneity-Based Query Pruning: If a macrocell's scalar range is below a user-defined threshold, the system skips the neural query, substituting the evaluation with a precomputed expected scalar value. This is refined via an exponential moving average (EMA) to reduce bias over time.

Key Contributions

Four-Stage Rendering Pipeline: A design that decouples traversal from neural inference, coordinating ray tracing cores and tensor cores to operate directly on original INRs without retraining.
Adaptive Ray Budgeting Schemes: Three strategies that prioritize sampling in changing regions using inter-frame color variation and high-frequency cues, balancing convergence speed with perceptual noise characteristics.
Homogeneity-Based Query Pruning: A strategy to detect near-uniform regions and probabilistically avoid unnecessary neural evaluations, significantly reducing query counts with minimal visual artifacts when applied conservatively.

Results
The system was evaluated on six INR datasets derived from dynamic X-ray CT scans and Computational Fluid Dynamics (CFD) simulations, using an NVIDIA RTX 4090 GPU at $1024^2$ resolution.

Performance: The framework achieves 30–40 FPS for emission-absorption rendering and ~30 FPS with ray-traced shadows. This represents a ~125x speedup over a naive delta tracker and a ~2x improvement over a wavefront delta tracker using DDA traversal instead of RT cores.
Temporal Interaction: Timestep updates occur in approximately 1–2 ms, enabling interactive animation of continuous temporal deformations.
Memory Efficiency: The system maintains a low memory footprint (e.g., ~600 MB total GPU usage for smaller macrocell grids), significantly lower than typical scientific visualization systems.
Query Reduction: Adaptive budgeting and homogeneity pruning successfully reduce the number of neural queries. While aggressive pruning can degrade quality, moderate thresholds strike a favorable balance, improving FPS with limited impact on Peak Signal-to-Noise Ratio (PSNR).
Convergence: All strategies produce images that converge to within 2% of a ground-truth 400-frame rendering.

Significance
The paper claims that this framework enables the direct rendering and interactive exploration of time-varying INRs without the need for resampling, retraining, or caching as prerequisites. By leveraging the continuous nature of INRs and optimizing for the specific bottleneck of neural inference, the system allows scientists to navigate continuous time domains with low latency. The authors assert that this approach broadens the practical utility of neural representations in scientific visualization, facilitating richer exploration of dynamic volumetric data that was previously restricted by performance limitations. The work demonstrates that stochastic rendering, when combined with heterogeneous hardware acceleration and query reduction, can overcome the inference costs that have historically hindered interactive INR visualization.

The Magic of the "Leaping" Ray

The Two-Brain Team

Smart Skipping: Not Asking Every Question

The Results: Smooth, Real-Time Science

Why This Matters

More like this