💻 computer science

Assessing the Feasibility of Real-Time Neural FluidRendering Using NVIDIA RTX Neural Shaders

This paper evaluates the feasibility of NVIDIA's RTX Neural Shaders for real-time fluid rendering, demonstrating that while neural networks successfully approximate pixel-level BSDFs at high frame rates, significant framework limitations currently prevent their effective deployment for vertex displacement tasks.

Original authors: Srihari Acharya, Thotreithem Hongray

Published 2026-07-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Srihari Acharya, Thotreithem Hongray

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a video game world, and you want the oceans to look real. In the past, artists had to write complex mathematical recipes, line by line, to tell the computer how to make water ripple, reflect light, and look wet. It was like writing a manual for every single drop of rain. But recently, a new idea has emerged: instead of writing the rules, why not teach the computer to learn them? This is the world of "neural rendering." Think of it like training a tiny, super-fast brain inside your graphics card. Instead of following a strict recipe, this brain looks at examples of how water behaves and figures out the pattern on its own. The big question researchers are asking is: Can we replace all those old, hand-written math recipes with these tiny, learned brains to make water look amazing in real-time, without the game slowing down?

This paper dives into that question using a specific new tool from NVIDIA called "RTX Neural Shaders." The researchers wanted to see if they could swap out the two main parts of a water shader: the part that moves the water (making waves) and the part that makes it shine (reflecting light). They set up a training system where a neural network tried to copy the behavior of a perfect, mathematical water shader.

The results were a tale of two very different stories. On the "shiny" side, the experiment was a huge success. The neural network learned to mimic how light bounces off water, oil, and thick fluids almost instantly. It could create realistic reflections, from the deep blue of the ocean to the glossy sheen of motor oil, running smoothly at 100 frames per second on an RTX 4060 graphics card. The computer took about 60 to 70 microseconds to figure out the color for every single pixel, proving that for making things look right, these tiny brains are a viable, real-time option.

However, the story took a sharp turn when they tried to use the neural network to move the water. They asked the brain to learn how to push the mesh vertices up and down to create Gerstner waves (the kind of rolling ocean swells you see in movies). This part failed spectacularly. When they tried to make the network slightly bigger to handle the complex math of moving waves, the entire application would freeze or crash. Even when it didn't crash, the water turned into a distorted, vibrating mess or simply vanished. The researchers found that the current framework has strict, hidden limits on how big these moving networks can be, and it struggles to handle the complex geometry of moving water. They also tried to train the network on a different computer and move the "brain" over, but the file formats were incompatible, acting like a locked door they couldn't pick.

So, what's the verdict? The paper suggests that while neural shaders are ready to take over the job of making water look beautiful and realistic, they aren't quite ready to take over the job of making it move in complex ways. The technology is promising for the visual effects, but the "muscle" side of the equation is still stuck in the training wheels phase, blocked by current hardware and software limitations. The authors conclude that we are on the right path, but the framework needs more tools and better documentation before it can handle the full dance of a real-time fluid simulation.

Technical Summary: Assessing the Feasibility of Real-Time Neural Fluid Rendering Using NVIDIA RTX Neural Shaders

Problem Statement
Real-time fluid rendering traditionally relies on hand-crafted mathematical shaders to balance visual realism with computational efficiency. These approaches, such as Gerstner waves for vertex displacement and analytical BSDFs for pixel shading, are highly optimized but rigid, requiring extensive manual tuning for different materials and lighting conditions. With the introduction of NVIDIA's RTX Neural Shaders (RTXNS) framework in early 2025, which allows small neural networks to execute directly within GPU pipelines via Tensor Cores, a critical question arises: Can neural shaders effectively replace both the geometric deformation (vertex) and optical shading (pixel) logic of fluid rendering? This work investigates the feasibility of using RTXNS to approximate these complex functions, aiming to understand the framework's capabilities and current architectural limitations rather than simply outperforming existing analytical methods.

Methodology
The authors implemented a dual-pipeline training system within the RTXNS framework to replace traditional shader logic with learned Multi-Layer Perceptrons (MLPs). The system was tested on an NVIDIA RTX 4060 GPU at 1080p resolution.

Dual-Pipeline Architecture:
- Pixel Shader Pipeline: Replaced the analytical fluid BSDF (Bidirectional Scattering Distribution Function) with a neural network. The ground truth utilized a two-stage architecture: a microfacet-based BRDF core (calculating Fresnel, GGX distribution, and geometry terms) followed by fluid-specific optical post-processing (translucency, depth absorption). The neural MLP approximated the core BRDF stage, taking six inputs (geometric dot products, roughness, IOR) and outputting four channels (specular term, view-dependent mix, scattering, Fresnel).
- Vertex Shader Pipeline: Attempted to replace procedural Gerstner wave displacement with a neural network. The ground truth was a generalized, material-dependent wave model incorporating viscosity ( $\mu$ ) and elasticity ( $\eta$ ) to modulate wave shape, temporal envelopes, and frequency. The neural MLP received eight inputs (position, time, and material parameters) to predict 3D displacement vectors.
Training and Inference:
- Training was conducted in real-time on the GPU using a host-side rtxns::HostNetwork class, dispatching forward/backward passes and optimizer steps (Adam) via compute shaders.
- The pixel pipeline utilized frequency encoding to expand input dimensions, while the vertex pipeline explored various architectures (V1–V5) in PyTorch for offline training attempts before attempting conversion to the proprietary .ns.bin format.

Key Contributions

Implementation of Dual Neural Pipelines: The work successfully designed and implemented neural approximations for both vertex displacement and pixel shading within a unified RTXNS pipeline.
Empirical Feasibility Assessment: The study provides a concrete evaluation of where RTXNS succeeds and fails. It demonstrates that neural approximation of smooth, bounded reflectance functions (pixel stage) is viable, while geometric displacement (vertex stage) faces significant hurdles.
Identification of Framework Limitations: The paper documents specific architectural barriers, including:
- Warp Alignment Constraints: The framework exhibits binary failure thresholds (e.g., 32 neurons work, 64 neurons cause immediate hangs), likely due to cooperative vector warp alignment issues in Tensor Core execution.
- Serialization and Tooling Gaps: The inability to import externally trained PyTorch weights due to the opaque .ns.bin format and the lack of diagnostic tooling for binary validation.
- Dependency Issues: The SlangPy training path is currently inaccessible in external environments due to unresolvable dependency conflicts with internal NVIDIA packages.

Results

Pixel Shader Success: The neural pixel shader achieved successful convergence within seconds. It produced visually accurate fluid appearances (water, oils, viscous fluids) across diverse materials at 100 FPS. The per-pixel inference time was 60–70 µs, slightly higher than the analytical baseline (20–30 µs) but sufficient for interactive applications. The network successfully learned the complex, non-linear relationships of the BSDF.
Vertex Shader Failure: The vertex pipeline encountered systematic failures.
- Instability: A 32-neuron network produced distorted, vibrating meshes that eventually collapsed or disappeared.
- Capacity Limits: Increasing the network to 64 neurons caused immediate, reproducible application hangs with no error messages, suggesting a hard limit in the current RTXNS scheduler or memory management.
- Deployment Failure: Attempts to train models offline in PyTorch and convert them to .ns.bin failed due to serialization incompatibilities and silent runtime crashes, preventing the deployment of more robust architectures.

Significance and Claims
The paper modestly claims to establish neural BSDF approximation as a viable technique for real-time applications using RTXNS, proving that compact MLPs can effectively learn complex optical behaviors. However, it explicitly identifies that neural vertex shader deployment is currently restricted by architectural barriers within the framework.

The significance of this work lies in its role as a "feasibility baseline" for an unexplored application domain. It highlights that while the hardware (Tensor Cores) and framework (RTXNS) offer a path toward learned rendering, the current software stack lacks the necessary documentation, validation tooling, and flexibility to support complex geometric learning. The authors conclude that the boundary between traditional shader programming and learned representation is blurring, but realizing the full potential of neural fluid rendering requires further refinement of the framework's internal constraints and tooling.

Technical Summary: Assessing the Feasibility of Real-Time Neural Fluid Rendering Using NVIDIA RTX Neural Shaders

More like this