⚡ electrical engineering

Temporally Coherent Fluid Surface and Wave Reconstruction from Monocular Video

This paper proposes a monocular video-based framework that reconstructs temporally coherent fluid surfaces and fine-scale waves by combining Shape from Shading, shallow-water equation constraints, and a novel frequency-aware adaptive source transport mechanism to eliminate temporal discontinuities without requiring multi-view data, training sets, or physical simulations.

Original authors: Dong-Hyeon Shin, Jong-Hyun Kim

Published 2026-08-31

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Dong-Hyeon Shin, Jong-Hyun Kim

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Water is one of the most difficult subjects to capture in a video. It is constantly moving, reflecting light in chaotic ways, and changing shape with every passing second. For decades, computer scientists have tried to recreate this behavior on screens, often relying on complex physics simulations that calculate the pressure and velocity of every drop of water. While these simulations can look realistic, they are computationally expensive and struggle to match the specific motion of a real video. If a filmmaker wants to recreate the exact waves of a stormy ocean seen in a single camera shot, they usually cannot simply tell a computer to "make it look like this." They would need to measure the water's depth, speed, and pressure at every point, or use multiple cameras to see the water from different angles. Without that extra data, the computer is left guessing, often producing results that look smooth but lack the tiny, chaotic ripples that make water feel real.

A team of researchers at Inha University has developed a new way to solve this problem using only a single video recording. Their method allows a computer to reconstruct a dynamic, three-dimensional water surface from a standard RGB video, without needing extra cameras, depth sensors, or pre-calculated physics data. The core idea is to separate the water's motion into two distinct layers: the large, rolling swells that define the overall shape of the surface, and the tiny, high-frequency ripples that dance across the top. By treating these two elements differently, the researchers can preserve the smooth, stable motion of the big waves while keeping the sharp, fleeting details of the small ones. This approach creates a fluid surface that is not only geometrically accurate but also temporally coherent, meaning the water moves naturally from one frame to the next without flickering or losing its fine texture.

The process begins by looking at the video frame by frame. The computer analyzes the shading and brightness of the water in each image to estimate its height, a technique known as shape from shading. However, doing this for each frame independently creates a problem: the estimated height of the water might jump up and down randomly between frames due to changes in lighting or reflections, making the water look jittery. To fix this, the researchers use a mathematical rule based on how water conserves mass. They estimate a flow field, which is essentially a map of how the water is moving across the screen from one moment to the next. This flow field acts as a guide, ensuring that the changes in the water's height from one second to the next are consistent with how a real fluid would move. This step creates a stable, large-scale surface that captures the main direction and shape of the waves.

But a stable surface is not enough to look like real water. The smoothing process used to stabilize the large waves inevitably washes away the tiny details, like the sharp crests of small ripples or the chaotic splashes near a rock. To bring these details back, the researchers generate a separate layer of fine-scale waves. They look at two clues in the original video: the curvature of the large waves they just reconstructed, and the rapid, high-frequency changes in the video's brightness that cannot be explained by the large waves alone. These clues are combined to create a source of new wave energy. Instead of simply blending this new energy with the old one, which would blur the details, the researchers use a sophisticated transport method. They break the wave source into different frequency bands, separating the slow, rolling waves from the fast, sharp ripples.

For the slow, large waves, the system relies heavily on the history of the previous frame, ensuring the motion remains smooth and continuous. For the fast, sharp ripples, the system trusts the current frame much more, allowing the new details to appear instantly without being smoothed out by the past. This adaptive blending ensures that the water retains its fine texture while still moving smoothly over time. The resulting wave source is then fed into a solver that simulates how waves propagate, creating a layer of fine-scale motion that is added back to the large-scale surface. The final result is a complete water surface that combines the stability of the big waves with the chaotic energy of the small ones.

The researchers tested their method on a variety of videos, including ocean waves, calm undulating water, and fast-flowing streams in a valley. In every case, the system successfully reconstructed the water's surface without needing any external data. The large-scale geometry remained stable and consistent with the video, while the fine-scale wave layer added the necessary complexity to make the water look alive. Quantitative analysis showed that the method preserved the fine details much better than previous techniques that simply averaged the motion over time. The system was able to maintain the continuity of the wave patterns across hundreds of frames, preventing the flickering and loss of detail that often plagues computer-generated water.

One of the most significant aspects of this work is what it does not require. Unlike many other methods that need training data from thousands of examples, or multiple cameras to triangulate depth, this system works with a single video file. It does not need to know the physical properties of the water, such as its density or viscosity, nor does it need to measure the pressure or velocity of the fluid. It simply observes the visual changes in the video and uses the laws of fluid motion to infer the shape and movement of the surface. The researchers acknowledge that the system has limits; it cannot perfectly reconstruct breaking waves that crash and splash, or situations where the water surface folds over itself, because it represents the water as a single layer of height. However, for the vast majority of water scenes where the surface remains visible as a continuous sheet, the method provides a powerful and efficient way to bring the fluid to life.

The computational cost of this process is also notable. While the system is not yet fast enough for real-time rendering on a standard computer, it processes a video frame in about seventy milliseconds, which is close to the speed of human perception. The most time-consuming part of the process is the initial estimation of the water's shape and the calculation of the flow field, which together take up the majority of the processing time. The final step of adding the fine-scale waves is relatively fast, taking only a small fraction of the total time. This suggests that with further optimization, particularly by using the powerful graphics processors found in modern computers, the system could eventually run in real time.

Ultimately, this research offers a new way to understand and recreate the fluid world from a single perspective. By separating the problem into large-scale stability and fine-scale detail, and by intelligently managing how information flows from one moment to the next, the researchers have created a framework that captures the essence of moving water. The result is a digital surface that feels both physically grounded and visually rich, bridging the gap between the raw data of a video and the complex reality of a fluid in motion. This approach opens the door for more realistic visual effects in film and gaming, as well as new tools for analyzing fluid dynamics in scientific research, all starting from a simple video recording.

Technical Summary: Temporally Coherent Fluid Surface and Wave Reconstruction from Monocular Video

Problem Statement
Reconstructing dynamic fluid surfaces from a single RGB video is a significant challenge in computer graphics and visualization. While numerical simulations based on Navier-Stokes equations can generate realistic results, they require extensive tuning of physical parameters and initial conditions, making it difficult to reproduce specific motions observed in real footage. Conversely, direct observation-based reconstruction using monocular video avoids the need for multi-view setups, depth sensors, or precomputed fluid states. However, existing frame-wise reconstruction methods often suffer from temporal incoherence, where illumination changes, reflections, and noise cause height fields to fluctuate between frames. Furthermore, standard stabilization techniques, such as uniform temporal blending, tend to attenuate small-scale surface waves and high-frequency details, resulting in a loss of the fine-scale structures essential for realistic fluid appearance. The core problem addressed is how to reconstruct a large-scale free surface and preserve temporally coherent fine-scale waves from a single RGB video without relying on external simulation data, training sets, or physical measurements.

Methodology
The proposed framework reconstructs the fluid surface as a sum of two components: a large-scale base surface ( $H_{base}$ ) and a fine-scale wave layer ( $H_{wave}$ ), defined as $H_{final} = H_{base} + \lambda_w H_{wave}$ . The process involves two main stages:

Base Surface Reconstruction:
- Shape from Shading (SFS): An initial height field is estimated for each frame using SFS under a Lambertian reflectance assumption. To handle illumination changes, a fixed light direction is used, and outliers are removed based on 3D neighborhood distances.
- Flow-Constrained Velocity Estimation: To ensure temporal coherence, an image-space velocity field ( $u_t$ ) is estimated by minimizing an energy function that enforces the mass-conservation relation of the shallow-water equation ( $\partial h/\partial t + \nabla \cdot (hu) = 0$ ) while maintaining spatial smoothness. This velocity field acts as an auxiliary variable to connect consecutive frames rather than simulating full 3D internal flow.
- Coarse-to-Fine Residual Reconstruction: A Gaussian pyramid is constructed to decompose the SFS height field into residual bands. These bands are propagated independently using semi-Lagrangian advection and blended with current observations. This multi-scale approach recovers local geometric variations that might be smoothed out during velocity estimation, constructing the stable $H_{base}$ .
Fine-Scale Wave Reconstruction:
- Wave Source Extraction: Wave sources are generated from two cues: the Gaussian curvature of the base surface and motion-compensated high-frequency variations in the input video (computed as the difference between the current luminance and the advected previous luminance).
- Frequency-Aware Adaptive Source Transport: To address the trade-off between temporal continuity and detail preservation, the wave source is decomposed into low-, mid-, and high-frequency bands. Instead of using a fixed blending ratio, the method employs an adaptive transport mechanism. For each band, the agreement between the advected previous source and the current source is evaluated to compute a temporal confidence metric. The blending coefficient is adjusted spatially: low-frequency bands retain more historical data for stability, while high-frequency bands respond more strongly to current observations to preserve fine details.
- Wave Synthesis: The adaptively blended source is injected into an iWave-based surface-wave solver, which propagates the waves using a dispersive wave operator with temporal damping. The resulting wave field is advected by the same velocity field used for the base surface and combined with $H_{base}$ .

Key Contributions

Unified Framework: The paper proposes a video-based framework that reconstructs both large-scale free-surface geometry and temporally coherent fine-scale waves from a single RGB video, eliminating the need for multi-view images, training data, or measured velocity/pressure fields.
Physics-Guided Temporal Coherence: By connecting frame-wise SFS height fields using an image-space velocity field constrained by the shallow-water equation, the method ensures that consecutive height changes are consistent with fluid mass conservation.
Frequency-Aware Adaptive Transport: The authors introduce a novel source transport mechanism that decomposes wave sources into frequency bands. This allows for adaptive blending based on temporal consistency, effectively reducing high-frequency attenuation caused by uniform temporal smoothing while maintaining inter-frame continuity.
Comprehensive Evaluation: The method is validated on diverse datasets (ocean waves, streams, undulating water) with both qualitative visualizations and quantitative metrics demonstrating improved temporal coherence and detail preservation compared to frame-independent generation and uniform blending.

Experimental Results
Experiments were conducted on a sequence of 582 frames (25 fps) resized to a longer dimension of approximately 640 pixels.

Qualitative Results: The reconstructed surfaces successfully preserve large-scale wave directions and undulations in $H_{base}$ while adding rich, short-wavelength details in $H_{wave}$ . The final textured surfaces ( $H_{final}$ ) visually match the input video frames, reproducing wave orientation and fine surface structures without the temporal flickering common in frame-wise methods.
Quantitative Analysis:
- Wave Detail Preservation: The mean wave-detail RMS ( $E_{wave}$ ) was 0.01523, indicating persistent fine-scale contributions throughout the sequence.
- Temporal Coherence: The ablation study showed that the full method achieved a source discontinuity metric ( $E_{disc}$ ) of 0.058, significantly lower than frame-wise (0.184) and uniform blending (0.091) approaches.
- Detail Retention: While uniform blending reduced high-frequency energy to 78.2% of the frame-wise baseline, the full method retained 95.7%, demonstrating that frequency-aware transport effectively prevents the loss of fine details.
- Base Surface Stability: The correlation between the low-frequency components of $H_{base}$ and $H_{final}$ remained extremely high (0.99988), confirming that the addition of the wave layer does not distort the large-scale geometry.
Computational Performance: The pipeline runs at approximately 14.2 frames per second on a CPU (AMD Ryzen 9 9950X3D) for medium-resolution inputs. The frequency-aware adaptive transport adds only about 4.6% computational overhead (3.1 ms per frame) to the total pipeline time.

Significance and Claims
The paper claims to provide a practical approach for video-based free-surface reconstruction that operates without the constraints of multi-view capture, precomputed simulations, or deep learning training data. The significance lies in its ability to decouple large-scale motion from fine-scale wave dynamics, allowing for the recovery of details often lost in standard stabilization processes. The authors emphasize that their method constructs a geometrically consistent animation directly from observations rather than attempting to inversely recover a physically complete volumetric fluid state.

The work is modest about its limitations, acknowledging that the method relies on a single-valued height field representation, making it unsuitable for overturning waves, splashes, or breaking crests. Additionally, the reconstructed geometry is a proxy for relative surface shape rather than an absolute physical measurement, and the method remains sensitive to extreme specular reflections and rapid illumination changes. Future work is suggested to explore robust monocular depth estimation and multi-layer representations to handle more complex fluid phenomena.

More like this