Looking Into the Water by Unsupervised Learning of the Surface Shape

This paper proposes an unsupervised deep learning method using two neural-field networks with periodic activation functions to model water surface height and reconstruct undistorted underwater images from aerial views, outperforming existing approaches on both simulated and real data.

The Gist

Imagine you are standing on a boat, looking down into a clear, shallow pool. You see a colorful fish or a patterned tile on the bottom. But because the water surface is rippling and waving, the fish looks wobbly, stretched, and broken. It's like looking at a funhouse mirror that's constantly moving.

This paper introduces a clever new way to "fix" that view, not by taking a perfect photo, but by using a computer to mathematically figure out what the water is doing and then "un-distorting" the image.

Here is the breakdown of how they did it, using some everyday analogies:

The Problem: The Wobbly Window

When you look through water, light bends (refracts) as it moves from air to water. If the water surface is flat, the image is just shifted. But if the water is wavy, the surface acts like a shifting lens, warping the image underneath.

The authors wanted to look at underwater scenes from the air (like from a drone) and see them clearly, despite the waves. The problem is that real ocean waves are messy and unpredictable. You can't just take a "before" and "after" photo to teach a computer how to fix it, because you don't have the "after" photo (the clear view) to begin with.

The Solution: The "Two-Brain" System

The researchers built a computer program with two "brains" (neural networks) that work together. Think of it like a detective trying to solve a crime by looking at a blurry security video.

1. Brain A (The Wave Mapper): This brain tries to guess the shape of the water surface at every single moment. It asks, "Is there a bump here? A dip there?" It creates a 3D map of the waves.
2. Brain B (The Image Painter): This brain tries to guess what the underwater scene actually looks like if the water were perfectly still. It asks, "What color is the fish? What shape is the tile?"

How They Train: The "Unsupervised" Magic

Usually, to train a computer to fix images, you need thousands of pairs of "bad" and "good" photos. But here, they didn't have the "good" photos. So, they used a trick called Unsupervised Learning.

Imagine you have a jigsaw puzzle, but the pieces are all warped and you don't know what the final picture looks like.

• The computer makes a guess about the final picture (Brain B).
• It also makes a guess about the shape of the water (Brain A).
• It then takes its guess of the final picture and "warps" it using its guess of the water shape to see if it matches the blurry photo it actually took.
• If the warped guess looks like the blurry photo, the computer is on the right track. If not, it tweaks its guesses and tries again.

By doing this over and over, the computer eventually figures out the perfect combination of "what the water looked like" and "what the underwater scene looked like" that explains the blurry video.

The Secret Weapon: SIREN

The paper mentions a specific tool called SIREN. You can think of this as a special type of mathematical "muscle" that is really good at drawing smooth, wavy lines.

• Regular computer networks are sometimes bad at drawing smooth curves; they might make them look jagged or blocky.
• SIREN is like a master calligrapher. It can draw the smooth, continuous curves of a water wave and the sharp edges of a fish scale simultaneously. This is crucial because the distortion depends on the slope of the wave, and SIREN is great at calculating slopes.

What They Achieved

The results are impressive:

• Clearer Images: They successfully removed the wobbly distortion from videos of underwater scenes, revealing sharp details like numbers on a dice or patterns on a checkerboard that were previously invisible.
• Wave Mapping: As a bonus, their system actually produced a map of the water waves themselves. It's like getting a weather report for the water surface just by looking at the distorted video.
• Better than the Rest: They tested this against other methods, and their "two-brain" system worked better, even with short video clips and real-world messy water.

Why It Matters

This technology is like giving drones "super-vision" for the ocean.

• Scientists can count coral reefs or check for bleaching without needing to dive or send expensive underwater robots.
• Safety teams could spot people drowning in pools or the ocean more easily, even if the water is choppy.
• Fish farmers can monitor their stock without disturbing the water.

In short, the paper teaches a computer to be a "water detective," using the ripples on the surface to reverse-engineer the hidden world below, turning a blurry, wobbly mess into a crystal-clear picture.

Technical Summary

Here is a detailed technical summary of the paper "Looking Into the Water by Unsupervised Learning of the Surface Shape" by Lifschitz et al.

1. Problem Statement

The paper addresses the challenge of underwater scene reconstruction from aerial views. When observing underwater objects from above the water surface, light refraction at the air-water interface causes significant geometric distortions. These distortions are governed by Snell's Law and depend on the dynamic shape (height and gradients) of the water surface.

• Core Difficulty: Real-world water waves are complex superpositions of various wave types with different periods and amplitudes. Consequently, supervised learning methods struggle to generalize because large-scale, real-world ground truth data (paired distorted/clean images with known surface geometry) is unavailable.
• Goal: To reconstruct a sharp, undistorted image of the static underwater scene and simultaneously estimate the water surface height for every frame in a short video sequence, using unsupervised learning.

2. Methodology

The authors propose a novel unsupervised framework that models the scene and the water surface separately using Implicit Neural Representations (INRs), specifically SIREN (Sinusoidal Representation Networks).

A. Physical Model

The method relies on the physical relationship between water surface height and pixel distortion. Under a first-order approximation of Snell's Law, the distortion vector $d(x, t)$ at spatial position $x$ and time $t$ is proportional to the gradient of the water surface height $h(x, t)$:
$$d(x, t) = \left(1 - \frac{1}{n}\right) h_0 \nabla h(x, t)$$
Where:

• $n$ is the relative refractive index between air and water.
• $h_0$ is the average water depth.
• $\nabla h(x, t)$ is the spatial gradient of the surface height.

B. Neural Architecture

The system consists of two SIREN-based neural fields trained jointly on a sequence of distorted frames:

1. Surface Height Model ($H_\theta$):

   • Input: 2D spatial coordinates ($x$) and time ($t$).
   • Output: Water surface height $h(x, t)$.
   • Function: It predicts the height map for every frame. The network's output gradients ($\nabla h$) are explicitly computed to derive the pixel distortion map $d(x, t)$ via the physical equation above.
   • Why SIREN? SIRENs use sinusoidal activation functions, which are highly effective at modeling continuous signals and their derivatives. This is crucial because the distortion depends directly on the derivative of the height signal.

2. Image Model ($I_\phi$):

   • Input: 2D spatial coordinates ($x$).
   • Output: Pixel color values of the underlying undistorted scene.
   • Function: This network learns the static, clean image.

C. Reconstruction and Training

The training process is unsupervised, minimizing the difference between the observed distorted frames and the reconstructed frames:

1. Forward Pass:
   • The Image Model $I_\phi$ predicts the clean image at regular grid positions.
   • The Height Model $H_\theta$ predicts surface heights, which are differentiated to compute distortion offsets $d(x, t)$.
   • The clean image is "warped" using these offsets to simulate the observed distorted image: $I^{obs}_{t} = I_\phi(x + d(x, t))$.
2. Loss Function:
   • The model is trained to minimize the $L_1$ norm between the simulated distorted images and the actual input video frames.
   • Two-Stage Training:
     1. Initialization: Train $H_\theta$ to output zero distortion and $I_\phi$ to predict the average distorted image.
     2. Optimization: Jointly optimize both networks to minimize the reconstruction error of all frames in the sequence.

3. Key Contributions

• Unsupervised Surface-Aware Reconstruction: Unlike previous unsupervised methods that treat distortion as a generic warping field, this method explicitly models the water surface height and its gradients, grounding the restoration in physical laws (Snell's Law).
• SIREN for Spatio-Temporal Modeling: The paper demonstrates that SIRENs are uniquely suited for this task because they naturally handle the relationship between a signal (surface height) and its spatial derivatives (distortion), allowing for efficient modeling of complex wave dynamics without explicit parametric wave equations.
• Dual Output: The method simultaneously recovers the high-fidelity underwater scene and provides a quantitative estimate of the water surface height for every frame.
• Simplified Training: The authors propose a simplified loss function (single term) compared to state-of-the-art baselines (like NDIR), which require multiple complex loss terms for stability.

4. Experimental Results

The method was evaluated on three datasets:

1. Real1 (James Real1): 7 sequences from a water tank with significant motion blur and large frame-to-frame motion.
2. TianSet: Real-world captured data at 125 fps.
3. Synthetic Dataset: Generated using wave equations (Gaussian, Ripple, Ocean waves).

Quantitative Performance:

• Image Restoration: The method outperformed the state-of-the-art unsupervised baseline NDIR [11] and the supervised method Li et al. [12] across most metrics (PSNR, SSIM, LPIPS) on both real and synthetic datasets.
  • Notably, it achieved the best LPIPS (Learned Perceptual Image Patch Similarity) scores, indicating superior perceptual quality and sharpness.
• Surface Estimation: On synthetic data where ground truth height was available, the method achieved an RMSE of 0.115 and Abs Rel of 0.0635, comparable to supervised depth estimation methods.

Qualitative Results:

• The method successfully recovered fine details (e.g., text, grid lines, cartoon features) that were blurred or warped in the input.
• It handled motion blur and large wave fluctuations better than baselines.
• Ablation studies confirmed that modeling surface height and using temporal conditioning (a single network for time-varying height) are critical for performance.

5. Significance and Applications

• Scientific & Operational Utility: The ability to see clearly through water from the air is vital for:
  • Coral Reef Monitoring: Measuring bleaching scales after storms.
  • Fish Farming: Monitoring stock health.
  • Safety: Drowning detection in pools and oceans.
  • Oceanography: Estimating sea surface microlayers and wave dynamics without specialized sensors.
• Robustness: The method works on real-world data where assumptions (like orthographic cameras or perfectly planar scenes) are not strictly met, demonstrating robustness to complex 3D geometries and motion blur.
• Future Directions: The authors suggest using this restoration as a pre-processing step for downstream tasks like feature matching and Monocular SLAM (Simultaneous Localization and Mapping).

In summary, this paper presents a physics-informed, unsupervised deep learning approach that leverages the temporal consistency of video sequences and the properties of SIRENs to solve the difficult problem of seeing through dynamic water surfaces, outperforming existing methods in both image quality and surface estimation.