💻 computer science

High-speed Imaging through Turbulence with Event-based Light Fields

This paper presents the first system capable of imaging fast-moving, non-rigid objects through strong atmospheric turbulence at high frame rates by leveraging event-based light field cameras and machine learning to distinguish between correlated scene motion and uncorrelated turbulence-induced events.

Original authors: Yu-Hsiang Huang, Levi Burner, Sachin Shah, Ziyuan Qu, Adithya Pediredla, Christopher A. Metzler

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu-Hsiang Huang, Levi Burner, Sachin Shah, Ziyuan Qu, Adithya Pediredla, Christopher A. Metzler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a photo of a hummingbird flying at lightning speed, but you are doing it through a heat haze rising off a hot asphalt road. The air is shimmering, making the bird look like it's wobbling, stretching, and blurring into a ghostly mess. This is the problem of atmospheric turbulence.

For decades, cameras have struggled with this. Traditional cameras take a "snapshot" by holding their shutter open for a tiny fraction of a second. If the air is moving fast or the object is moving fast, that snapshot gets smeared. It's like trying to take a photo of a race car by keeping your eyes closed for a split second and then opening them; you just get a blur.

This paper introduces a clever new solution that combines two high-tech ideas: Event Cameras and Light Fields. Here is how it works, explained simply.

1. The Event Camera: The "Hyper-Aware" Eye

Standard cameras are like people who blink slowly. They capture a whole scene at once, even if things are moving too fast to see clearly.

Event cameras are different. They don't take pictures; they are more like a swarm of hyper-alert ants. They only "scream" (send a signal) when they see a change in brightness.

If a pixel is dark and stays dark? Silence.
If a pixel is bright and stays bright? Silence.
If something moves and the brightness changes? BAM! A signal is sent instantly, within microseconds.

This allows them to see things moving incredibly fast (like a Nerf dart traveling at 16,000 pixels per second!) without any motion blur. However, there's a catch: They can't tell the difference between the bird moving and the air shimmering. To the camera, both look like a chaotic mess of changing lights. It's like trying to hear a friend's voice in a room where the wind is howling; you hear noise, but you can't separate the voice from the wind.

2. The Light Field: The "Crowd of Witnesses"

This is where the second part of the magic comes in. The researchers built a special camera that doesn't just have one lens; it has a kaleidoscope inside it.

Think of it like this: Instead of one person trying to see the bird through the heat haze, you have nine people standing in a circle around the bird, all looking at it from slightly different angles at the exact same time.

The Bird (The Scene): The bird looks the same to all nine people. If it moves left, they all see it move left.
The Heat Haze (The Turbulence): The air is churning differently for each person. The person on the left sees the bird wobble one way; the person on the right sees it wobble a completely different way.

3. The Solution: "The Consensus Algorithm"

The researchers used a smart computer program (AI) to look at the data from all nine "witnesses" (the nine views from the kaleidoscope).

Here is the logic the AI uses:

"If all nine people see the bird move left at the same time, that's real motion."
"If Person A sees the bird wobble left, but Person B sees it wobble right, and Person C sees it stay still... that's not the bird moving. That's just the air shimmering."

By comparing all the views, the AI can mathematically "subtract" the random wobbling caused by the air and keep only the consistent movement of the object. It's like a detective asking nine witnesses what they saw; if nine people agree on the suspect's path, but their descriptions of the background are all different, the detective knows the background was just a distraction.

The Results: What Did They Achieve?

Using this "Event-based Light Field" system, the team was able to:

See through strong heat haze: They filmed objects moving through turbulence that would normally make a video look like a melted wax painting.
Catch super-fast motion: They captured a bouncing ball and a spinning stripe at speeds up to 600 frames per second (and even simulated 12,000 fps!), with the images staying sharp and clear.
No more blur: Because the event camera doesn't use a "shutter," fast-moving objects don't get smeared.

Why This Matters

This isn't just about taking cool videos of Nerf darts. This technology could revolutionize:

Surveillance: Seeing clearly through heat waves or smoke.
Astronomy: Seeing stars clearly through the Earth's turbulent atmosphere without needing massive, expensive mirrors.
Autonomous Vehicles: Helping self-driving cars see clearly through heat rising off the road on a hot day.

In short, they built a camera that acts like a team of super-attentive witnesses, using their collective agreement to filter out the "noise" of the atmosphere and reveal the true, sharp, high-speed world underneath.

1. Problem Statement

Imaging through atmospheric turbulence is a critical challenge for applications in astronomy, remote sensing, and surveillance. Turbulence introduces geometric distortion (wander), blur, and intensity fluctuations (scintillation), severely degrading image fidelity.

Existing solutions face two fundamental limitations:

Hardware/Software Constraints: Traditional methods (Adaptive Optics, Lucky Imaging, Deep Learning) rely on conventional frame-based sensors. These sensors integrate light over a fixed exposure time, which irrecoverably blends scene dynamics with turbulence fluctuations. This results in motion blur for fast-moving objects and limits the ability to distinguish between scene motion and turbulence.
The Event Camera Ambiguity: Event cameras (neuromorphic sensors) offer microsecond temporal resolution, making them ideal for high-speed imaging. However, when used alone, they cannot distinguish between events caused by scene motion and events caused by turbulence. Both generate asynchronous brightness changes, rendering existing event-to-video reconstruction algorithms ineffective in turbulent environments because they assume events arise solely from scene dynamics.

2. Methodology

The authors propose a novel system combining Event Cameras with Light Field Optics to resolve the ambiguity between scene motion and turbulence.

A. Hardware Architecture: Event-based Light Field

Design: The system utilizes a kaleidoscope-based light field design. A main lens focuses the scene onto an intermediate image plane, where a kaleidoscope (using planar mirrors) folds the optical path to create a $3 \times 3$ array of sub-aperture views on a single sensor.
Sensor: A Prophesee EVK4 event camera captures the light field.
Principle: Each of the $N$ $N$ sub-aperture views observes the same scene through a distinct, independent realization of atmospheric turbulence.
- Scene Dynamics: Events caused by the moving object are strongly correlated across all views (the object moves consistently).
- Turbulence: Events caused by atmospheric fluctuations are weakly correlated across views (each sub-aperture sees a different turbulence pattern).

B. Imaging Model

The intensity at the sensor is modeled as an incoherent imaging integral under an anisoplanatic Point Spread Function (PSF).

Single View: Recovering the scene $S$ from a single turbulent observation is an ill-posed blind inverse problem because scene motion ( $S$ ) and turbulence ( $h$ ) are multiplicatively coupled.
Multi-View: By capturing $N$ views simultaneously, the system provides cross-view constraints. The scene $S$ is assumed constant across views (negligible parallax at long range), while the turbulence PSFs $\{h^{(i)}\}$ vary. This structure allows the separation of the common scene signal from the view-specific turbulence noise.

C. Reconstruction Algorithm

Network Architecture: The authors adapt E2VID, a recurrent encoder-decoder network (U-Net with ConvLSTM), for multi-view processing.
Input Processing:
1. Each sub-aperture event stream is converted into a spatio-temporal voxel grid.
2. The $N$ voxel grids are stacked along the channel dimension to form a joint input.
Training Strategy:
- Data: The network is trained exclusively on a simulated dataset generated by combining a physics-based turbulence simulator (P2S) with an event simulator (V2E). The dataset includes REDS video clips degraded by varying turbulence strengths.
- Loss Function: The network minimizes the LPIPS (Learned Perceptual Image Patch Similarity) loss, which is superior to MSE for preserving perceptual quality and avoiding blur.
- Augmentation: Includes geometric transforms, noise injection, and "pause" augmentation to train the recurrent state to maintain output stability during event gaps.

3. Key Contributions

First Event-Only High-Speed Turbulence Mitigation: The paper introduces the first system capable of imaging fast-moving, non-rigid objects through strong turbulence at high frame rates (up to 12,000 fps) without relying on conventional frame-based sensors.
Resolution of the Motion-Turbulence Ambiguity: It demonstrates that light field optics provide sufficient cross-view constraints to disentangle scene motion from turbulence-induced events, a problem unsolvable with single-view event cameras.
Superiority over Frame-Based Methods: The method overcomes the motion blur inherent to conventional cameras and frame-based turbulence mitigation techniques (like MambaTM), which fail when scene dynamics are faster than the exposure time.
Sim-to-Real Generalization: The system is trained entirely on synthetic data but successfully generalizes to real-world tabletop experiments involving candles and space heaters as turbulence sources.

4. Experimental Results

The authors conducted both simulated and real-world tabletop experiments.

Quantitative Performance:
- On simulated data, the Light Field (LF) model outperformed a Single-View (SV) baseline (matched for total pixel count) across all metrics: PSNR (15.40 vs 14.84), SSIM (0.505 vs 0.501), and LPIPS (0.4332 vs 0.5185).
- Post-processing single-view event reconstructions with state-of-the-art frame-based turbulence mitigators (e.g., MambaTM) failed to recover quality, proving the necessity of an end-to-end event-light-field approach.
Qualitative Performance:
- Edge Preservation: The LF model reconstructed sharp, straight edges, whereas the SV baseline produced curved, distorted edges due to residual turbulence.
- Temporal Consistency: $x-t$ and $r-t$ plots showed that the LF method significantly reduced "wobble" (temporal jitter) compared to the SV baseline.
- High-Speed Dynamics:
  - Spinning Stripe: Recovered a straight trajectory in $\theta-t$ plots, whereas SV showed a wavy trajectory.
  - Bouncing Ball: Preserved surface texture and geometric shape, which were lost in SV reconstructions.
  - Water Balloon Burst: Successfully captured complex fluid dynamics without confusing them with turbulence.
  - Nerf Dart: Successfully reconstructed a dart traveling at 16,000 pixels/second at 12,000 fps, showing a clear straight-line trajectory.

5. Significance and Limitations

Significance:
This work represents a paradigm shift in atmospheric imaging. By leveraging the high temporal resolution of event cameras and the spatial redundancy of light fields, it achieves high-speed, turbulence-free imaging without the need for expensive hardware like Adaptive Optics (AO) or guide stars. It enables the observation of ultra-fast phenomena (e.g., ballistic projectiles, fluid dynamics) through the atmosphere, which was previously impossible with standard sensors.

Limitations:

Spatial Resolution: Event cameras have lower spatial resolution (~1 MP max) compared to conventional sensors. The light field design further reduces per-view resolution, trading spatial detail for view diversity.
Parallax Assumption: The method assumes the scene is at optical infinity (negligible parallax). It is not suitable for close-range imaging (e.g., biomedical) where parallax between views would be significant.
Latency: The current reconstruction algorithm introduces a processing latency of ~3.4 ms at 176 $\times$ 176 resolution, though this is negligible for many high-speed applications.

In conclusion, the paper successfully demonstrates that Event-based Light Fields can break the frame-rate barrier of conventional cameras and solve the fundamental ambiguity of distinguishing scene motion from atmospheric turbulence, enabling high-fidelity, high-speed imaging through the atmosphere.