💻 computer science

ReMATF: Recurrent Motion-Adaptive Multi-scale Turbulence Mitigation for Dynamic Scenes

ReMATF is a lightweight, recurrent framework that mitigates atmospheric turbulence in dynamic scenes by utilizing a multi-scale encoder-decoder and motion-adaptive temporal fusion on just two frames, achieving superior visual quality and temporal consistency with significantly lower computational costs than existing multi-frame transformer-based methods.

Original authors: Zhiming Liu, Zhicheng Zou, Nantheera Anantrasirichai

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiming Liu, Zhicheng Zou, Nantheera Anantrasirichai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a photo of a distant mountain range on a hot day. The air near the ground is shimmering, making the mountains look like they are dancing, rippling, or melting. This is atmospheric turbulence. It's the same effect that makes stars twinkle or makes a road look like it has water on it in the summer.

When you try to record a video of this, the problem gets worse. The image doesn't just look blurry; it flickers, warps, and jumps around, making it hard to see what's actually happening.

This paper introduces a new tool called ReMATF to fix these shaky, wobbly videos. Here is how it works, explained simply:

The Problem with Old Methods

Previous attempts to fix these videos were like trying to clean a dirty window by looking at 50 different photos of it at once.

Too Heavy: These methods required massive computer power and memory, like trying to carry a heavy backpack full of bricks. They were too slow for real-time use.
Too Rigid: They often assumed the scene was still (like a statue). But in the real world, cars move, people walk, and leaves blow. When the scene moves, these old methods get confused and create "ghosts" or weird trails behind moving objects.

The ReMATF Solution: The "Smart Memory" Approach

The authors created ReMATF, which is like a lightweight, smart editor that only needs to look at two frames at a time: the current blurry frame and the one it just fixed.

Here are the three main "tricks" ReMATF uses:

1. The "Two-Step Dance" (Recurrent Framework)

Instead of trying to process a whole video at once, ReMATF works step-by-step.

The Analogy: Imagine you are walking through a foggy forest. You don't need to see the whole forest to know where you are; you just need to remember where you were a second ago and look at where you are now.
How it works: ReMATF takes the current distorted image and the "clean" image it just made from the previous second. It combines them to guess what the clean image should look like. This keeps the memory usage low, like carrying a small notebook instead of a library.

2. The "Motion-Sensitive Switch" (Motion-Adaptive Temporal Fusion)

This is the paper's most important innovation. The system has to decide: Should I trust the new image, or the old one?

The Analogy: Think of a security guard watching a screen.
- If a static object (like a building) is flickering, the guard says, "That's just the air shimmering. I'll trust the memory of what the building looked like a second ago to smooth it out."
- If a moving object (like a car) is flickering, the guard says, "The car is moving! If I use the old picture, I'll get a ghostly trail. I must trust the new picture to keep the car sharp."
How it works: ReMATF has a special module (MATF) that looks at every single pixel. If the pixel is part of a moving object, it trusts the new frame. If it's part of a still background, it trusts the old frame. This stops the "ghosting" effect while keeping the background smooth.

3. The "Thermostat" (Turbulence-Level Conditioning)

Not all days are equally windy. Some days the air is just a little wobbly; other days it's a chaotic mess.

The Analogy: Imagine a thermostat that senses how cold the room is and adjusts the heater automatically.
How it works: ReMATF has a sensor that estimates how "bad" the turbulence is right now. If the air is very turbulent, the system turns up its "correction power" to handle the heavy warping. If the air is calm, it focuses on keeping the fine details sharp. This helps the tool work well in many different conditions without needing to be retrained.

The Results

The authors tested this tool on both computer-generated videos and real footage of shaky, hot-air scenes.

Speed: It is much faster than the heavy, complex methods used before. It can run in real-time on standard hardware.
Quality: It produces clearer images with less flickering and fewer "ghost" trails behind moving objects.
Efficiency: It achieves these high-quality results while using a fraction of the computer memory required by previous methods.

In short, ReMATF is a lightweight, smart video cleaner that remembers the past just enough to smooth out the air, but knows when to let go to keep moving objects sharp.

Technical Summary: ReMATF

Problem Statement

Atmospheric turbulence (AT) severely degrades video quality in long-range imaging systems by introducing geometric warping, blur, and temporal flickering. These distortions arise from spatial and temporal variations in the air's refractive index. While traditional methods (e.g., non-rigid registration, multi-frame fusion) work well for static scenes, they struggle with dynamic scenes where temporal redundancy is insufficient.

Recent deep learning approaches, particularly those based on transformers and 3D architectures, have shown promise but often require multi-frame inputs (e.g., 5 frames) and incur high computational costs and memory usage. This limits their deployment in resource-constrained scenarios. Furthermore, existing data-driven methods often lack generalization to real-world turbulence due to reliance on synthetic datasets and varying distortion severities.

Methodology

The authors propose ReMATF (Recurrent Motion-Adaptive Multi-scale Turbulence Framework), a lightweight recurrent framework designed to restore videos using only two frames at a time (the current distorted frame and the previous restored output) while preserving spatial detail and temporal stability.

Core Architecture

The framework operates in a recurrent manner, processing time steps $t$ by taking the current distorted frame $I_t$ and the previous restored frame $O_{t-1}$ .

Turbulence-Level Conditioning:
- To adapt to varying distortion severities, the model estimates a turbulence level for each frame using a CLIP-based Image Quality Assessment (CLIP-IQA) model.
- This scalar score is transformed into a latent embedding via a lightweight MLP and injected into the backbone network through a cross-attention modulation layer. This allows the model to dynamically reweight feature responses, prioritizing geometric stabilization during severe turbulence and detail refinement during mild turbulence.
The Mitigator (Multi-Stage Module):
- Multi-Scale Feature Extraction: Inputs are downsampled to create a three-scale hierarchy. 3D convolutional blocks extract local spatio-temporal representations (ripples, intensity fluctuations) at each scale.
- Temporal Alignment (TWT Blocks): At each scale, a Transformer-Warp-Transformer (TWT) block is employed.
  - The first transformer enhances embeddings and facilitates correspondence reasoning.
  - A lightweight flow estimation branch predicts optical flow to guide a flow-warp module (using deformable convolutions) for fine-grained geometric compensation.
  - A second transformer refines the aligned features.
- Feature Refinement: Aligned features from all scales are fused and processed by additional TWT blocks to reconstruct high-frequency textures and remove residual distortions, producing an intermediate restoration $\hat{O}_t$ .
Motion-Adaptive Temporal Fusion (MATF):
- To enforce long-term temporal consistency without increasing the memory window, the intermediate output $\hat{O}_t$ is fused with the motion-compensated previous output $O_{t-1}$ .
- A Motion-Adaptive Temporal Fusion (MATF) module performs per-pixel fusion. It generates a motion map using cues such as motion magnitude, photometric differences, and flow consistency.
- Dynamic regions rely more on the current restoration to avoid ghosting and preserve boundaries.
- Static regions rely more on the previous output to ensure temporal coherence and reduce flicker.
- An edge-aware suppression term prevents structural boundaries from being misinterpreted as motion.
- The final output $O_t$ is recursively propagated to the next time step.

Loss Functions

The training objective combines several losses to ensure spatial fidelity and temporal coherence:

Reconstruction Loss: Charbonnier penalty for robust pixel-wise supervision.
Wavelet-Domain Loss: Haar wavelet constraint to preserve high-frequency structures.
Laplacian Structural Loss: Penalizes discrepancies in Laplacian responses to enhance edge sharpness.
Static-Aware Temporal Consistency Loss: Enforces consistency between the current output and the warped previous prediction specifically in static regions identified by the MATF mask.
Multi-Step Flow Consistency Loss: Encourages coherence with multiple historical predictions using estimated flow fields.

Key Contributions

ReMATF Framework: The first method to mitigate AT distortions through coupled short- and long-term recurrent mechanisms, enabling restoration with only two frames at a time.
Short-Term Recurrence: Provides immediate restoration cues to suppress ripple artifacts.
Long-Term Recurrence via MATF: Integrates current and previous outputs via a motion-aware weight map, enabling memory-efficient long-range temporal aggregation where information from earlier frames is implicitly attenuated.
Turbulence-Level Conditioning: Introduces a mechanism to help the network adapt to a wide range of distortion severities without separate models.
Real-World Dataset Construction: Created a paired real-world AT dataset by replicating dynamic content from static scenes to generate pseudo ground truth for supervised training.

Results

Experiments were conducted on synthetic datasets (ATSyn-Dynamic, DeTurb) and real-world datasets (CLEAR, RLR-AT, ATD).

Synthetic Performance: ReMATF achieved state-of-the-art performance across all metrics (PSNR, SSIM, LPIPS) on the ATSyn-Dynamic dataset, outperforming baselines like MambaTM, DATUM, and TMT across weak, medium, and strong turbulence levels.
Real-World Performance: On real dynamic scenes, ReMATF improved PSNR by +2.83 dB, SSIM by +0.05, and reduced LPIPS by 0.048 compared to the previous best recurrent method (RMFAT). It demonstrated superior temporal stability (lowest tLPIPS) and preserved structural details in challenging regions (e.g., vehicle boundaries, distant objects) where other methods produced blur or flicker.
Efficiency: ReMATF is significantly faster and more memory-efficient than multi-frame transformer baselines. It processes frames in approximately 0.008 seconds on an NVIDIA RTX4090 with only 2.60M parameters, making it suitable for resource-constrained scenarios.
Ablation Studies: Removing the MATF module, multi-scale warping, or turbulence-level conditioning resulted in significant performance degradation, confirming the necessity of each component for handling spatially varying distortions and maintaining temporal stability.

Significance

The paper claims that ReMATF addresses the critical trade-off between temporal information utilization and computational efficiency in atmospheric turbulence mitigation. By decoupling short-term and long-term reasoning within a lightweight recurrent framework, ReMATF achieves robust restoration in dynamic scenes without the memory overhead of large multi-frame windows. The introduction of turbulence-level conditioning and motion-adaptive fusion allows the model to generalize across varying distortion severities and motion patterns, offering a practical solution for real-time deployment in scenarios where traditional multi-frame methods are infeasible.