💻 computer science

HATIR: Heat-Aware Diffusion for Turbulent Infrared Video Super-Resolution

This paper introduces HATIR, a novel heat-aware diffusion framework that jointly models turbulent degradation and resolution loss through a phasor-guided flow estimator and turbulence-aware decoder, alongside the creation of the first FLIR-IVSR dataset to advance infrared video super-resolution.

Original authors: Yang Zou, Xingyue Zhu, Kaiqi Han, Jun Ma, Xingyuan Li, Zhiying Jiang, Jinyuan Liu

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Zou, Xingyue Zhu, Kaiqi Han, Jun Ma, Xingyuan Li, Zhiying Jiang, Jinyuan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Heat Shimmer" Effect

Imagine you are looking at a hot road on a summer day. The air above the asphalt wiggles and ripples, making the cars and trees in the distance look like they are melting or dancing. This is called atmospheric turbulence.

Now, imagine trying to take a video of this scene using a thermal camera (which sees heat instead of light). It's even harder because thermal cameras already struggle with blurry edges and low contrast. When you add the "heat shimmer" of turbulence on top of that, the video becomes a confusing mess of wobbly shapes and drifting colors.

The Challenge:
Existing video tools try to fix this in two separate steps:

First, they try to stop the wiggling (remove turbulence).
Then, they try to make the blurry image sharp (super-resolution).

The paper argues this is like trying to fix a wobbly table by first gluing the legs, then sanding the top. If the gluing isn't perfect, the sanding makes the table worse. The errors pile up, and the final result is still blurry or distorted.

The Solution: HATIR (The "Heat-Smart" AI)

The authors created a new system called HATIR. Think of HATIR as a single, super-smart chef who doesn't just chop vegetables and then cook them separately. Instead, they understand exactly how the heat affects the food while they are cooking, adjusting the recipe in real-time to get a perfect dish.

HATIR uses a technology called Diffusion (which is like slowly turning a blurry sketch into a clear painting) but adds a special "heat sense" to it.

Here are the three main "secret ingredients" HATIR uses:

1. The "Phasor-Guided Flow" (The Heat Detective)

Standard video tools try to track movement by looking for edges and textures (like the outline of a car). But in thermal video, edges are often faint, and the "heat shimmer" makes things look like they are moving when they aren't.

The Analogy: Imagine trying to follow a dancer in a foggy room. You can't see their clothes clearly, but you can see the heat coming off their body.
How it works: HATIR looks at the heat patterns instead of just the shapes. It knows that hot objects (like a car engine or a person) stay hot and consistent over time, even if the air around them is wiggling. It uses this "heat consistency" to figure out where things actually are, ignoring the fake movement caused by the shimmering air.

2. The "Turbulence-Aware Decoder" (The Smart Editor)

Once the AI knows where things are, it has to rebuild the image. But some parts of the video are too distorted to trust.

The Analogy: Imagine a group of editors trying to assemble a puzzle. Some pieces are from the real picture, but others are warped by the heat. A normal editor might try to force the warped pieces into place, ruining the picture.
How it works: HATIR has a special "gatekeeper." It looks at the video and says, "This part is too wobbly to trust; let's ignore the movement there," while saying, "This edge is stable; let's sharpen it." It selectively fixes the parts that can be saved and stops trying to fix the parts that are too broken, preventing the "errors" from spreading.

3. The "Heat-Aware Guidance" (The Compass)

During the process of turning the blurry video into a sharp one, the AI needs a compass to keep it on track.

The Analogy: If you are walking through a foggy forest, you might wander off the path. HATIR gives the AI a compass that points toward "thermal stability."
How it works: It constantly checks: "Does this new sharp image still look like a stable heat source?" If the AI starts creating weird, unstable shapes that don't make sense physically, the compass nudges it back to a realistic, heat-consistent image.

The New Playground: FLIR-IVSR

To prove their method works, the authors couldn't just use old videos because no one had ever made a dataset for this specific problem.

What they did: They built the FLIR-IVSR dataset.
The Analogy: It's like creating the first-ever "Olympic training ground" specifically for athletes who have to run while wearing heavy, blurry goggles.
The Details: They filmed 640 different scenes using a high-end thermal camera. Some scenes had moving cameras, some had moving objects, and some were still. They captured both the "bad" (low-quality, wobbly) video and the "good" (high-quality) video to train their AI.

The Results

When they tested HATIR against other methods:

Two-step methods (fix turbulence, then fix resolution) produced videos that were still wobbly or had "ghost" artifacts.
Standard AI methods (designed for normal cameras) failed because they got confused by the lack of texture in thermal images.
HATIR produced the clearest, most stable videos, successfully removing the "heat shimmer" while keeping the details sharp.

Summary

HATIR is a new way to clean up thermal videos that are distorted by hot air. Instead of fixing the wobble and the blur separately, it fixes them together by using the physics of heat as a guide. It's like teaching an AI to "see" through the heat shimmer by trusting the heat itself, rather than the blurry shapes.

1. Problem Statement

Infrared (IR) video is critical for applications like autonomous driving and surveillance, but it suffers from severe degradation in open atmospheric environments due to atmospheric turbulence. This turbulence, caused by thermal gradients between hot ground surfaces and cooler upper atmospheres, creates turbulent eddies that distort the refractive index. This results in:

Geometric distortions: Random warping of objects.
Thermal blur and noise: Loss of edge definition and texture.
Grayscale drift: Fluctuations in intensity over time.

Current Limitations:
Existing Video Super-Resolution (VSR) methods fail in this domain due to two main issues:

Modality Gap: Standard VSR models are trained on RGB data and assume strong textures and clear boundaries, which are often absent in low-contrast, thermally dominated IR images.
Turbulence Ignorance & Error Propagation: Existing solutions often treat turbulence mitigation (TM) and super-resolution (VSR) as separate, cascaded steps. This decoupled approach leads to error accumulation, where the TM step fails to recover structural details, and the subsequent VSR step amplifies these errors.

2. Methodology: HATIR Framework

The authors propose HATIR (Heat-Aware Diffusion for Turbulent InfraRed Video Super-Resolution), a unified generative framework that models the inverse process of both turbulence degradation and resolution loss simultaneously.

Core Architecture

HATIR operates within a latent diffusion framework. It takes a low-resolution (LR) turbulent IR video sequence and reconstructs a high-resolution (HR) sequence by iteratively refining latent variables.

Key Components

A. Phasor-Guided Flow Estimator (PhasorFlow)
Standard optical flow fails in IR turbulence due to weak textures. PhasorFlow introduces a physics-driven approach:

Phasor Mask: Based on the physical principle that heat-emitting regions exhibit consistent temporal dynamics, while turbulence causes high-frequency, spatially varying perturbations. The method computes the Discrete Fourier Transform (DFT) over time to extract the magnitude of the first harmonic. This creates a soft mask ( $M_{phasor}$ ) that highlights thermally stable regions.
Frequency-Weighted Attention: Uses the Phasor Mask to modulate attention weights during flow estimation. This suppresses unstable turbulent regions and preserves heat-sensitive structural information, enabling robust motion guidance for the diffusion process.
Heat-Aware Guidance: A physics-informed guidance term is injected into the diffusion sampling trajectory. It minimizes symmetric warping errors between bidirectional flows, weighted by a joint mask (combining occlusion and Phasor masks), steering the denoising process toward temporally coherent and thermally stable representations.

B. Turbulence-Aware Decoder (TAD)
To reconstruct the final HR frames, TAD addresses the unreliability of temporal cues in distorted regions:

Turbulence Mask Gating (TMG): Generates a gating mask based on bidirectional warping errors. It adaptively filters out turbulence-corrupted regions, ensuring that cross-frame modeling only occurs in structurally stable areas.
IR Structure-Aware Attention (IR-SAA): Uses gradient magnitude to create an attention map that selectively enforces consistency in high-frequency regions (edges, contours) while avoiding redundant alignment in low-saliency areas.
Optimization: The decoder is fine-tuned using a composite loss function including:
- Thermal Reconstruction Loss: Emphasizes fidelity in thermally active regions.
- Thermal Edge Loss: Penalizes misalignment in thermal edges.
- Frame Difference Loss: Ensures temporal consistency across the sequence.

3. Key Contributions

Unified Framework (HATIR): The first diffusion-based framework to jointly model turbulence mitigation and video super-resolution for infrared imagery, eliminating error propagation inherent in cascaded pipelines.
Physics-Driven Priors: Introduction of PhasorFlow, which leverages thermal consistency (phasor analysis) to guide motion estimation in low-texture, turbulent environments, and a Turbulence-Aware Decoder that selectively suppresses unstable temporal cues.
FLIR-IVSR Dataset: The authors constructed the first dedicated dataset for turbulent IR VSR.
- Source: Captured using a FLIR T1050sc camera (1024 × 768 resolution).
- Scale: 640 diverse scenes with varying camera and object motion.
- Structure: Includes paired LR-HR sequences categorized into camera-static (static/dynamic scenes) and camera-moving subsets.

4. Experimental Results

Datasets & Metrics:

Evaluated on the new FLIR-IVSR dataset and a synthetic turbulence benchmark derived from the M3FD dataset.
Metrics: PSNR, SSIM, LPIPS, DISTS, and VMAF.

Performance:

Quantitative: HATIR achieved state-of-the-art results across all metrics. On the FLIR-IVSR test set, it outperformed the second-best method by a significant margin (e.g., +3.76 dB PSNR in the camera-moving scenario compared to the next best VSR method).
Qualitative: Visual results show that HATIR successfully restores thermal details and corrects geometric distortions while maintaining the intrinsic visual characteristics of IR imagery. In contrast, cascaded methods (TM + VSR) exhibited severe artifacts and error accumulation, while standard VSR methods failed to handle turbulence-induced blur.

Ablation Studies:

PhasorFlow vs. SpyNet: Replacing PhasorFlow with standard optical flow (SpyNet) resulted in a ~4.7 dB drop in PSNR, proving the necessity of thermal-aware flow estimation.
Decoder Components: Removing either the TMG or IR-SAA modules led to significant performance degradation, confirming that both turbulence filtering and structure-aware attention are essential.
Guidance Mask: The joint application of the Phasor and Occlusion masks yielded the best results, validating the complementary nature of these priors.

5. Significance

This work represents a significant leap in infrared computer vision by addressing the specific physical challenges of atmospheric turbulence. By unifying alignment and restoration within a diffusion framework guided by thermal physics, HATIR enables high-fidelity IR video recovery in scenarios previously considered too degraded for analysis. The release of the FLIR-IVSR dataset provides a crucial benchmark to drive future research in robust infrared perception for autonomous systems and surveillance.