💻 computer science

Dual-Branch Remote Sensing Infrared Image Super-Resolution

This paper presents a dual-branch remote sensing infrared image super-resolution solution for the NTIRE 2026 challenge that combines HAT-L and MambaIRv2-L models with test-time local conversion and self-ensemble fusion to achieve superior performance in PSNR, SSIM, and overall score by leveraging the complementary strengths of local transformer restoration and global state-space modeling.

Original authors: Xining Ge, Gengjia Chang, Weijun Yuan, Zhan Li, Zhanglu Chen, Boyang Yao, Yihang Chen, Yifan Deng, Shuhong Liu

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xining Ge, Gengjia Chang, Weijun Yuan, Zhan Li, Zhanglu Chen, Boyang Yao, Yihang Chen, Yifan Deng, Shuhong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to look at a blurry, grainy photo taken by a thermal camera from a drone. In these photos, everything looks like a smooth, glowing blob of heat. There are no sharp edges, no fine details like tree leaves or car grilles—just soft gradients of temperature. Your goal is to make this blurry heat-map sharp and clear without inventing fake details or making the image look weirdly distorted.

This paper describes a clever solution to that problem, created by a team of researchers for a competition called NTIRE 2026. Think of their solution not as a single super-smart robot, but as a dynamic duo of specialists working together.

The Problem: The "Blurry Heat" Dilemma

In regular photos (like what your phone takes), you have lots of texture and color. But in thermal (infrared) photos, the world looks very smooth.

If you try to sharpen the image too much, you might create "ghost" edges or make the heat look unstable (like a flickering light).
If you don't sharpen it enough, you can't tell a person from a tree.

The researchers realized that one single AI model is bad at doing both things at once. So, they built a system with two different brains.

The Two Specialists

1. The "Detail Detective" (HAT-L)

What it does: This model is like a magnifying glass. It is incredibly good at looking at tiny, local areas and sharpening the edges. It wants to make sure the outline of a car or a person is crisp and clear.
The Catch: Because it focuses so hard on tiny details, it sometimes gets a little jittery or "nervous" about the big picture, potentially creating small, unstable glitches.
The Trick: The team gave this detective a special tool called Test-Time Local Conversion (TLC). Imagine this as telling the detective, "Don't just look at the whole room at once; look at the room in small, manageable tiles to make sure you don't miss a single crack." This helps it stay focused and accurate.

2. The "Steady Architect" (MambaIRv2-L)

What it does: This model is like a calm architect looking at the whole building. It is great at understanding the big picture and keeping the "temperature map" consistent across the whole scene. It ensures that the sky doesn't suddenly look hotter than the ground just because of a glitch.
The Catch: It's so careful about the big picture that it might leave the tiny edges a little bit soft or blurry.
The Trick: The team used a technique called Self-Ensemble. Imagine asking the Architect to look at the image eight different times: flipping it sideways, flipping it upside down, and rotating it. Then, they take the average of all eight views. This cancels out any random mistakes and makes the final result rock-solid and stable.

The Magic: The "Equal Partnership"

Instead of trying to force these two very different models to talk to each other or merge their brains, the researchers simply let them do their jobs separately and then mixed their results 50/50.

The Recipe: Take 50% of the "Detail Detective's" sharp edges + 50% of the "Steady Architect's" smooth stability.
The Result: You get an image that has crisp edges (so you can see what the object is) but also smooth, stable heat (so the image doesn't look glitchy).

Why This Matters

Think of it like cooking a perfect soup.

If you only add salt (the Detail Detective), it tastes sharp but might be too intense.
If you only add water (the Steady Architect), it's smooth but bland.
By mixing them in the right proportion, you get a soup that is both flavorful and perfectly balanced.

The researchers tested this on real thermal images and found that this "teamwork" approach beat using either model alone. It proved that for thermal images, you don't just need one super-model; you need a balanced team that respects both the tiny details and the big picture.

In short: They built a system that combines a "sharp-eyed" AI with a "steady-handed" AI, proving that sometimes, two heads (or two models) are better than one.

1. Problem Statement

The paper addresses the challenge of Remote Sensing Infrared Image Super-Resolution (SR). Unlike visible-light images, infrared (thermal) imagery presents unique difficulties:

Weak Textures: Thermal objects often have smooth interiors with low-frequency gradients.
Sensitivity to Artifacts: Aggressive local sharpening can create unstable "hot spots" or distort scene layout, which is more detrimental than missing fine texture.
Conflicting Requirements: Effective infrared SR requires a balance between recovering sharp target contours (local detail) and preserving radiometric stability and scene-wide layout (global consistency).

The authors argue that single-model architectures often struggle to optimize both local and global modeling simultaneously, necessitating a complementary approach.

2. Methodology

The proposed solution is a Dual-Branch Framework designed for the NTIRE 2026 Infrared Image Super-Resolution Challenge. The system avoids feature-level interaction or learned fusion heads, instead relying on independent branches with specific test-time refinements and deterministic image-space fusion.

A. Dual-Branch Architecture

The system utilizes two distinct, independently optimized backbones:

HAT-L Branch (Local Specialist):
- Model: Based on the HAT (Hybrid Attention Transformer) Large configuration with a window size of 32.
- Role: Specializes in local contour modeling. It excels at recovering fine boundaries and small structures, which is critical for identifying thermal targets defined by weak edge cues.
- Refinement: Wrapped with Test-Time Local Conversion (TLC) to mitigate the discrepancy between patch-based training and full-image inference, ensuring locally consistent predictions.
MambaIRv2-L Branch (Global Specialist):
- Model: Based on the MambaIRv2 Large configuration (State-Space Model).
- Role: Specializes in long-range context aggregation. It propagates information across large spatial distances to maintain scene-wide structural consistency and suppress unstable local artifacts.
- Refinement: Employs Eight-way Self-Ensemble (using horizontal flip, vertical flip, and transpose transformations) to improve prediction robustness without modifying learned parameters.

B. Fusion Strategy

Mechanism: The outputs of the two refined branches ( $I_{HAT}$ and $I_{Mamba}$ ) are merged via fixed equal-weight image-space fusion.
Formula: $I_{SR} = 0.5 \times I_{HAT} + 0.5 \times I_{Mamba}$ .
Rationale: This deterministic approach avoids the complexity and potential instability of learned fusion modules, ensuring the final output inherits the sharpness of HAT and the global stability of MambaIRv2.

3. Key Contributions

Dual-Branch Framework: A novel architecture that explicitly leverages the complementary inductive biases of Transformer-based models (strong local detail) and State-Space Models (strong global propagation) for infrared SR.
Branch-Specific Test-Time Refinements: The paper demonstrates that applying different inference strategies (TLC for Transformers, Self-Ensemble for State-Space models) to specific branches maximizes their individual strengths.
Comprehensive Evaluation: The system is evaluated on both the official NTIRE 2026 hidden test set and a reproducible synthetic benchmark derived from the Caltech Aerial RGB-Thermal dataset.

4. Experimental Results

The system was evaluated using PSNR, SSIM, and a combined Score metric ( $Score = PSNR + 20 \times SSIM$ ).

Official Challenge Performance: The system achieved a final score of 54.23 on the NTIRE 2026 hidden test set.
Caltech Synthetic Benchmark (Scale $\times$ 4):
- Bicubic Baseline: Score 55.6982.
- MambaIRv2-L (Single): Score 56.4690.
- HAT-L (Single): Score 56.5070.
- Fused Output: Score 56.5128 (Highest).
Analysis:
- The fused output consistently outperformed both single branches across all metrics.
- Qualitative Results: Visual comparisons showed that the fused image retained the sharp boundaries of the HAT branch while maintaining the smooth, globally coherent radiometric distribution of the MambaIRv2 branch. This confirms that the branches capture non-redundant, complementary features.

5. Significance and Conclusion

The paper establishes that explicit complementarity between locally aggressive restoration (Transformers) and globally stable modeling (State-Space Models) is a highly effective strategy for infrared super-resolution.

Design Philosophy: It challenges the trend of seeking a single "best" backbone, suggesting instead that heterogeneous ensembles with tailored inference pipelines yield superior results for specific modalities like thermal imaging.
Practical Impact: The approach provides a robust, interpretable, and deployable solution for remote sensing applications where preserving target contours and scene layout is more critical than synthesizing high-frequency textures.
Future Direction: The success of this dual-branch, test-time refinement strategy suggests a promising direction for other low-level vision tasks involving sparse or unstable data modalities.