🤖 AI

ENAF: A Multi-Exit Network with an Adaptive Patch Fusion for Large Image Super Resolution

ENAF is a dynamic multi-exit network for large image super-resolution that improves the quality-complexity trade-off by using a tiny PSNR-estimating network to adaptively fuse image patches and assign them to appropriate early exits based on their texture complexity.

Original authors: Duong M. Nguyen, Tuan Nghia Nguyen, Xuan Truong Nguyen

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Duong M. Nguyen, Tuan Nghia Nguyen, Xuan Truong Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of digital imaging, there is a constant tension between clarity and speed. We live in an era where screens have grown from standard definition to high-definition, and now to 4K and 8K resolutions, demanding that images be sharper and more detailed than ever before. A common task in this landscape is single image super-resolution, a process where a computer takes a small, blurry, low-resolution photo and reconstructs it into a large, crisp version. For years, researchers have built powerful artificial intelligence models to perform this magic, but these models are often heavy and slow. They require immense computing power that grows rapidly as the image size increases, making them difficult to run on everyday devices like smartphones or televisions when dealing with very large pictures. The challenge has been to find a way to make these smart systems faster without sacrificing the quality of the final image, especially when the input is a massive photograph that needs to be processed piece by piece.

To solve this, a team of researchers from Hanoi University of Science and Technology and Seoul National University has developed a new approach called ENAF. Instead of forcing every part of a large image to go through the same complex, energy-hungry processing steps, this system acts like a smart traffic controller. It breaks a large image into many small squares, or patches, and then decides how much effort each patch deserves. The researchers realized that not all parts of a photo are equally difficult to fix. A patch of sky or a smooth wall contains very little detail and is easy to restore, while a patch of a tree with intricate leaves or a face with fine features is much harder. Previous methods tried to guess which patches were hard by looking for edges or lines, but this often led to mistakes, treating simple areas as if they were complex or vice versa. ENAF improves on this by using a tiny, specialized helper network that predicts exactly how good the result will be for each patch if it is processed quickly versus slowly.

The core of the ENAF system is a main network that has multiple "exits" along its path. Imagine a long hallway with several doors leading out; a patch can leave through an early door if the system thinks it has done enough work, or it can travel all the way to the end if it needs more processing. The system uses its tiny helper to estimate the quality score, known as PSNR, that a patch would achieve at each of these exits. If the helper predicts that a patch will look good enough after just a few steps, the system sends it out early, saving a tremendous amount of computing power. If the patch is complex, it is allowed to travel further through the network to ensure high quality. The researchers also added a clever trick for the simplest parts of an image, such as blank walls or clear skies. They found that for these areas, the most complex neural networks sometimes perform worse than a very simple, traditional mathematical smoothing technique. To fix this, ENAF includes a detector that identifies these "blank" areas and immediately sends them to the simple method, bypassing the heavy machinery entirely.

When the researchers tested this new system on standard datasets containing images ranging from 2K to 8K resolution, the results were significant. By using this adaptive method, ENAF was able to reduce the amount of computing work required by up to 55 percent compared to previous state-of-the-art methods, while still producing images of equal or better quality. For example, on very large 8K images, the system could save nearly half the computing power needed by older models. The study showed that this approach works well across different sizes of underlying AI models, from small ones designed for mobile phones to large ones for high-end servers. The team demonstrated that by carefully matching the difficulty of each image patch with the right amount of processing, they could achieve a much better balance between speed and image quality. This means that in the future, devices could render high-definition video or enhance photos in real-time without needing expensive, power-hungry hardware, bringing high-quality image processing to a wider range of everyday applications.

Technical Summary: ENAF - A Multi-Exit Network with an Adaptive Patch Fusion for Large Image Super Resolution

Problem Statement
Single Image Super-Resolution (SISR) algorithms based on Convolutional Neural Networks (CNNs) face significant challenges when processing large images (2K to 8K resolution). The computational cost and memory footprint of these models grow quadratically with input size, making them inefficient for practical applications like smartphones and IP TVs. While recent approaches decompose large images into smaller patches and utilize dynamic networks to assign different computational capacities based on patch difficulty, existing methods suffer from two primary limitations:

Inaccurate Patch Classification: Current methods often rely on handcrafted assessment scores (e.g., edge density) to estimate patch difficulty. As demonstrated in the paper, these heuristics weakly associate texture with the actual computational complexity required for restoration, leading to misclassification (e.g., categorizing distinct low-texture patches identically despite different restoration needs).
Poor Sub-network Selection: Existing dynamic approaches rarely optimize the quality-complexity trade-off effectively across different patches, often assigning patches to overly complex networks or failing to leverage simpler paths for easy patches.

Methodology
The authors propose ENAF (Enhanced Network with Adaptive Fusion), a dynamic network architecture designed to address these shortcomings. ENAF is built upon a standard SISR backbone but incorporates four key components:

Multi-Exit Backbone: The standard SISR backbone is modified to include multiple Early Exits (EEs) at various depths ( $j \in [0, L]$ ). Instead of sharing a single final sub-pixel layer, ENAF places learnable mappings (deconvolution layers) after intermediate encoders, allowing patches to be processed and output at different stages of the network.
PSNR Estimator: Unlike previous classifiers that assign patches to discrete difficulty groups ("easy," "medium," "hard"), ENAF employs a lightweight network to regress the expected Peak Signal-to-Noise Ratio (PSNR) for a patch at each exit. This shifts the problem from classification to regression, providing a more granular and accurate association between data texture and computation cost.
Blank Vector Generator: The authors observe that patches with minimal texture (blank areas) are often better reconstructed by simple bicubic interpolation than by complex neural networks, yet standard estimators may bias against this. A dedicated module detects these "blank" patches using Laplacian edge detection and generates a "blank vector." This vector guides the system to prefer the simplest execution path (bicubic) for low-texture regions, avoiding unnecessary computation.
Computation-Aware Fusion: During inference, ENAF selects the optimal execution path for each patch using a fusion strategy defined by Equation (9):
$j^* = \arg\max_j (u_j + \gamma b_j - \eta c_j)$
Where $u_j$ is the estimated PSNR, $b_j$ is the blank vector contribution, $c_j$ is the normalized computation cost, and $\gamma$ and $\eta$ are hyperparameters controlling the influence of blankness and the performance-computation trade-off, respectively.

Training Strategy
ENAF utilizes a three-phase training process:

Pre-training: The main backbone is trained conventionally using an L1 loss to establish general knowledge.
Multi-Exit Fine-tuning: Early exits are added and trained with a multi-exit loss function. Crucially, the backbone parameters are updated with a very small learning rate ( $10^{-8}$ ) to prevent the distortion of pre-trained weights while optimizing the new exits.
Estimator Training: The PSNR estimator is trained to predict the PSNR of outputs from different exits while the backbone parameters are frozen.

Key Results
Extensive experiments were conducted on standard datasets (Flickr2K, Test2K, Test4K, Test8K) using three backbone architectures: FSRCNN (small), CARN (medium), and SRResNet (large).

Efficiency: ENAF consistently outperforms state-of-the-art dynamic methods like ClassSR and ARM. It achieves comparable or superior restoration quality (PSNR) with significantly reduced computational costs.
FLOPs Reduction: Depending on the backbone and dataset, ENAF reduces FLOPs by 24%-55% (FSRCNN), 43%-52% (CARN), and 31%-52% (SRResNet) compared to the original static networks. Specifically, the savings across the F2K, Test2K, Test4K, and Test8K datasets are 24%-55%, 43%-52%, 43%-48%, and 31%-52%, respectively.
Resolution Scaling: The efficiency gains are most pronounced on larger images (Test8K), where the abundance of low-texture patches allows the blank vector and adaptive fusion to select simpler execution paths more frequently.
Ablation Studies: Removing the blank vector generator resulted in a 5%–15% increase in computational cost to achieve similar PSNR, validating its importance. The PSNR estimator demonstrated lower Mean Absolute Error (MAE) in predicting performance compared to the Look-Up Table (LUT) approach used in ARM.

Significance and Claims
The paper claims that ENAF provides a robust solution for accelerating SISR on large images by moving away from rigid, handcrafted difficulty metrics toward a data-driven, regression-based approach. By integrating a tiny PSNR estimator and a blank vector generator, the method effectively balances the trade-off between image quality and computational complexity. The authors assert that this approach allows existing SISR models to scale efficiently across various hardware constraints without significantly compromising restoration performance, making it suitable for practical deployment in resource-constrained environments. The source code is made available to facilitate further research.

Technical Summary: ENAF - A Multi-Exit Network with an Adaptive Patch Fusion for Large Image Super Resolution

More like this