🤖 machine learning

Restoring Without Forgetting: Continual Learning Across Image Degradations

This paper introduces Restoring without Forgetting (RwF), a continual learning framework that employs lightweight adapters and unsupervised routing to enable image restoration models to sequentially learn new degradation types without catastrophic forgetting or access to historical data, significantly outperforming standard fine-tuning approaches.

Original authors: Alif Ashrafee, Bartosz Krawczyk

Published 2026-08-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alif Ashrafee, Bartosz Krawczyk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a camera sensor that has spent its entire life in a sunny, clear desert. It has learned perfectly how to see the world under that specific light. Now, picture that same camera being moved to a rainy city, then a foggy mountain, and finally a dark cave. In the real world, sensors face these changing conditions one after another, often without access to the old data from previous locations due to privacy rules or storage limits. For a computer to learn how to fix images from a new environment, it usually has to be retrained from scratch. If you try to teach it the new conditions without wiping its memory of the old ones, it often forgets how to handle the first environment entirely. This is a major hurdle for building intelligent systems that can adapt over time. The field of image restoration, which aims to clean up blurry, noisy, or dark photos, has largely relied on specialized tools for each specific problem, or on massive "all-in-one" systems that require access to every type of problem at the same time to learn. Neither approach works well when a system must learn sequentially, encountering new challenges one by one while keeping its old skills intact.

Researchers at the Rochester Institute of Technology have developed a new method called "Restoring Without Forgetting" to solve this exact problem. Instead of training a massive new network for every new type of image damage, or trying to cram all the learning into one giant model that eventually gets confused, they built a system that learns to adapt without losing its past. The core of their approach is a pre-trained "backbone" network that is already very good at seeing clean images. This backbone is frozen, meaning its internal settings are locked and never changed. When the system encounters a new type of degradation, such as rain or fog, it does not retrain the whole brain. Instead, it attaches a tiny, lightweight add-on module specifically designed to handle that new condition. This add-on learns only the specific patterns of the new problem, leaving the original knowledge of the backbone and the previous add-ons completely untouched. Because each new skill gets its own dedicated, isolated space, the system never overwrites what it learned before.

To make this work in the real world, where the computer doesn't know which type of damage it is looking at, the researchers added a smart routing mechanism. When a new, damaged image arrives, the system briefly scans it to create a simple signature based on the visual clues. It then compares this signature against a small library of stored examples for each known degradation type. Based on which example it matches most closely, the system automatically selects the correct add-on to fix the image. This happens without any human telling the system what the problem is. The researchers tested this on a sequence of five different image problems: noise, motion blur, rain, haze, and low-light conditions. They found that a standard method of simply retraining the network on new data caused the system to forget previous skills so badly that its performance on earlier tasks dropped by more than 15 points on a standard quality scale. In contrast, their new method maintained high performance across all five tasks, effectively eliminating the forgetting problem entirely.

The study also demonstrated that this approach works on real-world images, not just synthetic ones created in a lab. The team tested their system on eleven different real-world datasets containing thousands of images with natural blurs, rain, and other imperfections. The system correctly identified which repair tool to use for nearly 90 percent of the images, and the final picture quality was almost as good as if a human had told the system exactly which problem it was facing. This suggests that the system can generalize its learning to new, unseen content as long as the type of damage is similar to what it has seen before. The researchers argue that this is a significant step forward because it establishes a practical way for image restoration systems to grow and adapt over time without needing to store massive amounts of old data or retrain from scratch. By isolating the learning of new skills from the core knowledge of the system, they have created a framework that allows artificial intelligence to accumulate capabilities sequentially, much like a human learning new skills without forgetting the old ones.

=== SUMMARY ===

Technical Summary: Restoring Without Forgetting (RwF)

Problem Formulation

The paper addresses the challenge of multi-degradation image restoration within a continual learning setting, specifically Domain-Incremental Learning (DIL). While recent "all-in-one" architectures can jointly handle multiple degradations, they assume simultaneous access to all target degradation data during training—a "closed-world" assumption that fails in real-world deployment. In practice, field-deployed systems encounter degradations (e.g., noise, blur, rain, haze, low-light) sequentially over time, often without access to historical training data due to storage, privacy, or licensing constraints.

Existing approaches face a dilemma:

Sequential Fine-Tuning: Retraining on new data overwrites representations for previous degradations, leading to catastrophic forgetting.
Retraining from Scratch: Training a new network for every new degradation is computationally expensive and unscalable.
All-In-One Joint Training: Requires reassembling all past and present data, which is infeasible when prior data is unavailable.

Furthermore, existing pipelines implicitly assume an "oracle" that knows which degradation type a test image belongs to, an assumption that breaks down in open-world scenarios where domain labels are unavailable at inference time.

Methodology: The RwF Framework

The authors propose Restoring without Forgetting (RwF), a parameter-efficient framework designed to learn degradations sequentially without retaining past data and without knowing the degradation label at test time.

1. Content-Controlled Benchmark Construction

To isolate degradation learning from dataset variation, the authors construct a benchmark where all five degradation domains (Noise, Motion Blur, Rain, Haze, Low-light) are synthesized from a single shared clean image corpus (DIV2K). This ensures that domain differences arise solely from the degradation operators ( $T_t$ ) rather than variations in scene content ( $p(c)$ ), allowing the routing mechanism to learn degradation-specific signatures rather than dataset-specific priors.

2. Frozen Backbone with Degradation-Isolated Adapters

RwF utilizes a pretrained restoration backbone (e.g., Restormer or NAFNet) initialized with Gaussian denoising weights and frozen throughout the learning process. This backbone serves as a shared, fixed substrate providing a strong natural-image prior.

Low-Rank Adapters: For each new degradation domain $t$ , a lightweight, network-spanning path of low-rank adapters is attached to the backbone. These adapters apply bottleneck residual corrections to features at every block.
Parameter Isolation: At stage $t$ , only the new adapter path $\Phi_t$ is trained; the backbone and all previous paths $\Phi_{<t}$ remain frozen. This strict parameter isolation ensures that gradients for new domains do not affect previous ones, resulting in zero catastrophic forgetting by construction.
Efficiency: The overhead is minimal (approx. 3–4% of backbone parameters per domain), scaling linearly with the number of domains rather than duplicating the backbone.

3. Unsupervised Prototype-Based Routing

Since domain labels are unavailable at test time, RwF employs a prototype-matching mechanism to route inputs to the correct adapter path without auxiliary classifiers.

Feature Readout: Contrary to classification tasks where deep features are used, the authors find that early encoder features (output of the first encoder stage) are most discriminative for degradations. As the restoration backbone progresses, it suppresses degradation signals to reconstruct clean images, making deep features less informative for distinguishing the type of corruption.
Embedding Aggregation: The spatial feature map is aggregated by computing both the mean and standard deviation across channels and spatial dimensions. These moments are $\ell_2$ -normalized independently and concatenated to form a domain embedding.
Routing: A prototype for each domain is formed by averaging embeddings of a subset of training samples. At inference, an unknown input is embedded and routed to the path with the highest cosine similarity to its prototype.

Key Contributions

Continual Learning Formulation: The paper frames multi-degradation restoration as a domain-incremental learning problem where degradations arrive sequentially, prior data is unavailable, and domain identity is unknown at test time.
Parameter-Efficient Framework: RwF introduces a backbone-frozen architecture that eliminates forgetting via degradation-specific low-rank adapters. It routes inputs unsupervised via a lightweight prototype-matching mechanism, avoiding the cost of dedicated per-domain networks.
Content-Controlled Benchmark & Sim-to-Real Evaluation: The authors provide a five-domain benchmark isolating degradation operators from content variation and demonstrate that the learned adapters transfer to eleven canonical real-degradation benchmarks (3,465 images) without retraining.

Experimental Results

The framework was evaluated on Restormer (Transformer-based) and NAFNet (Convolutional-based) backbones across a sequence of five degradations.

Performance vs. Sequential Fine-Tuning: Sequential fine-tuning suffers severe forgetting. RwF improves the final average PSNR over sequential fine-tuning by 15.25 dB (Restormer) and 11.83 dB (NAFNet).
Comparison with All-In-One: RwF achieves performance competitive with (and in the case of Restormer, superior to) joint "all-in-one" training on the union of all five domains, despite never seeing past data during the training of subsequent domains.
Forgetting Metrics: RwF achieves 0.00 dB forgetting ( $F=0$ ), meaning performance on earlier domains is perfectly preserved after learning new ones.
Real-World Transfer: On 11 real-world benchmarks, RwF achieves 89.5% routing accuracy with a residual +0.94 dB PSNR gap compared to an oracle that knows the ground-truth domain.
Ablation Studies:
- Readout Depth: Using early encoder features (Stage 1) yields significantly higher routing accuracy (79.7%) than deeper features or the bottleneck (58.8%), confirming that restoration backbones suppress degradation cues in deeper layers.
- Aggregation: Combining mean and standard deviation with separate normalization outperforms Global Average Pooling (GAP) alone.

Significance and Claims

The paper claims to establish, to the authors' knowledge, the first systematic baseline for continual multi-degradation image restoration. It demonstrates that:

Forgetting can be eliminated in generative restoration tasks through strict parameter isolation, without the need for replay buffers or regularization penalties.
Unsupervised routing is feasible for dense pixel-level tasks by leveraging early-stage feature statistics that are distinct to specific degradation operators.
Synthetic-to-Real transfer is possible even when the training corpus is content-controlled, provided the degradation operators are physically modeled to match real-world characteristics.

The authors acknowledge limitations, noting that routing accuracy degrades for degradations with primarily global signatures (e.g., low-light) and that adapter pathways could be enhanced with frequency-domain projections or kernel-aware re-parameterizations for further gains on real-world distributions exceeding the training synthesis range.