💻 computer science

RealRestorer: Towards Generalizable Real-World Image Restoration with Large-Scale Image Editing Models

The paper presents RealRestorer, a state-of-the-art open-source image restoration model trained on a large-scale dataset of nine real-world degradation types and evaluated via the new RealIR-Bench, which achieves performance comparable to closed-source large editing models while significantly reducing computational costs.

Original authors: Yufeng Yang, Xianfang Zeng, Zhangqi Jiang, Fukun Yin, Jianzhuang Liu, Wei Cheng, jinghong lan, Shiyu Liu, Yuqi Peng, Gang YU, Shifeng Chen

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yufeng Yang, Xianfang Zeng, Zhangqi Jiang, Fukun Yin, Jianzhuang Liu, Wei Cheng, jinghong lan, Shiyu Liu, Yuqi Peng, Gang YU, Shifeng Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a box of old, damaged photographs. Some are blurry because someone shook the camera, some are covered in rain, others are foggy, and some have weird rainbow patterns (moiré) from taking a picture of a screen.

For a long time, computer scientists tried to fix these photos using specialized tools: one tool for rain, another for blur, another for fog. But real life is messy. A photo might be blurry and rainy and dark all at once. The old tools would get confused, or they would fix the rain but accidentally erase the person's face.

Recently, a new type of "super-intelligent" computer model (like a giant, all-knowing artist) appeared. These models are amazing at fixing photos, but they are like closed-source luxury cars: only a few big companies own them, they cost a fortune to run, and no one knows how they work under the hood.

RealRestorer is the project that says: "Let's build our own open-source super-car that runs just as well as the luxury ones, but anyone can drive it."

Here is how they did it, explained with some everyday analogies:

1. The Problem: The "Synthetic" vs. "Real" Gap

Imagine you are learning to drive.

Old Method: You practiced in a video game simulator. The game made rain look like blue lines and fog look like gray fog. You got really good at fixing "game rain," but when you got on a real road with actual wet, slippery asphalt and real fog, you crashed.
The Issue: Most AI photo fixers were trained on "simulated" damage. They didn't know what real-world dirt, weird lighting, or complex scratches actually looked like.

2. The Solution: A New Training Pipeline

The authors built a massive, realistic training ground. Instead of just using a video game simulator, they created a "Digital Photo Lab" that mimics the real world perfectly.

The "Synthetic" Part: They used advanced AI to generate millions of fake damaged photos. But they didn't just make simple blurs; they used complex rules to make the rain look like real rain and the blur look like a shaky hand.
The "Real" Part: They also scoured the internet for actual damaged photos taken by real people. They used a smart AI assistant (a "Quality Control Inspector") to check these photos and pair them with a "perfect" version of the same photo.
The Result: They created a dataset of over 1.6 million photo pairs covering 9 different types of damage (blur, rain, noise, low light, etc.).

3. The Training Strategy: The "Two-Stage Diet"

Training a giant AI model is like training an athlete. You can't just throw them into the Olympics immediately. The authors used a clever two-step diet:

Stage 1: The "General Knowledge" Phase (Transfer Training)
They started with a model that was already good at editing images (like changing a sky or adding a cat). They taught this model how to fix the "simulated" damage first. This gave the model a broad understanding of what "broken" looks like.
- Analogy: This is like learning the rules of driving in a simulator. You learn how the car reacts, but you haven't felt real wind or rain yet.
Stage 2: The "Real World" Phase (Supervised Fine-Tuning)
Then, they switched to the real, messy internet photos. They taught the model to fix actual damage. Crucially, they didn't throw away the simulated data entirely. They mixed a little bit of the "simulator" data with the "real world" data.
- Analogy: This is like taking driving lessons on a real road, but keeping a safety net (the simulator data) so you don't forget the basic rules. If you only drove on real roads, you might get too used to specific potholes and forget how to handle a smooth highway.

4. The New Test: RealIR-Bench

How do you know if a photo fixer is actually good?

Old Tests: They used math formulas that compared the fixed photo to the "perfect" original. But in the real world, you often don't have the perfect original. It's like trying to grade a student's essay without having the answer key.
The New Test (RealIR-Bench): The authors created a test using 464 real, damaged photos where no one has the perfect answer key.
The Grader: Instead of a math formula, they used a super-smart AI (a Vision-Language Model) to act as a human judge. This AI looks at the "Before" and "After" and asks: "Did the rain go away? Did the face look weird? Does it look natural?"
- Analogy: Instead of checking if the student's answer matches the key, you hire a human teacher to read the essay and say, "Yes, this is much better, and they didn't change the main character's name."

5. The Results: The Underdog Wins

When they tested RealRestorer against the big, closed-source "luxury cars" (like Nano Banana Pro and GPT-Image-1.5):

It performed just as well as the expensive, closed models.
It was the best open-source model available.
It could fix 9 different types of damage and even handle things it wasn't explicitly trained on (like snow or old photos) because it learned the concept of "fixing," not just the specific rules.

Summary

RealRestorer is a breakthrough because it proves you don't need a billion-dollar budget to fix real-world photos. By building a smarter training ground (mixing fake and real data) and a better way to test the results (using AI judges instead of math formulas), they created an open-source tool that is ready to clean up the messy photos of the real world.

It's like taking a master chef's secret recipe, figuring out how to cook it with ingredients anyone can buy, and proving that the resulting dish tastes just as good as the one from the fancy restaurant.

1. Problem Statement

Real-world image restoration faces two critical challenges that limit the effectiveness of existing solutions:

Domain Gap and Generalization: Traditional restoration models are often trained on synthetic data with simplified degradation distributions. Consequently, they fail to generalize to the complex, diverse, and co-existing degradations found in real-world scenarios (e.g., combined blur, noise, and compression artifacts).
Closed-Source Limitations: While large-scale image editing models (e.g., Nano Banana Pro, GPT-Image-1.5) have demonstrated superior generalization and consistency in restoration tasks, they are typically closed-source, relying on proprietary data and compute. This limits reproducibility and accessibility for the research community.
Evaluation Deficiencies: Existing benchmarks often rely on synthetic data or reference-based metrics (PSNR/SSIM) that do not accurately reflect perceptual quality or content consistency in non-reference real-world scenarios.

2. Methodology

The authors propose RealRestorer, an open-source image restoration model that leverages the priors of large-scale image editing models. The methodology consists of three core components:

A. Data Construction Pipeline

To bridge the gap between synthetic and real-world data, the authors developed a comprehensive data generation pipeline covering nine degradation types: blur, rain, noise, low-light, moiré patterns, haze, compression artifacts, reflection, and flare.

Synthetic Degradation Data: Instead of simple synthetic noise, they use advanced tools (SAM-2, MiDaS, VLMs) to generate realistic degradations. This includes granular noise modeling, segment-aware perturbations, and "web-style" degradation processes (simulating compression and resizing artifacts).
Real-World Degradation Data: They curated real degraded images from the web and paired them with high-quality clean references generated by high-performance restoration models. Rigorous filtering (using CLIP, watermark detection, and Qwen3-VL) ensures the paired data is consistent and free of artifacts.
Dataset Scale: The final training set comprises approximately 1.65 million image pairs (1.5M synthetic + 87k real-world).

B. Model Architecture and Training Strategy

Base Model: RealRestorer is built upon Step1X-Edit, a large Diffusion Transformer (DiT) model equipped with a Qwen-VL text encoder. The Flux-VAE and text encoder are frozen; only the DiT backbone is fine-tuned.
Two-Stage Training Strategy:
1. Transfer Training Stage: The model is trained on the large-scale synthetic dataset to transfer high-level image editing priors to restoration tasks. This stage establishes a broad foundation for handling diverse degradation types.
2. Supervised Fine-Tuning (SFT) Stage: The model is fine-tuned on real-world degradation data. Crucially, they employ a Progressively-Mixed strategy, retaining a small proportion of synthetic data during this stage to prevent overfitting to specific real-world patterns and maintain cross-task robustness.
Training Details: Training occurs at 1024×1024 resolution using a cosine annealing learning rate schedule.

C. RealIR-Bench and Evaluation

To address the lack of reliable evaluation for real-world restoration, the authors introduced RealIR-Bench:

Dataset: 464 non-reference, naturally degraded images collected from the web, covering the nine target degradation types.
Metrics: Since ground-truth clean images are unavailable, they propose a VLM-based evaluation framework:
- Restoration Score (RS): Uses a VLM (Qwen3-VL) to rate the severity of degradation before and after restoration (0–5 scale). The improvement in score constitutes the RS.
- LPIPS (LPS): Measures perceptual similarity to ensure content consistency.
- Final Score (FS): A weighted combination of RS and LPS ( $FS = 0.2(1-LPS) \times RS$ ) to balance restoration quality and content preservation.

3. Key Contributions

RealRestorer Model: An open-source model that achieves state-of-the-art (SOTA) performance among open-source methods, narrowing the performance gap with leading closed-source systems (e.g., Nano Banana Pro) to within 0.007 points on the benchmark.
High-Quality Data Pipeline: A novel synthesis pipeline that generates diverse, realistic degradation patterns, significantly reducing the synthetic-to-real domain gap.
RealIR-Bench: A new benchmark and evaluation protocol specifically designed for real-world, non-reference image restoration, utilizing VLMs to assess both degradation removal and content consistency.

4. Experimental Results

Benchmark Performance: On RealIR-Bench, RealRestorer ranks first among all open-source models and third overall (behind only Nano Banana Pro and GPT-Image-1.5). It achieves the best performance in Deblurring and Low-light Enhancement and second-best in Moiré Pattern Removal.
Generalization: The model demonstrates strong zero-shot generalization to unseen tasks (e.g., snow removal, old photo restoration) and performs competitively on the FoundIR dataset.
Ablation Studies:
- Training solely on synthetic data leads to limited generalization and failure on rare real-world cases.
- Training solely on real-world data leads to overfitting and loss of structural consistency (e.g., object deformation).
- The Two-Stage + Progressively-Mixed strategy is proven essential for balancing restoration capability with content fidelity.
User Study: In a study with 32 participants, RealRestorer achieved a 21.54% first-ranking rate, trailing only the top closed-source models, confirming the alignment between the proposed metrics and human perception.

5. Significance

This work represents a significant step forward in making high-performance, generalizable image restoration accessible. By demonstrating that open-source models can rival closed-source commercial systems through careful data engineering and training strategies, it democratizes access to advanced restoration tools. Furthermore, the introduction of RealIR-Bench and VLM-based metrics sets a new standard for evaluating image restoration in realistic, non-reference scenarios, moving the field beyond the limitations of synthetic benchmarks and pixel-wise metrics. The release of the model, data pipeline, and benchmark will facilitate future research in robust, real-world computer vision applications.