💻 computer science

EvoIR-Agent: Self-Evolving Image Restoration Agentic System via Experience-Driven Learning

EvoIR-Agent addresses the limitations of existing training-based and training-free image restoration agents by introducing a self-evolving system that constructs a hierarchical experience pool to provide coarse-to-fine guidance, thereby achieving a Pareto-optimal balance between performance and efficiency without requiring retraining.

Original authors: Kailin Zhuang, Jiawei Wu, Zhi Jin

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kailin Zhuang, Jiawei Wu, Zhi Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, old photograph that is damaged. It might be blurry, faded, have scratches, or be covered in rain stains. Fixing it is like trying to clean a dirty window: if you wipe it with a cloth while it's still wet with rain, you just smear the dirt. You need to know what to clean first (the rain), what tool to use (a squeegee vs. a cloth), and in what order to do it.

This is the problem EvoIR-Agent solves. It is a smart computer system designed to fix damaged images automatically.

Here is the simple breakdown of how it works, using everyday analogies:

The Problem: The "Guessing Game"

In the past, smart AI systems (called "Agents") tried to fix images by guessing. They would say, "Maybe I should remove the rain first?" or "Let's try this tool."

The Issue: Without experience, the AI had to guess blindly. It would try a tool, fail, undo it, try another, and fail again. This is called "trial-and-error." It takes a long time and wastes energy, like a student trying to solve a math problem by guessing numbers until they get it right.

The Two Old Ways (and why they failed)

The paper says previous methods tried to fix this in two ways, but both had a catch:

The "Hard-Training" Way: You teach the AI a specific set of rules by training it on thousands of images.
- Pros: It's fast.
- Cons: It's rigid. If you give it a new type of damage it hasn't seen before (like a new kind of blur), it gets confused and fails. It's like a chef who only knows how to cook pasta; if you ask for sushi, they can't do it.
The "No-Training" Way: You give the AI a giant notebook of past experiences to look at while it works.
- Pros: It can handle new things because it can look up advice.
- Cons: The notes in the notebook are often too simple or "naive." The AI still spends a lot of time flipping through pages and making mistakes because the advice isn't detailed enough.

The Solution: EvoIR-Agent (The "Self-Evolving Apprentice")

The authors created EvoIR-Agent, which combines the best of both worlds. It doesn't just have a static notebook; it has a living, learning memory system.

Think of it like a master craftsman training a new apprentice:

1. The Three Things It Remembers

Instead of just remembering "Remove Rain," the system remembers three specific things for every image:

Which Tool to Use: Not just "a tool," but the best tool for that specific look of the damage.
The Order of Operations: Knowing that you must remove the rain before you fix the blur.
The Visual Goal: Understanding that sometimes you want the photo to look exactly like the original (Fidelity), and other times you want it to look artistic and sharp (Perception).

2. The "Three-Layer" Library (Hierarchical Experience Pool)

The system organizes its memory like a library with three sections, from broad to specific:

The "Big Picture" Section (Insight Level): General rules written by a smart AI (e.g., "Always remove rain before blur").
The "Category" Section (Coarse Level): Rules for specific types of damage (e.g., "For Motion Blur, use Tool A").
The "Fine Detail" Section (Fine Level): This is the magic. It remembers specific patterns. For example, it knows that "Motion Blur caused by a fast car at night" looks different from "Motion Blur caused by a shaky hand in daylight," and it uses different tools for each.

3. The "Self-Evolving" Mechanism (The Learning Loop)

This is the most important part. The system doesn't just read its library; it writes to it.

Step 1: The AI tries to fix an image.
Step 2: It checks the result. Did it look good? Did it take too many steps?
Step 3: It records this experience. If it found a faster way to fix a specific type of blur, it updates its library.
Step 4: Next time it sees a similar image, it uses that new, better advice immediately.

It's like a video game character who learns from every battle. The first time they fight a dragon, they might take 10 hits. The second time, they remember the dragon's pattern and only take 2 hits. By the tenth time, they win instantly.

The Results

The paper tested this system against other top methods.

Better Quality: The fixed images looked clearer and more accurate.
Faster: Because the system learned the "best path" quickly, it didn't waste time guessing or undoing mistakes. It achieved a perfect balance between high quality and speed.

Summary

EvoIR-Agent is a smart image fixer that stops guessing and starts learning. It builds a detailed, organized memory of what works and what doesn't, constantly updating itself with every new image it fixes. It's the difference between a student who memorizes a textbook once and a student who studies, takes notes, and improves their study guide every single day.

Technical Summary: EvoIR-Agent: Self-Evolving Image Restoration Agentic System via Experience-Driven Learning

Problem Statement

Image restoration (IR) in real-world scenarios often involves degradation coupling, where multiple types of image degradation (e.g., blur, noise, haze) occur simultaneously. While Multimodal Large Language Model (MLLM)-driven Image Restoration Agents (IRAs) have emerged as a promising solution for flexibly selecting tools and determining removal orders, they face a critical bottleneck: zero-shot planning without experience often fails, leading to severe trial-and-error overhead.

Existing approaches to this dilemma fall into two paradigms, each with significant limitations:

Training-based methods: Embed intrinsic experience into model parameters. While efficient at inference, they lack compatibility with new tools or unseen degradation types without retraining.
Training-free methods: Utilize explicit external experience storage for high compatibility. However, they often rely on "naive" experience mechanisms that are simplistic and static, failing to provide the dynamic guidance necessary to avoid excessive trial-and-error.

The core challenge is to design an IRA that retains the compatibility of training-free methods while achieving the inference efficiency of training-based methods, specifically by addressing the underdeveloped nature of current experience mechanisms.

Methodology: EvoIR-Agent

The authors propose EvoIR-Agent, a self-evolving agentic system designed to systematically formulate and utilize experience components. The framework operates through a closed-loop process involving an outer inference workflow and an inner experience evolution mechanism.

1. Systematic Formulation of Experience Components

The paper identifies three critical experience components that current IRAs often neglect or oversimplify:

Tool Selection: The optimal tool depends not just on the broad degradation type, but on the specific degradation pattern.
Degradation Removal Order: In coupled settings, the sequence of removal is highly pattern-dependent.
Visual Quality (Preference): Different tool paths can successfully remove degradation but yield distinct visual outcomes. The system introduces Preference ( $Q$ ) as a configurable component (e.g., Fidelity vs. Perception) to guide these choices.

2. Hierarchical Experience Pool

To manage these components effectively, EvoIR-Agent constructs a hierarchical experience pool that provides coarse-to-fine guidance. This pool instantiates a priority-driven mapping function $p$ across three levels:

Insight Level (Preference-oriented): Uses an LLM to leverage distilled textual experience for high-level planning on removal orders ( $O$ ) based on preference ( $Q$ ) and degradation types ( $D$ ).
Coarse-grained Level (Degradation Type-oriented): A direct key-value mapping (without MLLM involvement) that maps degradation types and preferences to optimal removal orders.
Fine-grained Level (Degradation Pattern-oriented): Addresses the nuance that a single degradation type can cluster into different patterns. This level uses a Pattern Profile ( $P$ )—consisting of a set of degraded images and semantic descriptions—retrieved via a CLIP encoder and refined by an MLLM. It maps specific patterns to optimal tool rankings.

3. Self-Evolving Experience Mechanism

Unlike static experience storage, EvoIR-Agent introduces a mechanism to iteratively update the pool using accumulated records. This process occurs in a batch-wise manner:

Experience Acquisition: The system performs exhaustive exploration of potential tool combinations and removal orders for a batch of degraded images. It evaluates results using Image Quality Assessment (IQA) metrics aligned with the user's preference ( $Q$ ).
Statistical Modeling: Using pairwise comparisons, the system calculates win rates and employs the Bradley-Terry-Davidson (BTD) model to estimate ability parameters ( $\theta$ ) for tools and orders.
Evolution & Refinement:
- Coarse-grained: The BTD parameters are sorted to update the priority function $p$ .
- Fine-grained: A Multi-Agent Debate (MAD) paradigm is employed to partition images into new pattern profiles. This process ensures Semantic Consistency (images in a profile are semantically close) and Ranking Consistency (priority mappings are statistically comparable).
- The system iteratively refines these profiles, stabilizing the priority function $p$ for specific patterns.

Key Contributions

Systematic Formulation: The authors reformulate IRA experience by explicitly introducing Visual Quality (Preference) as a core component and identifying the limitation of current methods in focusing solely on broad degradation types rather than finer degradation patterns.
Self-Evolving Architecture: They propose a hierarchical experience pool equipped with tailored retrieval methods (Insight, Coarse, Fine) and a self-evolving mechanism that updates the pool from scratch using accumulated records, enabling adaptation to new data sources without retraining.
Empirical Validation: Extensive experiments demonstrate that EvoIR-Agent achieves a superior performance-efficiency trade-off compared to state-of-the-art methods, validating the impact of the experience mechanism and its capacity for self-evolution.

Experimental Results

The authors evaluated EvoIR-Agent on synthetic datasets (MiO100 with 16 degradation combinations) and real-world datasets (FoundIR).

Performance: EvoIR-Agent significantly outperforms existing training-free agents (e.g., AgenticIR, 4KAgent) and all-in-one restoration models (e.g., AirNet, PromptIR). In Setting I (AgenticIR-based tools), it led baselines by an average of 2.35 dB in PSNR and 0.065 in SSIM.
Efficiency: The system achieves a Pareto-optimal balance between performance and efficiency. Compared to strong baselines, it reduced tool invocation numbers by up to 70% while improving restoration quality.
Ablation Studies:
- Granularity: Moving from Insight to Fine-grained experience significantly reduced tool invocations and rollbacks, confirming the value of pattern-specific guidance.
- Evolution Times: The system reached an optimal sweet spot at 2 evolution times, where performance peaked and rollback overhead was minimized. Further evolution showed diminishing returns.
- Generalization: On out-of-distribution (OOD) real-world data, EvoIR-Agent demonstrated that initializing from scratch (EvoIR-Agent-S) with self-evolving capabilities yielded better results than zero-shot application of prior experience or simple transfer learning.

Significance and Claims

The paper claims that EvoIR-Agent successfully bridges the gap between the limited compatibility of training-based methods and the inference inefficiency of naive training-free paradigms. By systematically formulating experience components and introducing a self-evolving mechanism, the system enables an agent to learn from accumulated records, adapting to specific tools and degradation patterns dynamically.

The authors position this work as a step toward sustainable, experience-driven learning in image restoration, where the agent improves its own planning capabilities without requiring parameter updates or massive retraining datasets. They acknowledge current limitations regarding the computational cost of exhaustive exploration for highly entangled scenarios (4+ degradations) and the reliance on high-performance MLLMs for fine-grained pattern learning, suggesting future work in experience distillation and advanced exploration-exploitation strategies.