FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration

FAPE-IR is a unified image restoration framework that leverages a frozen Multimodal Large Language Model to generate frequency-aware restoration plans, guiding a LoRA-based Mixture-of-Experts diffusion executor to dynamically select appropriate experts for achieving state-of-the-art performance and strong zero-shot generalization across diverse degradation tasks.

The Gist

Imagine you have a beautiful, old family photo that has suffered from a mix of problems: it's a bit blurry, covered in rain streaks, looks a bit dark, and has some dust specks on it.

In the past, fixing this was like hiring seven different specialists. You'd hire a "blur fixer," then a "rain remover," then a "light enhancer." If you tried to do it all at once with one person, they would get confused, mixing up their tools and making the photo look worse (like trying to use a hammer to fix a watch).

FAPE-IR is a new, super-smart system that acts like a Master Restorer who doesn't just fix the photo, but first thinks about how to fix it.

Here is how it works, broken down into simple steps:

1. The "Brain" (The Planner)

Imagine a very smart art critic (powered by a giant AI brain called an MLLM) who looks at your damaged photo.

• What it does: Instead of guessing, this critic analyzes the image and writes a short, clear note.
• The Note: It says things like: "Okay, this photo has rain streaks (which are sharp, fast lines) and it's also a bit dark (which is a slow, global change). To fix the rain, we need to focus on the sharp details. To fix the darkness, we need to focus on the overall glow."
• The Magic: It breaks the problem down into two categories: High Frequency (sharp edges, textures, rain, snow, noise) and Low Frequency (overall brightness, haze, color shifts).

2. The "Hands" (The Executor)

Once the "Brain" writes the plan, it hands it to a team of specialized workers (called LoRA-MoE).

• The Team: Imagine a workshop with two main stations:
  • Station A (High-Frequency Expert): Good at fixing sharp lines, removing rain streaks, and cleaning up grainy noise.
  • Station B (Low-Frequency Expert): Good at fixing overall brightness, removing fog, and balancing colors.
• The Switch: The "Brain" tells the system: "Hey, the rain is the biggest problem here, so send the image to Station A first, then maybe a little bit to Station B."
• Why this helps: In the past, one big machine tried to do everything at once and got confused. This system dynamically switches between the experts, ensuring the right tool is used for the right part of the image.

3. The "Quality Control" (Adversarial Training)

Imagine a strict art teacher (the Discriminator) who sits next to the restorer.

• The Problem: Sometimes, AI gets too creative. It might remove the rain but accidentally paint a fake tree or make the sky look too smooth and plastic-like. This is called "hallucination."
• The Solution: The teacher constantly checks the work. If the restorer tries to add fake details or smooth things out too much, the teacher says, "No, that doesn't look real. Go back and fix it."
• The Result: The final image looks sharp and real, without those weird, fake-looking artifacts.

Why is this a big deal?

• It's One-Stop-Shop: You don't need to know if your photo has rain, blur, or fog. You just dump the bad photo in, and FAPE-IR figures out the mix and fixes it all.
• It's Smart: It understands that fixing rain (sharp lines) is different from fixing fog (soft clouds). By separating these tasks, it avoids the "confusion" that ruins other AI tools.
• It Works on Mixed Messes: Real life is messy. A photo might be rainy and dark. FAPE-IR handles these complex, mixed-up situations better than any previous method.

In a nutshell:
FAPE-IR is like hiring a Project Manager (the Planner) who understands the problem, assigns the Specialized Workers (the Experts) to the right tasks, and has a Strict Inspector (the Adversarial Trainer) to ensure the final product looks natural and perfect. It turns a chaotic mess of image problems into a clean, restored masterpiece.

Technical Summary

1. Problem Statement

All-in-One Image Restoration (AIO-IR) aims to build a single unified model capable of handling multiple types of image degradations (e.g., rain, snow, haze, blur, noise, low-light, super-resolution) simultaneously. However, existing approaches face significant challenges:

• Cross-Task Interference: Methods using multi-branch mappings often suffer from gradient conflicts when optimizing for different tasks within a shared parameter space.
• Gradient Isolation: Methods relying on latent routing or clustering often separate tasks too rigidly, preventing the discovery of shared structures and limiting generalization to compound degradations.
• Lack of Semantic Understanding: Most current models rely on fixed task labels or hand-crafted priors, lacking the ability to autonomously perceive and disentangle degradation attributes in real-world, unknown scenarios.
• Artifacts in Generative Models: Recent diffusion-based approaches often suffer from over-generation, hallucinated details, or structural distortions due to spectral biases in their training objectives (e.g., Flow Matching underweights high frequencies).

2. Methodology: FAPE-IR

The authors propose FAPE-IR, a framework that couples a Multimodal Large Language Model (MLLM) as a "Planner" with a Diffusion-based Executor equipped with a Frequency-Aware LoRA-Mixture-of-Experts (LoRA-MoE) architecture.

A. Frequency-Aware Planner (The "Brain")

Instead of relying on pre-defined task labels, FAPE-IR uses a frozen MLLM (Qwen2.5-VL) to analyze the degraded image and generate a structured restoration plan.

• Label-Free Feature Pool: The system computes simple, label-free low-level statistics (e.g., gradient orientation for rain, blob density for snow, dark channel for haze) directly from pixel values to characterize the degradation.
• Instruction & Planning: The MLLM receives these visual hints, universal restoration instructions, and expert rules. It outputs a structured tuple:
  • Task Type: The specific degradation(s) present.
  • Frequency Focus: Whether the restoration requires High-Frequency (edges, textures, rain/snow/noise removal) or Low-Frequency (global illumination, haze removal) processing.
  • Pipeline: A step-by-step reasoning trace.
• Token Encoding: The planner's output is encoded into compact tokens that guide the executor, ensuring the restoration process is semantically driven and interpretable.

B. Frequency-Aware LoRA-MoE Executor (The "Hands")

The executor is a diffusion-based model (FLUX transformer) that refines the image latent representation.

• Dual-End Gating Mechanism: A novel Frequency-Aware LoRA-MoE module dynamically selects between two specialized experts: a High-Frequency Expert and a Low-Frequency Expert.
  • Text Router: Uses the planner's semantic tokens to propose an initial expert selection.
  • Spectral Router: Analyzes intermediate generation tokens using a depth-wise FIR (Finite Impulse Response) filter to separate high/low-frequency energy, refining the selection based on actual spectral content.
• Dynamic Routing: The final gating weights are a fusion of semantic and spectral cues, allowing the model to handle mixed degradations by activating the appropriate expert or a combination thereof.

C. Training Strategy: Adversarial + Frequency Regularization

To address the artifacts common in diffusion models (like Flow Matching), FAPE-IR employs a specific training recipe:

• Adversarial Training: Replaces standard flow-matching objectives with a composite adversarial loss. This includes a discriminator (SigLIP-based) to ensure distributional alignment, pixel-wise MSE, and perceptual loss (LPIPS). This prevents "over-generation" and hallucinated details.
• Frequency Regularization: A specific loss term ($\mathcal{L}_{freq}$) penalizes experts for synthesizing energy outside their designated frequency band (e.g., preventing the low-frequency expert from generating high-frequency noise). This enforces strict specialization and reduces cross-task interference.

3. Key Contributions

1. Unified Understanding-Generation Framework: FAPE-IR is the first AIO-IR framework to explicitly couple a semantic planner (MLLM) with a frequency-specialized executor, enabling autonomous degradation perception without task labels.
2. Frequency-Aware LoRA-MoE: Introduces a dual-gating mechanism that dynamically routes information between high- and low-frequency experts based on both semantic intent and spectral evidence, effectively mitigating gradient conflicts.
3. Theoretical & Empirical Fidelity: Provides theoretical analysis showing why Flow Matching leads to artifacts in restoration (spectral bias) and demonstrates that adversarial training with frequency regularization yields higher fidelity and fewer artifacts.
4. State-of-the-Art Performance: Achieves SOTA results across seven restoration tasks and demonstrates strong zero-shot generalization to unseen compound degradations.

4. Experimental Results

• Quantitative Performance: FAPE-IR outperforms existing AIO-IR methods (e.g., PromptIR, AdaIR, DFPIR, MoCE-IR) and specialized SR models on seven benchmarks (Deraining, Desnowing, Dehazing, Deblurring, Denoising, Low-light, SR).
  • Weather Tasks: Significant gains in PSNR (e.g., +6–8 dB on deraining) and perceptual metrics (LPIPS, FID).
  • Mixed Degradations: Exhibits robust zero-shot generalization on the CDD-11 benchmark (compound degradations), where other models fail.
• Qualitative Quality: Visual comparisons show FAPE-IR preserves fine textures and edges better than unified models (which hallucinate) and AIO-IR baselines (which over-smooth). It effectively removes artifacts like ringing and color shifts.
• Efficiency: Despite a larger parameter count, inference time is reasonable (~1.57s on H200 GPU) and significantly faster than some unified autoregressive counterparts.

5. Significance

FAPE-IR represents a paradigm shift in low-level vision restoration. By moving from task-conditioned generation to semantic planning and frequency-aware execution, it solves the fundamental trade-off between task specialization and unified modeling.

• Interpretability: The planner provides a human-readable reasoning trace, making the restoration process transparent.
• Robustness: The frequency-aware design ensures that the model adapts to the specific spectral nature of the degradation, leading to more stable optimization and fewer artifacts in real-world, complex scenarios.
• Future Direction: It establishes a blueprint for integrating high-level semantic reasoning (via MLLMs) with low-level signal processing (via diffusion and frequency analysis) for general-purpose vision tasks.