💻 computer science

Leveraging Multimodal Large Language Models for All-in-One Image Restoration via a Mixture of Frequency Experts

This paper proposes a novel multimodal large language model (MLLM)-guided image restoration framework that utilizes MLLM-derived features and a mixture-of-frequency-experts (MoFE) module with relational alignment to effectively handle continuous and composite degradations, achieving state-of-the-art performance on benchmarks like CDD11.

Original authors: Eunho Lee, Youngbae Hwang, Rei Kawakami

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eunho Lee, Youngbae Hwang, Rei Kawakami

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful photo, but it's been ruined by a mix of problems: maybe it's blurry, covered in rain, and looks too dark all at the same time. Fixing one of these issues is hard enough, but fixing them all together is like trying to untangle a knot while wearing blindfolded gloves.

This paper introduces a new way to fix these "all-in-one" messy photos using a smart digital assistant called a Multimodal Large Language Model (MLLM). Think of this MLLM not just as a text-chatting robot, but as a super-observant art critic who has seen millions of photos and knows exactly what "rain," "fog," or "blur" looks like and how they feel.

Here is how their new system works, broken down into simple parts:

1. The Problem: The "One-Size-Fits-All" Trap

Old methods tried to fix photos by treating problems like separate boxes. They had a "blur box," a "rain box," and a "darkness box." But real life isn't so neat. A photo might be 70% rainy and 30% foggy. Old systems struggled with this mix because they couldn't see the smooth connection between the two; they just saw two separate, unrelated problems.

2. The Solution: A Smart Guide (The MLLM)

The authors realized that these big AI models (MLLMs) already understand the nuance of bad photos. They don't just say "this is rain"; they understand that "heavy rain" feels different from "light drizzle," and how rain mixes with fog.

They use this AI model as a guide for the photo-fixing tool.

The Analogy: Imagine a master chef (the photo restorer) trying to cook a complex dish. Instead of just giving them a recipe, they hire a food critic (the MLLM) to stand next to them. The critic tastes the ingredients and whispers, "Hey, that sauce needs a little more salt, and the texture is a bit off." The chef then adjusts the cooking in real-time based on that advice.

3. The Two Secret Weapons

To make this guide work, the paper introduces two special tools:

A. The "Whispering Bridge" (MGFB)

Usually, the photo-fixing tool and the AI guide speak different languages. The fixer looks at pixels (dots of color), while the guide thinks in concepts (like "foggy" or "grainy").

What they did: They built a "bridge" (called the MLLM-Guided Fusion Block) that translates the guide's advice directly into the fixer's language.
How it works: As the fixer looks at the photo, the bridge whispers, "Focus on the structure here because it looks like fog," and later, "Pay attention to the colors here because it looks like rain." This helps the fixer know exactly what to look for at every step.

B. The "Frequency Orchestra" (MoFE)

This is the most clever part. The authors realized that different problems live in different "frequencies."

The Analogy: Think of a photo like a song.
- Low frequencies are the bass notes (the big shapes, the overall brightness, the fog).
- High frequencies are the high-pitched notes (the sharp edges, the rain streaks, the fine details).
The Problem: Old tools tried to fix the whole song at once, which often muddied the sound.
The Solution: They created a Mixture of Frequency Experts (MoFE). Imagine a band with 8 different musicians.
- One musician is an expert at fixing the bass (fog).
- Another is an expert at fixing the high-pitched squeaks (rain).
- Another fixes the middle tones (blur).
The Conductor: The AI guide acts as the conductor. It listens to the messy photo and tells the band, "Hey, we need more help from the bass player right now, but the violin player can relax." This allows the system to pick the right "expert" for the specific type of noise in the photo.

4. The Result: A Better Mix

Because the system uses the AI guide to understand the relationship between different problems (like how rain and fog blend together), it doesn't just fix them one by one. It fixes the whole mix at once.

The paper tested this on a very difficult dataset called CDD11, which contains photos with three different problems happening at the same time (like Low-light + Haze + Rain).

The Outcome: Their method produced clearer, sharper photos than any previous method. It improved the quality by a significant margin (up to 1.35 dB), which in the world of image fixing is like going from a blurry, muddy photo to a crisp, high-definition one.

Summary

In short, this paper teaches a photo-fixing robot to listen to a super-smart AI guide. Instead of guessing what's wrong with a messy photo, the robot asks the guide, "What does this look like?" The guide explains the mix of problems, and the robot uses a team of specialized "frequency experts" to fix each part of the problem perfectly, resulting in a clean, natural-looking image.

Technical Summary: Leveraging Multimodal Large Language Models for All-in-One Image Restoration via a Mixture of Frequency Experts

Problem Statement

All-in-one image restoration aims to recover clean images from inputs affected by diverse and unknown degradations using a unified framework. While recent methods have improved performance by identifying degradation characteristics, they often treat degradations as discrete categories. This approach fails to model the continuous relational structure inherent in composite degradations (e.g., an image with simultaneous low-light, haze, and rain where one degradation is dominant). Existing degradation-sensitive representations, often learned via contrastive objectives, tend to partition degradation types into isolated clusters, overlooking the continuous spectrum of mixed degradations. Furthermore, current approaches leveraging Multimodal Large Language Models (MLLMs) frequently rely solely on high-level textual representations, underutilizing the rich internal visual knowledge encoded within these models, which creates a representation gap for fine-grained, pixel-level reconstruction.

Methodology

The authors propose MMFE-IR, a framework that integrates a pre-trained Multimodal Large Language Model (specifically Qwen2.5-VL) with a low-level image restoration network. The architecture consists of three primary components:

1. MLLM-Guided Fusion Block (MGFB)

To bridge the gap between high-level semantic knowledge and low-level restoration, the framework injects MLLM-derived features into an encoder-decoder backbone (based on Restormer). The MGFB utilizes a cross-attention mechanism to assign specific MLLM embeddings to different network stages:

Encoder: Receives $E_{Image}$ , a contextualized vision-derived embedding pooled from the MLLM's image tokens. This strengthens structural feature extraction.
Decoder: Receives $E_{Joint}$ , vision-language conditioned embeddings pooled from all tokens. This facilitates degradation-aware reconstruction by incorporating semantic context.
The fusion is achieved via a residual connection where the visual feature map serves as the query and the MLLM embeddings serve as keys and values.

2. Mixture-of-Frequency Experts (MoFE)

Recognizing that different degradations exhibit distinct frequency characteristics, the authors introduce a MoFE module operating in the frequency domain rather than the spatial domain.

Frequency Decomposition: The original degraded image is decomposed using Discrete Wavelet Transform (DWT) into subbands. The low-frequency component ($LL$) is excluded, as it is inherently captured by self-attention mechanisms in Transformers. The remaining high-frequency subbands ($LH, HL, HH$) are processed.
Expert Routing: A set of lightweight convolutional experts ( $N=8$ ) processes these high-frequency components. A router, guided by the MLLM's high-level textual embedding ( $E_{Answer}$ ), predicts mixture weights for these experts. This allows for dynamic composition to handle diverse degradations.
Fusion: The aggregated frequency features are fused back into the spatial features using Multi-Dconv Head Transposed Attention (MDTA), allowing spatial features to adaptively query necessary high-frequency details.

3. MLLM-Guided Router Optimization

To ensure the router preserves the continuous semantic relationships of degradations captured by the MLLM, a specific loss function is introduced.

Relational Alignment Loss ( $\mathcal{L}_{MGL}$ ): This loss computes the pairwise cosine similarity of the MLLM's high-level embeddings and aligns it with the pairwise similarity of the router's output weights. This encourages the router to assign similar expert combinations to semantically similar degradations and distinct combinations to different degradations, maintaining the continuous structure of the embedding space.
Training Strategy: The model employs a progressive multi-scale training strategy, gradually increasing input resolution to stabilize router optimization and improve robustness across varying image sizes.

Key Contributions

Continuous Degradation Modeling: The framework leverages pre-trained MLLMs' continuous degradation embeddings to overcome the limitations of rigid prompt discretization, enabling more accurate capture of complex degradation structures.
MGFB Integration: An MLLM-guided fusion block is introduced to bridge the representation gap by integrating both MLLM vision embeddings and vision-language-conditioned embeddings directly into the restoration network's encoder and decoder.
Frequency-Domain Expert Routing: A Mixture-of-Frequency-Experts (MoFE) module is proposed, utilizing an MLLM-guided router trained with a relational alignment loss. This allows the model to learn distributed degradation relationships and adaptively combine frequency experts for diverse and composite degradations.

Experimental Results

The method was evaluated on multiple benchmarks, including the challenging CDD11 dataset for composite degradations and standard all-in-one settings for single degradations.

Composite Degradation (CDD11): The proposed method achieved state-of-the-art performance, outperforming previous methods by up to 1.35 dB on the CDD11 dataset. It demonstrated superior robustness in double (e.g., Low-light + Haze) and triple (e.g., Low-light + Haze + Rain) degradation scenarios, achieving an average PSNR of 30.40 dB and SSIM of 0.889.
All-in-One Benchmarks: The method achieved the highest average PSNR (31.44 dB) on the five-degradation setting and 32.94 dB on the three-degradation setting. It showed particular strength in both low-frequency tasks (dehazing) and high-frequency tasks (deblurring).
Ablation Studies: Experiments confirmed that:
- Using $E_{Image}$ for the encoder and $E_{Joint}$ for the decoder yields optimal performance.
- The MoFE module significantly boosts performance (+1.02 dB over baseline) with minimal parameter increase.
- The MLLM-guided loss and progressive multi-scale training are critical for performance gains.
- Frequency-domain MoE outperforms spatial-domain MoE.
- The framework remains robust even when using smaller MLLMs (e.g., FastVLM-0.5B), though the largest MLLM (Qwen2.5-VL-3B) provides the best balance of quality and latency.

Significance and Claims

The paper claims that MMFE-IR establishes a new state of the art in all-in-one image restoration, particularly for complex, composite degradations. The significance of the work lies in its successful extension of MLLM guidance beyond mere textual representations to include internal visual features and frequency-domain expert routing. By preserving the continuous semantic relationships of degradations through the MLLM-guided router and loss, the framework effectively mitigates the representation gap often found in low-level vision tasks. The authors assert that this approach allows the model to handle the coexistence and interaction of multiple degradations more accurately than methods that treat degradations as discrete, isolated categories.