💻 computer science

CLEAR: Unlocking Generative Potential for Degraded Image Understanding in Unified Multimodal Models

The paper introduces CLEAR, a framework that enhances the robustness of unified multimodal models on degraded images by establishing a generate-then-answer reasoning pattern through supervised fine-tuning, a direct latent representation bridge, and interleaved reinforcement learning, thereby effectively leveraging generative capabilities to restore fine-grained visual structure lost to degradation.

Original authors: Xiangzhao Hao, Zefeng Zhang, Zhenyu Zhang, Linhao Yu, Yao Chen, Yiqian Zhang, Haiyun Guo, Shuohuan Wang, Yu Sun

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiangzhao Hao, Zefeng Zhang, Zhenyu Zhang, Linhao Yu, Yao Chen, Yiqian Zhang, Haiyun Guo, Shuohuan Wang, Yu Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, all-knowing assistant who can both read a document and draw a picture. This assistant is trained on perfect, crystal-clear images and text.

Now, imagine you hand this assistant a photo that has been dropped in the mud, covered in static, or blurred by rain. You ask, "What is this person holding?"

The Problem:
Even though this assistant has a superpower to draw (generate) images, when it looks at your muddy photo, it just tries to guess the answer using its reading skills. It fails. It doesn't think, "Hey, I can't see clearly, let me use my drawing power to clean up the picture in my mind first, and then answer."

The paper calls this a "functional disconnect." The assistant has two superpowers (Reading and Drawing), but they are in separate rooms and never talk to each other when things get messy.

The Solution: CLEAR
The researchers built a new framework called CLEAR (Comprehension via Latent Enhancement and Adaptive Reasoning). Think of it as training the assistant to be a detective who knows when to call in a forensic artist.

Here is how they did it, using three simple steps:

1. Teaching the "Stop and Fix" Habit (Behavioral Initialization)

Imagine you are teaching a child to solve a puzzle.

Before: If the puzzle piece was broken, the child just guessed.
Now: You show the child a broken puzzle. You say, "If you can't see the picture clearly, stop. First, use your magic marker to redraw the missing parts on a piece of paper. Then, look at your new drawing and tell me the answer."

The researchers created a special training set where the model learns this exact pattern: Analyze -> (If blurry) -> Redraw -> Analyze Again -> Answer.

2. Building a Secret Tunnel (The Latent Representation Bridge)

Here is the tricky part. In old models, if the assistant drew a picture, it had to:

Print the drawing on paper (convert to pixels).
Scan the paper back into the computer (re-encode).
Then read the scanned image.

This is slow and loses detail. It's like trying to talk to someone by shouting through a thick wall.

CLEAR builds a secret tunnel. When the assistant draws the "cleaned" version of the image, it sends the raw blueprint of that drawing directly into its brain to be read. No printing, no scanning. This allows the "drawing" part to directly influence the "thinking" part instantly.

3. The "Reward System" (Interleaved GRPO)

Now, how do we teach the model to draw better pictures that actually help it answer?

Old Way: "Draw a picture that looks exactly like the original clean photo." (This is like asking an artist to copy a photo perfectly, even if the photo is irrelevant to the question).
CLEAR Way: "I don't care if your drawing looks like a masterpiece. I only care if your drawing helps you get the right answer."

They use a reinforcement learning method (like training a dog with treats).

If the model draws a picture and gets the answer right? Treat!
If it draws a picture and gets it wrong? No treat.
If it didn't need to draw but did anyway? No treat.

The Surprising Discovery:
The researchers expected that if they stopped forcing the model to draw "perfect" pictures (pixel-perfect reconstruction), the drawings would look worse.
They were wrong.
Because the model was rewarded only for getting the answer right, it learned to draw exactly what it needed to see the answer. It turned out that drawing a helpful image and drawing a beautiful image are the same thing. The model naturally created sharper, clearer images that were perfect for reasoning, without needing a teacher to say "make it look pretty."

The Result: The Adaptive Detective

The final model is like a smart detective:

Clear Photo? It looks, thinks, and answers immediately. (Fast and efficient).
Blurry Photo? It says, "I can't see that," triggers its internal "forensic artist," redraws the missing details in its mind, and then answers.

Why this matters:
Real-world photos are rarely perfect. They are taken in the rain, at night, or with shaky hands. CLEAR proves that by connecting a model's ability to generate (fix) images with its ability to understand them, we can make AI much more robust and reliable in the messy real world.

In short: They taught the AI to stop guessing when things are blurry, and instead, to use its own imagination to clean up the mess before answering.

1. Problem Statement

The Challenge of Image Degradation: Real-world visual data (e.g., from autonomous driving, surveillance, mobile photography) is frequently corrupted by blur, noise, compression, and poor illumination. These degradations destroy low-level visual cues essential for recognition and reasoning.
The Limitation of Current Unified Models: Unified Multimodal Models (UMMs) possess both visual understanding and image generation capabilities within a single architecture. However, the paper identifies a critical functional disconnect:

Behavioral Gap: Existing UMMs are never trained to invoke their generative capabilities as part of the reasoning process for understanding tasks. They treat generation and understanding as separate objectives.
Structural Bottleneck: Even if a model attempts to generate an image to aid reasoning, the standard pipeline requires decoding the generated latent representation into pixel space and then re-encoding it through a frozen vision encoder. This "decode-reencode" detour severs the gradient flow, preventing answer-level supervision from optimizing the generation process.
Result: Unified models fail to leverage their generative capacity to compensate for degraded inputs, leading to substantial performance drops on degraded images compared to clean ones.

2. Methodology: The CLEAR Framework

The authors propose CLEAR (Comprehension via Latent Enhancement and Adaptive Reasoning), a three-stage framework designed to bridge the gap between generation and reasoning.

Stage 1: Behavioral Initialization (Supervised Fine-Tuning)

Goal: Teach the model the "generate-then-answer" reasoning pattern.
Data Construction: A degradation-aware dataset is constructed from LLaVA-OneVision. Samples are split based on the base model's performance:
- Direct-Answer Pathway: For mild/no degradation where the model answers correctly.
- Generate-Then-Answer Pathway: For severe degradation where the model fails. Here, the model is trained to first diagnose the image, trigger a restoration tool (<image_restore>), generate an intermediate visual state, and then answer.
Training Objective: Uses Next-Token Prediction (LCE) for the reasoning trajectory. It includes auxiliary losses:
- MSE Loss: Encourages the generated latent to approximate the clean image.
- KL Distillation Loss: Aligns the VAE latent representations with the ViT feature distribution (used as a teacher) to ensure the language model can interpret the generated latents.

Stage 2: Latent Representation Bridge

Goal: Eliminate the structural bottleneck.
Mechanism: Instead of decoding generated latents to pixels and re-encoding them, CLEAR injects the generated VAE latent tokens directly into the reasoning context alongside the original text and ViT features.
Impact: This creates a differentiable connection between the generation process and the reasoning output. Gradients from the final answer can now flow directly back to the generation parameters, enabling joint optimization.

Stage 3: Interleaved GRPO (Reinforcement Learning)

Goal: Jointly optimize text reasoning and visual generation using answer correctness as the primary signal.
Algorithm: Combines GRPO (Group Relative Policy Optimization) for text and Flow-GRPO for image generation (flow matching models).
- Trajectory Sampling: The model generates $G$ interleaved sequences (text + denoising steps) per input.
- Efficiency: To manage memory, only one denoising step per trajectory is selected for the forward pass, while the rest are recorded as state pairs. This reduces the cost from $N$ forward passes to 1.
- Shared Advantage: Both text and image components share the same group-relative advantage ( $\hat{A}_i$ ) derived from a single reward signal.
Reward Design:
- $R_{acc}$ : Answer correctness (evaluated by an LLM-as-a-judge).
- $R_{fmt}$ : Output structure validity.
- $R_{dec}$ : Decision Reward. Rewards the model for generating when it helps (correct answer after generation) and penalizes skipping generation when it was needed (incorrect answer without generation). This encourages an adaptive generation strategy.
Key Insight: No explicit pixel-level reconstruction loss is used in this stage. The model learns to generate visual states solely based on their utility for the final answer.

3. Key Contributions

Identification of Disconnect: The paper pinpoints that the failure of UMMs on degraded images is due to a lack of training to use generation for reasoning and a structural barrier (decode-reencode) preventing joint optimization.
CLEAR Framework: A novel pipeline integrating Behavioral Initialization, a Latent Representation Bridge, and Interleaved GRPO to enable end-to-end optimization of generation for understanding.
MMD-Bench: A new benchmark constructed by applying 16 real-world corruption types at three severity levels across six standard multimodal benchmarks, providing a rigorous test for degraded-image understanding.
Counter-Intuitive Finding: The paper demonstrates that removing explicit pixel-level reconstruction supervision and relying solely on task-driven (answer-correctness) rewards actually improves the perceptual quality of generated intermediate states. This suggests that task optimization and visual quality are naturally aligned, and reconstruction targets act as constraints.

4. Experimental Results

Performance: CLEAR significantly outperforms existing open-source unified models (Bagel, Janus-Pro, Emu3) and even some closed-source models (GPT-4o-mini) on degraded inputs.
- On MMD-Bench (Hard degradation), CLEAR-RL achieves an average score of 65.26, compared to Bagel's 60.15 and Janus-Pro's 48.64.
- It improves robustness by reducing the performance drop from clean to hard inputs by 24% compared to the base model.
Adaptive Efficiency: The model learns to skip generation on clean or mildly degraded images (triggering only ~~5% of the time at low degradation) and triggers it frequently on severe degradation (~~36% at high degradation), balancing accuracy and inference cost.
Ablation Studies:
- Removing the Latent Bridge (reverting to decode-reencode) causes a significant performance drop, confirming the necessity of the direct gradient path.
- Removing Interleaved GRPO (SFT only) yields gains but fails to reach the full potential of joint optimization.
- Perceptual Quality: Intermediate states generated by the RL stage have higher perceptual quality (better BRISQUE, NIQE, MUSIQ scores) than those from the SFT stage, despite lacking MSE supervision.

5. Significance

Paradigm Shift: CLEAR moves beyond treating image restoration as a pre-processing step or a separate task. It integrates restoration into the reasoning loop, allowing the model to "think" about what visual information is missing and "generate" it to solve the problem.
Efficiency: By using an adaptive policy, the model avoids unnecessary computation on clean inputs, making it practical for real-world deployment where image quality varies.
Optimization Insight: The finding that task-driven rewards yield higher perceptual quality than reconstruction losses challenges the conventional wisdom in image restoration, suggesting that "good enough" visual features for the specific task are superior to generic pixel-perfect reconstruction.

In summary, CLEAR successfully unlocks the latent generative potential of unified models, transforming them from passive observers of degraded images into active agents that can reconstruct missing visual details to achieve robust understanding.