🤖 machine learning

A Principled Self-Referenced Early Stopping Approach for Deep Image Prior

This paper proposes a principled self-referenced early stopping framework for Deep Image Prior that constructs pseudo self-referenced images to robustly detect overfitting and achieve near-optimal reconstruction performance across various inverse imaging problems without requiring accurate noise level estimates.

Original authors: Chaoyan Huang, Cheng-Han Huang, Ismail R. Alkhouri, Rongrong Wang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chaoyan Huang, Cheng-Han Huang, Ismail R. Alkhouri, Rongrong Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Perfect" Photo That Never Happens

Imagine you have a blurry, noisy photo of a beautiful landscape. You want to clean it up. You use a smart computer program (called Deep Image Prior or DIP) that tries to "guess" what the clean photo looks like by learning from the blurry one.

Think of this program like a sculptor trying to carve a statue out of a block of noisy marble.

The Beginning: At first, the sculptor carves out the big, obvious shapes (the mountains, the trees). The statue looks great!
The Middle: As they keep working, they start adding fine details. The statue looks even better.
The Danger Zone: If the sculptor keeps working too long, they start carving the flaws in the marble itself (the tiny cracks and dust) thinking they are part of the art. Suddenly, the statue looks weird and ruined. This is called overfitting.

The biggest challenge with DIP is knowing when to stop.

Stop too early? The image is still blurry.
Stop too late? The image is full of weird artifacts (noise).

The Old Way: Guessing by Feeling

Previously, scientists tried to figure out when to stop by watching the "shakiness" of the sculpture. They thought, "If the statue starts wobbling or vibrating, we must be over-carving the noise."

However, the paper points out a flaw in this method. Sometimes, the statue wobbles naturally when adding fine details (like a leaf blowing in the wind), not just when it's ruined. This causes the old method to stop the sculptor too early, leaving the image blurry.

The New Idea: The "Self-Referenced" Mirror

The authors propose a clever new way to know exactly when to stop, without needing a "perfect" clean photo to compare against (which we don't have, otherwise we wouldn't need to fix the image).

They suggest creating a mirror image or a pseudo-reference from the noisy photo itself.

The Core Concept: The "Twin" Test

Imagine you have two identical twins who both have a cold. You want to see how well a medicine works.

The Problem: You can't give the medicine to one twin and compare them to the other if they are both sick in the exact same way.
The Solution: You give the medicine to Twin A. For Twin B, you give them the medicine plus a tiny, random sprinkle of confetti.
The Test: You watch Twin A. If Twin A starts looking sick again (overfitting), you stop. But you use Twin B (who has the extra confetti) as a reference to see if Twin A is doing something weird that Twin B isn't.

The paper creates three specific ways to build these "Twins" depending on the type of image:

The Three New Tools (Algorithms)

1. The "Color Swap" (CSR-ES) for Color Photos

How it works: Color photos have Red, Green, and Blue channels. Usually, the Red channel looks very similar to the Green channel, but the noise (the grain) is slightly different in each.
The Analogy: Imagine you are listening to a song on three different speakers. The music is the same, but the static (noise) is different on each speaker. The paper says, "Let's use the Green speaker to check if the Red speaker is getting too noisy."
Result: It finds the perfect moment to stop before the noise takes over.

2. The "Hide and Seek" (MR-ES) for Black & White Photos

How it works: Black and white photos (like medical X-rays) only have one channel, so you can't swap colors. Instead, the paper suggests hiding a tiny percentage of the pixels (like 2%) from the computer while it learns.
The Analogy: Imagine a teacher grading a student's homework. The teacher lets the student solve 98% of the problems but keeps 2% hidden. The student solves the 98%, and then the teacher checks the 2% they didn't see. If the student gets the hidden ones wrong, they are just memorizing the answers (overfitting) rather than learning.
Result: This works great for medical images like CT scans.

3. The "Extra Noise" (ACR-ES) for Tricky Noises

How it works: Sometimes the noise is very weird (like Poisson noise in low-light photos). The paper suggests taking the photo and adding extra random noise to it to create a "fake copy."
The Analogy: Imagine you are trying to hear a whisper in a noisy room. You put on headphones that play even more random static. If your brain starts trying to make sense of that extra static, you know you've gone too far.
Result: This is a flexible tool that works even when the noise is very difficult to handle.

Why This Matters (The Results)

The authors tested these methods on thousands of images, from nature photos to medical scans, with different types of noise (grain, static, light issues).

The Old Way: Often stopped too early, leaving images blurry.
The New Way: Consistently stopped at the "Goldilocks" point—just when the image looked perfect and before the noise ruined it.

They proved mathematically that by creating these "pseudo-references" (the twins, the hidden pixels, the extra noise), the computer can tell the difference between "good details" and "bad noise" much better than before.

Summary

The paper solves the problem of "when to stop" for AI image cleaning. Instead of guessing, it creates a self-check system using parts of the image itself to act as a reference. This allows the AI to stop at the perfect moment, creating clearer, sharper images without needing a "perfect" original photo to compare against.

Technical Summary: A Principled Self-Referenced Early Stopping Approach for Deep Image Prior

Problem Statement
Deep Image Prior (DIP) has emerged as a powerful training-data-free method for solving Inverse Imaging Problems (IIPs), such as denoising, super-resolution, and medical image reconstruction. By optimizing a randomly initialized convolutional neural network (CNN) to fit noisy measurements, DIP captures low-frequency image statistics effectively. However, due to network over-parameterization, DIP is prone to overfitting measurement noise or null-space artifacts. Consequently, determining the optimal stopping point (Early Stopping, ES) is critical.

Existing ES methods face significant limitations:

Variance-based methods (e.g., WMV-ES): These track fluctuations in the running variance of the network output. While effective in some settings, they often detect overfitting prematurely for high-resolution images or those with fine structures (e.g., textured natural images or medical scans), leading to unstable reconstructions far from the optimal "oracle" stopping point.
Risk-based methods (e.g., SURE): Stein's Unbiased Risk Estimator provides a principled risk estimate but requires precise knowledge of the noise level. In per-instance DIP settings, these estimates can exhibit high variance, leading to unreliable decisions.
Lack of Validation Data: Standard validation requires a clean ground-truth image, which is unavailable in IIPs. While obtaining two fully independent noisy copies of the degraded image would theoretically allow for near-optimal overfitting detection, this is generally infeasible in practice.

Methodology
The authors propose a framework that reduces DIP early stopping to a validation problem using pseudo self-referenced images. The core theoretical insight is that a pseudo-reference need not be perfectly independent of the fitting data; it is sufficient if the component shared between the fitting and validation signals is small or not preferentially fitted before the remaining noise.

The paper introduces three specific algorithms based on this principle, tailored to different image types and noise models:

Channel-Similarity Reference ES (CSR-ES) for RGB Images:
- Mechanism: Exploits the fact that RGB channels share structural information but contain partially decorrelated noise.
- Procedure: The method identifies the pair of noisy channels $(i, j)$ with the minimum distance (highest similarity). It trains the DIP on the full RGB image but uses the cross-channel consistency (fitting channel $i$ against reference channel $j$ ) as the stopping criterion.
- Rationale: This reduces clean cross-channel bias while ensuring the channel-specific noise remains less correlated with the fitted channel.
Mask-Reference ES (MR-ES) for Grayscale Images:
- Mechanism: Constructs a pseudo-reference by holding out a small subset of pixels (e.g., 2% of pixels) via a high-retention mask $M$ .
- Procedure: The DIP is trained using a masked loss $\|M \odot (f_\theta(z) - y)\|^2$ , explicitly excluding held-out pixels from the optimization. The stopping criterion is the error on the held-out pixels: $\|H \odot (\hat{x}_t - y)\|^2$ , where $H = 1-M$ .
- Rationale: Since the held-out pixels are excluded from the training loss, the noise at these locations is independent of the fitted trajectory, providing a valid validation signal.
Augmented-Channel Reference ES (ACR-ES) for Flexible/Challenging Noise:
- Mechanism: Augments the output space of the DIP network to handle cases where simple masks or channel splits are unreliable (e.g., signal-dependent Poisson noise).
- Procedure: Two pseudo-copies of the noisy image are generated: $y^{(1)} = y + \zeta_1$ and $y^{(2)} = y + \zeta_2$ , where $\zeta_1, \zeta_2$ are independent auxiliary perturbations. A 2C-channel DIP network is trained to fit $Y = \text{concat}(y, y^{(1)})$ . The primary channels reconstruct the image, while the auxiliary channels are monitored against the independent pseudo-copy $y^{(2)}$ .
- Stopping Metric: Instead of raw MSE, ACR-ES uses a frequency-bias score (the second moment of the residual spectrum). This score peaks when the network transitions from recovering structural details to fitting noise, making it more sensitive to the spectral bias of DIP.

Theoretical Contributions
The paper provides theoretical justification for these approaches:

Single-Reference Validation: Proves that if an independent noisy reference were available, the validation loss would concentrate around the clean-image Mean Squared Error (MSE) plus a constant, allowing for near-oracle stopping.
Pseudo-Validation Analysis: Demonstrates that even with a shared component $s$ between fitting and validation, if the shared energy is a small fraction of the non-shared variance, the excess MSE relative to the oracle is quadratically tighter than the worst-case bound.
Generalization: Extends the validation theory to general linear inverse problems ( $y = A(x) + \eta$ ) and establishes conditions under which measurement-domain validation transfers to image-domain risk.

Experimental Results
The authors evaluated their methods against baselines (WMV-ES and SURE-ES) across natural image restoration (Set14, CBSD68, Urban100, etc.) and medical image reconstruction (MRI, CT, PD, T2) under Gaussian, Poisson, and Impulse noise.

Performance: The proposed methods (CSR-ES, MR-ES, ACR-ES) consistently achieved stopping points much closer to the oracle (the iteration yielding maximum PSNR) than existing methods.
- For natural images, CSR-ES and ACR-ES reduced the PSNR gap to the oracle significantly (e.g., from ~1.0 dB to ~0.1–0.2 dB in many cases).
- For medical images, MR-ES and ACR-ES outperformed WMV-ES and SURE-ES, particularly in challenging Poisson noise scenarios where WMV-ES often stopped prematurely.
Robustness: The methods did not require accurate noise level estimates. ACR-ES showed robustness to variations in auxiliary noise magnitude and mismatched noise types.
Generalization: The stopping principle was successfully extended to super-resolution and inpainting tasks, where variance-based methods failed to align with the oracle stopping region.

Significance and Claims
The paper claims that early stopping for DIP can be effectively reduced to a validation problem solvable without clean ground truth or precise noise estimates. By constructing pseudo self-referenced images that sufficiently decorrelate the validation signal from the fitting process, the authors provide a principled alternative to heuristic variance tracking.

The significance lies in:

Principled Approach: Moving beyond heuristic variance monitoring to a validation-based framework supported by concentration inequalities.
Practical Applicability: Offering specific, task-adapted algorithms (CSR, MR, ACR) that work across diverse IIPs (denoising, super-resolution, inpainting) and noise types without requiring external training data or noise level tuning.
Reliability: Demonstrating that these methods consistently select iterations closer to the optimal reconstruction quality than current state-of-the-art ES techniques, thereby mitigating the overfitting issue inherent in over-parameterized DIP networks.