Technical Summary: Weak Diffusion Priors Can Still Achieve Strong Inverse-Problem Performance

1. Problem Statement

Inverse problems aim to recover an unknown signal $x$ from noisy measurements $y = A(x) + \epsilon$. Recently, diffusion models have become state-of-the-art priors for these tasks, typically utilizing "full-strength" models (e.g., 1000-step DDPMs) trained on data distributions matching the target signal. However, practical constraints often necessitate the use of "weak priors," defined by two axes of weakness:

1. Generator Quality: The model is truncated to very few reverse steps (e.g., 3-step DDIM), resulting in low-fidelity unconditional samples.
2. Domain Mismatch: The model is trained on a distribution different from the target signal (e.g., using a bedroom-trained model to reconstruct faces).

Standard intuition suggests that reconstruction quality is bounded by prior quality. Yet, empirical observations indicate that weak priors often perform nearly as well as strong, matched baselines. This paper investigates when and why weak diffusion priors succeed in inverse problems and identifies the regimes where they fail.

2. Methodology

2.1. Algorithmic Framework: Initial Noise Optimization

The authors focus on solving inverse problems by optimizing the initial latent noise $z$ of a generative model $G$, rather than modifying the reverse diffusion chain at every step. The objective is:
$$\min_z \|A(G(z)) - y\|^2$$
To address specific challenges in optimizing weak generators, the authors propose two algorithmic refinements:

• ADAMSPHERE: A modified ADAM optimizer that constrains the latent vector $z$ to remain on the typical sphere $\|z\| \approx \sqrt{d}$, preventing the optimization from pushing the noise off the high-probability shell where diffusion models are trained.
• HOLDOUTTOPK Early Stopping: To mitigate overfitting to measurement noise, a subset of measurements is held out. The algorithm tracks the loss on this holdout set and returns the latest iterate among the top-$K$ best-performing checkpoints, rather than selecting the single minimum error.

2.2. Theoretical Analysis

The paper employs two complementary theoretical mechanisms to explain the robustness of weak priors:

1. Bayesian Consistency and Data Dominance:
   Using a Gaussian mixture surrogate for the prior, the authors analyze the posterior distribution $\pi(x|y)$. They demonstrate that in high-dimensional settings with a large number of observed pixels ($m$), the posterior concentrates exponentially on the component of the mixture that minimizes the measurement residual (score).

   • Key Insight: If the measurement is sufficiently informative (large effective dimension $m$) and the best-matching candidate is distinguishable from others (score gap $\delta_0 > 0$), the posterior collapses to the correct mode regardless of the prior's specific weights, provided the weights are bounded. Thus, the data dominates the prior.

2. Local Spatial Correlation:
   The authors analyze the spatial autocorrelation profiles of unconditional samples from weak (few-step) and mismatched (out-of-domain) priors versus strong (many-step, in-domain) priors.

   • Key Insight: Despite significant differences in global fidelity and semantic content, weak and strong priors share nearly identical short-range local spatial correlations. This suggests that even weak priors can effectively propagate local constraints (anchors provided by measurements) to unobserved regions, provided the measurements supply enough local anchors.

3. Key Contributions

1. Empirical Evidence: Extensive experiments across inpainting, deblurring, and super-resolution tasks show that 3-step DDIM priors (both in-domain and out-of-domain) achieve performance comparable to or better than 1000-step DDPM baselines (e.g., DPS) in data-informative regimes. For instance, a bedroom-trained 3-step model successfully reconstructs face images.
2. Mechanistic Explanation: The paper provides a principled justification for this phenomenon:
   • Data Dominance: High-information measurements force the posterior to concentrate on the measurement-consistent signal, reducing sensitivity to prior choice.
   • Shared Structure: Weak priors retain sufficient local spatial structure to support reconstruction when guided by strong measurements.
3. Identification of Failure Modes: The authors identify regimes where weak priors fail, specifically when the effective measurement dimension is small or the missing region is large (e.g., large box inpainting, high-factor super-resolution). In these cases, the score gap $\delta_0$ becomes small, and the posterior fails to concentrate, making the reconstruction highly dependent on the prior's global semantic capabilities.
4. Algorithmic Refinements: The introduction of ADAMSPHERE and HOLDOUTTOPK stabilizes the initial-noise optimization process, reducing overfitting and improving reconstruction quality compared to standard optimization baselines.

4. Experimental Results

• Cross-Domain Performance: In random inpainting and 4x super-resolution, the proposed method using a 3-step out-of-domain prior (e.g., Bedroom model for CelebA) achieved PSNR gains of 0.88–1.80 dB over the standard DPS baseline (which uses a 1000-step in-domain prior).
• Comparison with Optimization Baselines: Compared to DMPlug (a state-of-the-art optimization-based method), the proposed method (using ADAMSPHERE and HOLDOUTTOPK) consistently achieved higher PSNR (gains of 0.15–1.0 dB) and significantly lower LPIPS (up to 30% improvement).
• Failure Analysis: In large box inpainting (masking 60% of an image) and 16x super-resolution, weak priors degraded significantly. The out-of-domain prior produced semantically inconsistent structures (e.g., face-like artifacts in a church scene), while the strong prior maintained better coherence.

5. Significance and Practical Implications

The paper argues that the reliance on expensive, domain-matched, full-strength diffusion models may be unnecessary in data-informative regimes.

• Practical Guide: Weak priors (few-step or mismatched) serve as effective, reasonable defaults for tasks like random inpainting, deblurring, and moderate super-resolution, where measurements provide many local anchors.
• Caveats: In low-information regimes (large missing regions, high noise, or tasks requiring global semantic completion), strong, domain-matched priors remain essential to avoid generating plausible but incorrect structures.
• Broader Impact: These findings suggest that in fields like medical imaging or scientific reconstruction, where domain-specific training data is scarce, researchers can leverage pre-trained models from mismatched domains or truncated generators, provided the measurement process is sufficiently informative. However, in high-stakes scenarios, the limitations in low-information regimes necessitate rigorous validation and uncertainty checks.