Spectral Collapse in Diffusion Inversion

1. Problem Statement: Spectral Collapse

The paper addresses a critical failure mode in unpaired image-to-image translation using conditional diffusion inversion. Specifically, it focuses on tasks where the source domain is spectrally sparse (lacking high-frequency information, e.g., low-resolution images, sketches) and the target domain is spectrally dense (requiring high-frequency textures, e.g., high-resolution photos, realistic images).

• The Phenomenon: The authors identify a phenomenon they term "Spectral Collapse." When using standard deterministic inversion methods (like DDIM) to map a spectrally sparse input back to a latent noise vector ($z_T$), the resulting latent does not follow the expected isotropic Gaussian distribution. Instead, it retains low-frequency structural imprints of the input and lacks high-frequency variance.
• The Consequence: Because the recovered latent lacks the necessary stochastic variance (noise) to "seed" new details, the subsequent generative sampling process produces oversmoothed, texture-poor outputs. The model fails to hallucinate plausible high-frequency textures because the latent space has "collapsed" into a low-dimensional subspace.
• The Trade-off: Existing attempts to fix this using stochastic noise injection (e.g., TABA, ReNoise) successfully restore texture but break the semantic link to the input, causing structural drift (the generated image looks realistic but no longer matches the source structure).

2. Methodology: Orthogonal Variance Guidance (OVG)

To resolve the trade-off between structural fidelity and texture realism, the authors propose Orthogonal Variance Guidance (OVG), an inference-time method that corrects the Ordinary Differential Equation (ODE) dynamics of the diffusion model.

Core Mechanism

OVG operates by injecting high-frequency variance strictly into the null-space of the structural gradient. It ensures that texture generation does not interfere with the preservation of the input's semantic layout.

The method involves two competing objectives at each timestep $t$:

1. High-Frequency Correction (Gaussianity): A loss function ($L_{HF}$) penalizes deviations of the predicted noise energy from its theoretical expectation (concentration of measure on a hypersphere). This forces the latent to regain the variance required for realistic textures.
2. Low-Frequency Correction (Structure): A loss function ($L_{LF}$) penalizes drift in the denoised estimate relative to the low-frequency reference of the input (e.g., the sketch or low-res image). This preserves structural fidelity.

Orthogonal Projection

When the gradients of these two objectives conflict (i.e., they point in opposite directions), OVG applies a geometric projection inspired by multi-task learning (specifically PCGrad):

• It projects the high-frequency update vector onto the normal plane of the structural gradient.
• This ensures that variance is injected orthogonally to the structural features.
• Result: The model can hallucinate realistic textures (restoring Gaussianity) without corrupting the semantic layout of the input.

Parameterization Insight

The paper also highlights that the choice of training objective matters. Models trained with an $x_0$-prediction objective (predicting the clean image directly, as in EDM) are inherently more robust to spectral collapse than those using $\epsilon$-prediction (predicting the noise), though $x_0$-prediction alone still suffers from structural drift without OVG.

3. Key Contributions

1. Identification of Spectral Collapse: The authors provide a systematic empirical analysis demonstrating that standard deterministic inversion fails in spectrally sparse-to-dense translation tasks because the inverted latent retains low-frequency residuals rather than converging to pure Gaussian noise.
2. Theoretical Analysis: They prove that the collapse stems from the spectral bias of deep neural networks (ReLU/Transformers) combined with the linear assumptions of deterministic ODE solvers. In $\epsilon$-prediction regimes, the score estimator vanishes for high frequencies in low-SNR regimes, freezing texture generation.
3. Orthogonal Variance Guidance (OVG): A novel inference-time algorithm that decouples texture restoration from structural preservation by injecting variance orthogonally to the structural gradient.
4. Comprehensive Evaluation: Extensive experiments on microscopy super-resolution (BBBC021) and sketch-to-image (Edges2Shoes) across different architectures (U-Net, DiT) and spaces (Latent, Pixel).

4. Experimental Results

The method was evaluated on two primary datasets:

• BBBC021: Biological microscopy images for super-resolution (×8, ×16, ×32).
• Edges2Shoes: Sketch-to-image translation.

Key Findings:

• Superior Trade-off: OVG achieves the best Pareto frontier between High-Frequency Score (SHF) (texture realism) and Low-Frequency Score (SLF) (structural fidelity).
• Quantitative Gains:
  • On Edges2Shoes, OVG combined with EDM achieved an LPIPS of 0.381 (best perceptual quality) while maintaining an SLF of 0.60, significantly outperforming stochastic baselines like TABA (SLF 0.46).
  • On BBBC021 (×16), OVG restored realistic biological textures (nuclei, membranes) with an SHF of 0.95–0.97, whereas deterministic baselines suffered from severe oversmoothing (SHF ~0.83).
• Robustness: The method works effectively across different downscaling factors (up to ×32), different architectures (U-Net vs. DiT), and different data spaces (Latent vs. Pixel), confirming that spectral collapse is an intrinsic issue of inversion dynamics, not just a specific model artifact.

5. Significance

This work fundamentally shifts the understanding of diffusion inversion. It demonstrates that deterministic inversion is insufficient for tasks requiring the generation of new high-frequency information from sparse inputs.

• Theoretical Impact: It challenges the assumption that simply reversing the diffusion ODE is sufficient for high-fidelity editing, revealing a "spectral void" in the latent space.
• Practical Impact: OVG provides a plug-and-play solution for critical applications like medical imaging super-resolution (where hallucinating false structures is dangerous) and artistic generation (sketch-to-image), enabling the creation of photorealistic textures while strictly adhering to the user's structural constraints.

In summary, the paper solves the "structure-texture trade-off" in diffusion models by mathematically enforcing that new details are generated in a direction orthogonal to the existing structure, allowing for both high-fidelity reconstruction and realistic texture synthesis.