Technical Summary: SONIC (Spectral Optimization of Noise for Inpainting with Consistency)

Problem Statement

Current approaches to image inpainting using off-the-shelf text-to-image models (specifically denoising diffusion and flow models) face a trade-off between versatility and performance. While training-free methods that rely on guidance or posterior sampling allow generic models to be used for inverse problems without expensive fine-tuning, they often produce results with inconsistent image structures compared to specialized, trained inpainting models.

The authors argue that the primary limitation of existing training-free methods is the oversight of the initial noise sample. While it is well-established that the initial noise largely determines the final generation's layout and structure, directly optimizing this noise is computationally prohibitive. Standard approaches require back-propagating through the entire denoising chain (unrolling $T$ steps), which is infeasible for modern large-scale models due to memory and compute constraints. Consequently, existing methods often settle for suboptimal noise initialization, leading to structural misalignments in the inpainted regions.

Methodology

SONIC proposes a novel, training-free framework that optimizes the initial noise sample to faithfully reproduce the unmasked regions of an input image, thereby ensuring structural consistency in the final inpainting. The method avoids back-propagation through the denoiser by employing two core technical innovations:

1. Linearization of the Denoising Trajectory

To circumvent the need for unrolling the denoising chain, SONIC approximates the denoising trajectory as a linear path.

• Formulation: Instead of treating the denoising process $x_0 = D_T(x_T)$ as a complex non-linear function requiring gradient computation through every step, the authors approximate the trajectory $x(t)$ as:
  $$\hat{x}(t) \approx \text{stopgrad}(D_T(x_T) - x_T)(1 - \frac{t}{T}) + x_T$$
• Mechanism: This linearization assumes that the difference between the final denoised image and the initial noise is constant throughout the process. This allows the optimization objective to be formulated as a differentiable loss with respect to $x_T$ without back-propagating through the denoiser $D_T(\cdot)$. The loss minimizes the difference between the observed masked image $y$ and the corrupted version of the linearly approximated denoised output.

2. Spectral Preconditioning

Direct optimization of the initial noise in the spatial domain often leads to instability, where different spatial frequencies converge at different rates, causing flickering or artifacts.

• Spectral Domain Optimization: The method transforms the optimization problem into the frequency domain. The initial noise $x_T$ is represented by its Fourier transform $X_T$.
• Adam Optimization: The optimization is performed on $X_T$ using the Adam optimizer. Since Adam divides parameter updates by their second moments, operating in the spectral domain effectively acts as a preconditioner. This allows different frequency components to converge at their preferred paces, stabilizing the optimization process and preventing the noise from diverging from the acceptable manifold.
• Spatial Constraints: To prevent the unobserved (masked) regions of the noise from drifting away from the Gaussian distribution expected by the model, gradient updates are masked. Only gradients corresponding to observed (unmasked) pixels are applied.

Implementation Details

• Latent Space Efficiency: The method operates entirely in the latent space. For masked images, the authors employ a simple nearest-neighbor fill strategy before encoding to the latent space, finding this sufficient for robust performance.
• Workflow: The process involves optimizing the initial noise $X_T$ for a fixed number of iterations (e.g., 20) using the linearized loss and spectral preconditioning. The resulting optimized noise is then fed into a standard training-free inpainting pipeline (e.g., BLD-SD3.5) to generate the final image.

Key Contributions

1. Optimization of Initial Noise without Back-propagation: The paper introduces a linearization strategy that enables the optimization of the initial noise sample for inverse problems without the computational cost of unrolling the denoising chain.
2. Spectral Preconditioning: The authors demonstrate that optimizing in the spectral domain with Adam provides significantly more stable convergence than spatial optimization, addressing the frequency-dependent convergence rates inherent in diffusion/flow models.
3. Training-Free State-of-the-Art: The method achieves superior performance compared to existing training-free baselines and competitive results against specialized trained models, all without requiring additional model training or fine-tuning.

Experimental Results

The authors evaluated SONIC on three standard datasets: FFHQ (faces), DIV2K (natural images), and BrushBench (complex masks).

• Quantitative Performance: SONIC outperformed state-of-the-art training-free methods (FLAIR, FlowChef, FlowDPS, BLD-SD3.5) and the specialized trained model BrushNet on key perceptual metrics: SSIM, LPIPS, and FID.
  • On FFHQ, SONIC achieved an LPIPS of 0.121 and FID of 12.883, significantly better than FLAIR (0.266 LPIPS, 18.982 FID) and BrushNet (0.237 LPIPS, 17.950 FID).
  • On DIV2K and BrushBench, similar trends were observed, with SONIC showing the best structural consistency and perceptual similarity.
• Qualitative Results: Visual comparisons showed that SONIC produces inpainted regions that align better with the global structure, lighting, and layout of the input image compared to baselines, which often suffered from structural misalignment or blurry artifacts.
• Ablation Studies:
  • Removing spectral preconditioning (optimizing in spatial domain) led to significantly worse performance (higher LPIPS and FID), confirming the necessity of frequency-domain stabilization.
  • Removing spatial constraints caused the optimization to diverge, introducing visible artifacts and color shifts.
  • Using ground-truth pixels for the encoder (upper bound) yielded results nearly identical to the simple nearest-neighbor fill, validating the robustness of the latent space strategy.

Significance and Claims

The paper claims that SONIC addresses a critical gap in the application of generic generative models to inverse problems. By identifying the initial noise sample as the "missing ingredient" and providing a practical, stable method to optimize it, SONIC enables off-the-shelf models to achieve performance previously reserved for specialized, trained inpainting networks.

The authors emphasize that their method is training-free, avoiding the costs and generalization issues associated with fine-tuning models on specific mask types. They position SONIC not just as an inpainting tool, but as a proof-of-concept that linearization and spectral preconditioning can unlock the potential of generic flow models for a broader range of inverse problems, such as super-resolution and deblurring, without the need for complex posterior sampling or back-propagation.