🤖 AI

InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting

InverFill is a one-step inversion method that injects semantic information from masked images into the initial noise of few-step diffusion models, enabling high-fidelity, artifact-free inpainting without requiring model retraining or real-image supervision.

Original authors: Duc Vu, Kien Nguyen, Trong-Tung Nguyen, Ngan Nguyen, Phong Nguyen, Khoi Nguyen, Cuong Pham, Anh Tran

Published 2026-03-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Duc Vu, Kien Nguyen, Trong-Tung Nguyen, Ngan Nguyen, Phong Nguyen, Khoi Nguyen, Cuong Pham, Anh Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful photo, but someone has scribbled a black marker over a part of it. You want to fix it. You ask an AI, "Fill in this blank space with a dog sitting on the grass."

In the past, AI artists were like slow, meticulous painters. They could create a perfect dog, but it took them 50 or 100 brushstrokes (steps) to get there. If you asked them to do it in just 5 strokes, they would panic. They'd start with a blank, random canvas, guess wildly, and end up with a blurry mess that didn't match the rest of the picture. The dog might look like a potato, or the grass might be floating in the sky.

InverFill is a new tool that teaches these "fast painters" how to do a perfect job in just a few strokes.

Here is how it works, using some simple analogies:

1. The Problem: The "Blank Canvas" Mistake

Normally, when an AI tries to fix a photo, it starts with a canvas covered in static noise (like the "snow" on an old TV).

The Slow Way: If the AI has 50 steps, it can slowly turn that static into a dog, adjusting its shape and color as it goes. It can look at the surrounding grass and say, "Oh, the dog needs to be green-tinted by the grass."
The Fast Way: If the AI only has 2 or 4 steps, it doesn't have time to fix its mistakes. If it starts with random static, it guesses wrong immediately, and there's no time to correct it. The result is a glitchy, mismatched patch.

2. The Solution: The "Smart Blueprint" (Inversion)

Instead of starting with random static, InverFill acts like a smart blueprint generator.

Before the AI even starts painting, InverFill looks at the existing parts of your photo (the parts not covered by the marker). It asks: "What kind of noise would create this specific background?"

It then creates a customized starting point (a "noise latent") that already "knows" the texture, lighting, and style of your photo.

Analogy: Imagine you are building a Lego castle.
- Old Way: You dump a bucket of mixed-up Legos on the table and try to build a tower in 5 seconds. It falls apart.
- InverFill Way: You look at the base of the castle that's already built. You sort the Legos before you start, so you only have the exact red bricks you need for the next layer. Now, you can build the rest of the tower in 5 seconds, and it fits perfectly.

3. The Secret Sauce: "Re-Blending" and "Gaussian Regularization"

The paper mentions two clever tricks to make this work:

Re-Blending (The "Don't Leak" Rule):
When InverFill looks at the photo to make its blueprint, it has to be careful. It shouldn't accidentally "leak" information from the visible parts into the hidden parts.
- Analogy: Imagine you are trying to guess what's inside a wrapped gift by looking at the box. You don't want to accidentally see the gift through a hole in the paper! InverFill has a special filter that says, "Okay, I know what the box looks like, but for the part I can't see, I'll just put in some random, neutral noise so I don't cheat."
Gaussian Regularization (The "Standard Shape" Rule):
AI models are trained to expect noise to look a specific way (like a perfect bell curve). If the blueprint InverFill makes looks too weird, the AI gets confused.
- Analogy: Think of the AI as a chef who only knows how to cook with perfectly round eggs. If you hand them a square egg, they burn it. InverFill makes sure the "noise egg" it creates is perfectly round and standard, so the chef (the AI) can cook it perfectly without burning it.

4. The Result: Speed Without Sacrifice

The paper shows that InverFill adds almost zero time to the process (about 0.06 seconds, which is less than a blink of an eye).

Before: You had to choose between High Quality (slow, 30 steps) or Fast (low quality, 2 steps).
With InverFill: You get High Quality in 2 steps.

It allows fast AI models (like SDXL-Turbo or SANA-Sprint) to fix photos as well as the slow, specialized experts, without needing to be retrained or taught new skills. It's like giving a sprinter a pair of wings; they were already fast, but now they can fly.

Summary

InverFill is a "pre-game" tool for AI image repair. Instead of letting the AI guess randomly, it gives the AI a smart, pre-sorted starting point that matches the rest of the photo. This lets the AI finish the job in a flash, with perfect harmony and no blurry messes.

1. Problem Statement

Recent advancements in diffusion models have achieved photorealistic image inpainting but typically require dozens to hundreds of sampling steps (NFEs), resulting in high latency and limiting real-time applications. While few-step text-to-image models (e.g., SDXL-Turbo, SANA-Sprint) offer rapid generation, naively applying them to inpainting yields poor results.

The core issue identified is semantic misalignment caused by random Gaussian noise initialization.

Multi-step models can gradually correct the mismatch between random noise and the image context over many iterations.
Few-step models perform large, discrete updates. Starting from random noise leads to a semantic gap between the initial latent and the unmasked background, causing artifacts, poor harmonization, and blurry regions when using standard Blended Sampling (iteratively merging predictions with the unmasked image).

Existing few-step inpainting solutions (e.g., TurboFill) often require complex adversarial training, real-image supervision, and heavy computational overhead, making them difficult to generalize.

2. Methodology: InverFill

The authors propose InverFill, a training-free, one-step inversion network designed to generate semantically aligned initial noise latents for few-step inpainting. Instead of starting with random noise, InverFill maps the masked input image directly to a noise latent that preserves background semantics while remaining compatible with the diffusion prior.

The method consists of four key technical components:

A. Masked Image Inversion Network

Architecture: A one-step inversion network ( $F_\theta$ ) that shares the architecture of the pretrained one-step generator (e.g., SANA-Sprint or SDXL-Turbo).
Input: Takes a masked image latent ( $z^m_0$ ) and a text prompt ( $c$ ).
Output: Predicts an inverted noise latent ( $\hat{z}_T$ ).
Training Strategy: Uses a "synthetic" training pipeline where the generator creates ground-truth images from random noise, which are then masked to create training triplets (Image, Mask, Prompt). This eliminates the need for curated real-image-mask-text datasets.

B. Re-Blending Operation

Directly training on masked images causes information leakage, where the visible background "leaks" into the predicted noise latent for the masked region, violating the Gaussian prior expected by the diffusion model.

Solution: During training and inference, the masked regions of the predicted latent $\hat{z}_T$ are replaced with fresh random Gaussian noise ( $\epsilon'$ ).
Formula: $\hat{z}^{blend}_T = \hat{z}_T \odot (1-m) + \epsilon' \odot m$ .
Effect: This prevents the model from overfitting to the visible pixels and ensures the latent structure remains consistent with the diffusion process.

C. Gaussian Regularization Loss ( $L_{reg}$ )

Even with Re-Blending, the blended latent may deviate from the standard Gaussian distribution required by the generator, leading to blurry reconstructions.

Solution: A moment-matching loss is introduced to enforce the statistical properties of a standard Gaussian distribution on the blended latent.
Mechanism: It matches the first (mean) and second (variance) moments of the predicted latent to the theoretical moments of a standard Gaussian.
Result: This ensures the inverted noise is statistically compatible with the generator, stabilizing the reconstruction.

D. Adversarial Loss

To further enhance visual fidelity, the method employs an adversarial training objective (following LADD) using a frozen multi-step teacher model as a feature extractor. This helps the inversion network produce latents that result in higher-quality image reconstructions.

3. Key Contributions

InverFill Framework: A novel one-step inversion network specifically tailored for inpainting that replaces random initialization with semantically informed noise, enabling high-fidelity few-step inpainting.
Re-Blending Operation: A mechanism to prevent information leakage from visible regions into the noise latent during training, ensuring the latent remains valid for the diffusion process.
Gaussian Regularization: A novel loss function that aligns the inverted noise with the required Gaussian prior, significantly improving reconstruction stability and detail.
Image-Free Training: The training pipeline does not require real-image-mask-text triplets; it synthesizes its own training data on the fly using the generator, making it highly scalable.
Plug-and-Play Compatibility: The method works with existing few-step text-to-image models (SDXL-Turbo, SANA-Sprint) without any task-specific fine-tuning of the generator.

4. Experimental Results

The authors evaluated InverFill on BrushBench and MagicBrush benchmarks using SANA-Sprint 0.6B and SDXL-Turbo.

Performance Gains: InverFill consistently outperforms baseline few-step models.
- On SANA-Sprint (2 NFEs), adding InverFill improved ImageReward (IR) from 11.02 to 11.65 and HPS from 26.21 to 27.93.
- On SDXL-Turbo (4 NFEs), it improved IR from 11.42 to 12.38.
Comparison with Specialized Models: InverFill-equipped few-step models match or surpass specialized multi-step inpainting models (e.g., SDXL-Inpainting with 30 steps) in quality metrics while being significantly faster.
Efficiency: The method adds negligible overhead (~0.04s to 0.06s on an NVIDIA A100 GPU).
Ablation Studies:
- Removing Re-Blending leads to structural artifacts and information leakage.
- Removing Gaussian Regularization results in blurry outputs and poor background preservation.
- The method significantly outperforms 50-step DDIM inversion baselines in both speed (0.74s vs 4.32s) and quality.

5. Significance

InverFill addresses a critical bottleneck in generative AI: speed vs. quality trade-off in inpainting.

Practicality: It enables real-time, high-quality image editing on consumer-grade hardware by reducing inference time from seconds to milliseconds without sacrificing photorealism.
Generalization: Unlike previous few-step inpainting methods that require complex, model-specific training (like TurboFill), InverFill is a lightweight, universal wrapper that enhances any existing few-step text-to-image model.
Theoretical Insight: It demonstrates that the primary failure mode of few-step inpainting is not the sampling strategy itself, but the initialization noise. By correcting the initialization via one-step inversion, the entire few-step pipeline becomes viable for complex editing tasks.