Coarse-Guided Visual Generation via Weighted h-Transform Sampling

This paper proposes a novel training-free method for coarse-guided visual generation that leverages the h-transform to steer diffusion model sampling toward high-fidelity outputs from low-quality references, utilizing a noise-level-aware schedule to balance guidance adherence with synthesis quality without requiring knowledge of the forward transformation operator.

The Gist

Imagine you have a blurry, low-resolution photo of a beautiful landscape, or a video that looks like it was filmed through a wobbly window. You want to turn that "coarse" (rough) image into a crystal-clear, high-definition masterpiece.

This paper introduces a new, clever way to do that without needing to train a massive new AI model from scratch. Instead, it uses a smart "guidance system" to steer an existing AI toward the perfect result.

Here is the breakdown using simple analogies:

1. The Problem: The "Blind Artist" vs. The "Rough Sketch"

Imagine you have a world-class artist (a pre-trained AI diffusion model) who can paint amazing pictures from scratch.

• The Old Way (Training): To make this artist paint a specific scene based on your rough sketch, you'd have to hire them, show them thousands of "rough sketch vs. perfect painting" pairs, and spend months teaching them. This is expensive and slow.
• The "Inverse" Way: Some methods try to work backward mathematically. But they require you to know the exact recipe of how the image got blurry (e.g., "it was blurred by a specific lens"). If you don't know the recipe, this method fails.
• The "Start-Point" Way: Other methods just take your rough sketch, add some noise to it, and tell the artist to start painting from there. But this is a gamble: add too much noise, and the artist forgets your sketch entirely; add too little, and the result looks messy.

2. The Solution: The "GPS for Art" (Weighted h-Transform)

The authors propose a new method called Weighted h-Transform Sampling. Think of it as giving the artist a GPS navigation system that updates every second while they paint.

• The "h-Transform" (The GPS Signal): In math, this is a tool that forces a random walk (like the AI's painting process) to end up at a specific destination.
  • The Catch: The perfect GPS signal requires knowing the final destination (the perfect image) before you even start. But that's the whole point of the task! We don't know the final image yet.
• The "Approximation" (The Best Guess): Since we don't have the perfect GPS signal, the authors use a "good enough" signal based on your rough sketch. It's like saying, "Hey artist, aim generally toward this blurry shape."
• The "Weighted" Part (The Smart Brake): Here is the genius twist. The "good enough" signal is imperfect.
  • At the beginning of the process: The AI is working with a lot of "noise" (chaos). The rough sketch is a very reliable guide here. So, the method turns the GPS volume up high.
  • At the end of the process: The AI is almost done, and the image is clear. The rough sketch is now a bad guide because it's blurry. If the AI follows the blurry sketch too strictly at the end, it ruins the fine details. So, the method gradually turns the GPS volume down, letting the AI's own artistic judgment take over for the final polish.

3. How It Works in Real Life

The paper tested this on two main tasks:

1. Fixing Images: Turning blurry photos, low-res images, or damaged (in-painted) photos into sharp, clear ones.

   • Result: Their method produced clearer, more realistic images than previous "training-free" methods, without needing to know exactly how the image got blurry in the first place.

2. Fixing Videos: Imagine a video where the camera is shaky or the perspective is warped. They used a rough, warped version of the video as a guide to generate a smooth, stable, high-quality video.

   • Result: The videos looked much more stable and followed the intended camera movement better than other methods.

The Big Takeaway

Think of this method as a smart steering wheel for AI image generation.

• Old methods either required a manual (training data) or a perfect map (known math formulas).
• This method says: "I'll give you a rough map (the coarse image). I'll steer you hard toward it when you're far off course, but I'll let you take the wheel when you're close to the finish line."

This allows anyone to use powerful, pre-existing AI models to fix or improve their own messy images and videos instantly, without needing a supercomputer or a team of researchers to train a new model.

Technical Summary

1. Problem Statement

The paper addresses the challenge of Coarse-Guided Visual Generation. The goal is to synthesize high-fidelity ("fine") visual samples (images or videos) from degraded, low-fidelity, or incomplete "coarse" references (e.g., blurred images, low-resolution inputs, warped videos, or incomplete masks).

Existing approaches suffer from three main limitations:

1. Training-Based Methods: Require paired coarse-fine data for specific tasks. This incurs high training costs and lacks generalization to new degradation types.
2. Inverse Problem Solvers (Training-Free): Rely on knowing the exact forward operator (e.g., bicubic downsampling, Gaussian blur) to approximate the posterior distribution. This limits robustness when the degradation process is unknown or complex.
3. Start-Guided Synthesis (e.g., SDEdit): Injects noise into the coarse sample and denoises it. This creates an unstable trade-off: too much noise loses the guidance signal, while too little noise yields poor quality improvements.

2. Methodology: Weighted h-Transform Sampling

The authors propose a training-free, operator-free method based on Doob's h-transform, a mathematical tool used to constrain stochastic processes to reach a specific target state.

Core Concept

The method modifies the sampling trajectory of a pre-trained diffusion model (or flow matching model) to steer the generation toward the ideal fine sample ($y$) using the coarse sample ($\tilde{y}$) as a guide.

Step 1: Theoretical Formulation (h-Transform)
In standard diffusion sampling, the reverse process is governed by a Stochastic Differential Equation (SDE) or Ordinary Differential Equation (ODE). To guarantee the process ends at a specific target $y$, Doob's h-transform adds a "drift" term to the original equation:
$$d\mathbf{x} = [\mathbf{f}(\mathbf{x}_t, t) - g^2(t)\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t) - g^2(t) h_{\mathbf{x}_0=y}] dt + g(t)d\bar{\mathbf{w}}$$
Here, $h_{\mathbf{x}_0=y} = \nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_0 = y | \mathbf{x}_t)$ acts as a traction force pulling the sample toward $y$. However, $y$ (the ground truth) is unknown, making this term untractable.

Step 2: Tractable Approximation
Since $y$ is unknown, the authors approximate the untractable $h_{\mathbf{x}_0=y}$ using the known coarse sample $\tilde{y}$:
$$h_{\mathbf{x}_0=y} \approx h_{\mathbf{x}_0=\tilde{y}}$$
Using Bayes' rule and the properties of the forward diffusion process (assuming a Gaussian conditional distribution), they derive a closed-form solution for $h_{\mathbf{x}_0=\tilde{y}}$:
$$h_{\mathbf{x}_0=\tilde{y}} = \frac{1}{\sigma_t^2}(\alpha_t \tilde{y} - \mathbf{x}_t) - s_\theta(\mathbf{x}_t, t)$$
Where $s_\theta$ is the pre-trained score predictor, and $\alpha_t, \sigma_t$ are noise schedule parameters.

Step 3: Approximation Error Analysis
The authors analyze the Euclidean distance between the true drift ($h_{\mathbf{x}_0=y}$) and the approximated drift ($h_{\mathbf{x}_0=\tilde{y}}$). They find that the approximation error ($\mathcal{J}$) is negatively correlated with the noise level ($\sigma_t$):

• High Noise ($\sigma_t \to 1$): The error is small; the coarse sample $\tilde{y}$ is a good proxy for the target $y$.
• Low Noise ($\sigma_t \to 0$): The error grows significantly; relying solely on $\tilde{y}$ leads to artifacts or deviation from the true structure.

Step 4: Weighted h-Transform Sampling
To mitigate the error at low noise levels, the authors introduce a noise-level-aware weight function $\lambda_\sigma$. The final sampling ODE becomes:
$$d\mathbf{x} = \left[ \mathbf{f}(\mathbf{x}_t, t) - \frac{1}{2}g^2(t) \left( s_\theta + \lambda_\sigma \cdot h_{\mathbf{x}_0=\tilde{y}} \right) \right] dt$$

• When noise is high (early steps), $\lambda_\sigma \approx 1$, strongly incorporating the coarse guidance.
• When noise is low (late steps), $\lambda_\sigma \to 0$, reducing the influence of the potentially erroneous approximation to preserve high-quality synthesis.
• The weight function is defined as $\lambda_\sigma = \sigma_t^\alpha$, where $\alpha$ is a hyperparameter controlling the balance.

3. Key Contributions

1. Novel Framework: Proposes Weighted h-Transform Sampling, the first training-free method that leverages Doob's h-transform for coarse-guided generation without requiring knowledge of the forward degradation operator.
2. Error-Aware Design: Theoretically derives the relationship between approximation error and noise level, introducing a dynamic weighting schedule to balance guidance adherence and generation quality.
3. Generalization: Demonstrates compatibility with both Score-based models (e.g., DDPM, SDE) and Flow Matching models (e.g., Rectified Flow), making it applicable to a wide range of modern generative architectures.
4. Training-Free: Eliminates the need for collecting paired datasets or fine-tuning models, relying solely on pre-trained unconditional or text-to-visual models.

4. Experimental Results

The method was evaluated on diverse image and video tasks:

• Image Restoration (FFHQ dataset):
  • Tasks: Super-resolution, Inpainting, Gaussian Deblur, Motion Deblur.
  • Baselines: Compared against inverse-problem solvers (DPS, ILVR, etc.) and Start-Guided methods (SDEdit).
  • Results: Outperformed SDEdit across most metrics (FID, LPIPS) and was competitive with state-of-the-art operator-dependent methods (like DPS) without knowing the forward operator.
• Camera-Controlled Video Generation (DL3DV dataset):
  • Task: Generating videos following specific camera motions using a coarse warped video as guidance.
  • Baselines: Compared against training-based (GWTF) and training-free (TTM) methods.
  • Results: Achieved the best performance in motion consistency (Optical Flow error) and structural similarity (FVD, MSE), generating videos with better alignment to ground truth than baselines.
• Ablation Studies:
  • Confirmed that the weight hyperparameter $\alpha$ is critical; too low causes quality degradation (high error), while too high reduces guidance fidelity. An optimal balance was found around $\alpha=5$.
• Compatibility: Successfully applied to Wan2.2 (a Flow Matching model) and CogVideoX (a Score-based model), proving the method's model-agnostic nature.
• Image Editing: Extended to text-based image editing, showing competitive results against inversion-based editing methods.

5. Significance

This paper provides a unified, robust, and theoretically grounded solution for conditional visual generation. By decoupling the guidance mechanism from the need for specific degradation operators or paired training data, it significantly lowers the barrier for applying diffusion models to real-world restoration and editing tasks. The introduction of the noise-level-aware weighting mechanism offers a new paradigm for handling approximation errors in guided sampling, potentially influencing future research in conditional generation and inverse problems.