h-Flow: Flexible Flow-based Image Editing via Doob's h-Transform

Technical Summary of h-Flow: Flexible Flow-based Image Editing via Doob's h-Transform

Problem Statement
Text-based image editing using pre-trained text-to-image Rectified Flow (RF) models faces a fundamental trade-off between target alignment (faithfully adhering to the target prompt) and source consistency (preserving the structure and unedited regions of the original image). Existing approaches generally fall into two categories with distinct limitations:

1. Inversion-based methods: These invert the source image into a latent noise distribution before regenerating it. While they aim for high fidelity, they often rely on heuristic, architecture-specific designs and suffer from error accumulation during the inversion process.
2. Inversion-free methods: These construct direct transformation trajectories from source to target. While they enforce structural consistency, they often lack a unified theoretical foundation, relying on empirical heuristics that make them sensitive to hyperparameters (e.g., random seeds, flow steps) and prone to instability.

The core challenge is to achieve a controllable, robust balance between these two objectives without requiring additional training or relying on fragile heuristic constructions.

Methodology
The authors propose h-Flow, a training-free framework that reformulates image editing as a conditional generation problem under multiple terminal events, grounded in Doob's h-Transform.

1. Theoretical Extension to Deterministic RF:
   Classical Doob's h-Transform is defined for stochastic differential equations (SDEs) with non-zero diffusion coefficients. However, RF models operate via deterministic Ordinary Differential Equations (ODEs) where the diffusion coefficient $g(t) \equiv 0$, rendering the standard h-Transform undefined. To resolve this, the authors construct an equivalent SDE that shares the exact same marginal distributions as the RF dynamics but possesses a non-zero diffusion coefficient ($g_{eq}(t) = \sqrt{2t/(1-t)}$). This allows the rigorous application of the h-Transform to the deterministic RF framework.

2. Dual-Guidance via h-Functions:
   The editing process is guided by two harmonic functions ($h$-functions) corresponding to the two terminal constraints:

   • Source Consistency ($h_{rec}$): Encodes the constraint that the final image must match the source image $x_0^{src}$. This yields a closed-form reconstruction guidance term, $v_{rec} = (x_t - x_0^{src})/t$, which steers the trajectory exactly toward the source.
   • Target Alignment ($h_{edit}$): Encodes the semantic transition to the target prompt $c^{edit}$. The guidance signal is derived as the difference between the velocity fields of the target and source prompts ($\Delta_{edit} = v_\theta^{edit} - v_\theta^{src}$), isolating the semantic change while discarding shared content.

3. Velocity Orthogonal Decomposition:
   A key innovation is the orthogonal projection of the editing signal. Since the editing direction $\Delta_{edit}$ may contain components parallel to the reconstruction velocity $\tilde{v}_{rec}$, simply adding them can cause interference. The authors project $\Delta_{edit}$ onto the orthogonal complement of $\tilde{v}_{rec}$:
   $$\Delta_{edit}^\perp = \Delta_{edit} - \frac{\langle \Delta_{edit}, \tilde{v}_{rec} \rangle}{\|\tilde{v}_{rec}\|^2} \tilde{v}_{rec}$$
   This decouples the two objectives: the reconstruction velocity preserves the source structure, while the orthogonal editing component modifies only the semantic content. This allows for independent control via hyperparameters $\lambda_{rec}$ and $\lambda_{edit}$ without mutual degradation.

4. Plug-and-Play Compatibility:
   The backward editing ODE is agnostic to the forward initialization. h-Flow can be seamlessly combined with various forward processes (e.g., RF-Inversion, SDEdit, ODE-Inv) in a plug-and-play manner.

Key Contributions

• Theoretical Grounding: The paper introduces the first application of Doob's h-Transform to deterministic Rectified Flow models by constructing an equivalent SDE, providing a principled theoretical bridge for guided generation in ODE-based frameworks.
• Decoupled Control: By designing dedicated h-functions and employing velocity orthogonal decomposition, the method explicitly separates reconstruction and editing directions, enabling a controllable trade-off between source fidelity and target alignment.
• Training-Free Flexibility: The approach requires no additional training or gradient updates to the pre-trained model, relying solely on forward passes of the velocity network.

Experimental Results
The authors evaluated h-Flow on the PIE-Bench and PIE-Bench++ datasets, comparing it against seven baselines (including RF-Inversion, FlowEdit, SDEdit, and UniEdit).

• Quantitative Performance: h-Flow achieved the best average rank across 9 metrics (including PSNR, LPIPS, SSIM, CLIP similarity, and aesthetic scores) on both benchmarks. It significantly outperformed baselines in balancing source consistency and target alignment. For instance, while UniEdit achieved high source consistency, it suffered in target alignment; conversely, ODEInv achieved high alignment but degraded image quality. h-Flow maintained competitive source consistency while delivering top-tier alignment.
• Qualitative Performance: Visual comparisons demonstrated h-Flow's ability to execute complex edits (e.g., subject translation, localized additions) while strictly preserving background structures and unedited regions, avoiding the artifacts common in other methods.
• Ablation Studies:
  • Compatibility: Integrating h-Flow with diverse forward processes (RF-Inversion, SDEdit, ODE-Inv) consistently improved the trade-off between editability and fidelity, acting as a structural stabilizer for unstable forward methods.
  • Orthogonal Projection: Removing the projection step resulted in lower text alignment scores without improving structural fidelity, confirming that the projection effectively eliminates interference between editing and reconstruction.
  • Hyperparameters: The study showed that $\lambda_{rec}$ and $\lambda_{edit}$ control their respective objectives independently, validating the decoupling strategy.

Significance and Claims
The paper claims that h-Flow provides a unified, theoretically grounded framework for flow-based image editing that overcomes the heuristic nature and sensitivity of prior methods. By rigorously extending Doob's h-Transform to deterministic flows and decoupling the editing and reconstruction objectives via orthogonal decomposition, the method achieves robust, flexible, and faithful editing across diverse scenarios. The authors emphasize that their approach is training-free and plug-and-play, making it immediately applicable to existing RF models without architectural modifications. The work suggests that future extensions could apply this framework to other modalities like video or 3D generation.