SteerFlow: Steering Rectified Flows for Faithful Inversion-Based Image Editing

1. Problem Statement

The paper addresses the challenge of faithful image editing using Rectified Flow (RF) models. The goal is to edit a source image based on a new text prompt while strictly preserving the original image's structure, pose, and background.

Existing inversion-based methods (which map an image to a latent noise vector and regenerate it) suffer from two primary issues that degrade source fidelity:

1. Inversion Error: Discretizing the Ordinary Differential Equation (ODE) during the forward inversion process introduces truncation errors. This causes a mismatch between the forward trajectory (inversion) and the backward trajectory (generation), leading to imperfect reconstruction of the source image.
2. Trajectory Divergence: Even with a perfect latent, the velocity fields required to generate the target image differ from those required to reconstruct the source. This causes the editing trajectory to drift away from the original structure, resulting in over-editing or loss of background details.

Current solutions attempt to fix these via:

• Higher-order solvers: Reduce error but double the computational cost (NFEs).
• Truncated inversion: Stops the forward process early to preserve structure, but this is heuristic, sensitive to the truncation point, and limits editability (under-editing).
• Feature injection: Heuristically injects attention maps from the forward pass, but lacks architectural transferability and requires manual tuning.

2. Methodology: SteerFlow

SteerFlow is a model-agnostic framework that provides theoretical guarantees on source fidelity. It operates in two main stages:

A. Forward Process: Amortized Fixed-Point (AFP) Solver

To minimize inversion error, SteerFlow aims to "straighten" the forward trajectory by enforcing velocity consistency.

• Mechanism: Instead of standard Euler steps, it solves for a velocity $v^*_i$ that satisfies an implicit fixed-point condition: $v^*_i = v_\theta(Z_{t_i} + \Delta t \cdot v^*_i, t_{i+1})$.
• Amortization: Solving this iteratively for every timestep is expensive. The authors observe that the "target" velocity shifts if refined at every step (the "moving target" problem).
• Solution: They apply $K$ iterations only at the first timestep to strictly minimize initial velocity mismatch. For all subsequent timesteps, they perform a single iteration using the previous step's velocity as a "warm start."
• Benefit: This significantly reduces the forward-backward velocity mismatch with minimal computational overhead (only $K-1$ extra NFEs total), yielding a high-fidelity inverted latent.

B. Backward Process: Trajectory Interpolation

To prevent trajectory divergence during generation, SteerFlow avoids hard truncation or feature injection.

• Mechanism: It constructs the editing velocity by adaptively blending the target-conditioned velocity ($V^{tar}$) and the source-reconstruction velocity ($V^{src}$).
  $$V^{edit}_t = V^{src}_t + \alpha_t \cdot (V^{tar}_t - V^{src}_t)$$
• Adaptive Scheduler ($\alpha_t$): The blending factor $\alpha_t$ is not constant. It is defined as:
  $$\alpha_t = \text{CosSim}(V^{src}_t, V^{tar}_t) \cdot (1 - t^\gamma)$$
  • Time Decay: Early in the generation process ($t \approx 1$), $\alpha_t$ is low, anchoring the trajectory to the source to preserve structure. As $t \to 0$, $\alpha_t$ increases, allowing the image to follow the target prompt.
  • Directional Awareness: The Cosine Similarity term ensures that if the target velocity aligns with the source (structurally safe), the edit is allowed; if they diverge, the source anchor is strengthened.
• Theoretical Guarantee: The authors prove that this interpolation reduces the editing error bound to $\alpha \cdot \mathcal{B}^{tar}_0$, ensuring tighter bounds than standard Euler integration.

C. Adaptive Masking Mechanism

To address "editing leakage" (where background changes unintentionally), SteerFlow introduces a spatial constraint.

• Base Mask: Uses SAM3 (Segment Anything Model 3) guided by source editing tokens to create an initial mask.
• Refinement: Since SAM3 masks can be too restrictive for structural changes, SteerFlow refines the mask using the velocity difference ($\Delta V = V^{tar} - V^{src}$).
• Process: The velocity difference is normalized and combined with the SAM3 mask via morphological operations. This creates a dynamic mask that expands to necessary structural changes while strictly gating the editing velocity in the background.

3. Key Contributions

1. Theoretical Analysis: Formal derivation of error bounds for Inversion Error and Trajectory Divergence, establishing the mathematical motivation for the proposed framework.
2. SteerFlow Framework:
   • AFP Solver: An efficient, amortized fixed-point solver that implicitly straightens the forward trajectory, reducing inversion error without doubling NFEs.
   • Trajectory Interpolation: A principled backward process that adaptively anchors the editing path to the source, minimizing drift.
   • Adaptive Masking: A mechanism combining concept-guided segmentation with velocity-driven refinement for precise localization.
3. Multi-Turn Editing: The framework naturally extends to complex, multi-turn editing paradigms without accumulating drift, by updating the source trajectory at each turn.

4. Experimental Results

The authors evaluated SteerFlow on FLUX.1-dev and Stable Diffusion 3.5 Medium using the PIE-Bench dataset.

• Quantitative Performance:
  • SteerFlow achieved the best overall rankings across Source Preservation (DinoDist, LPIPS, SSIM) and Target Alignment (CLIP, ImageReward).
  • It outperformed state-of-the-art inversion-based methods (UniEdit, FireFlow, RF-Solver) and inversion-free methods (FlowEdit, FlowAlign).
  • Specifically, it balanced source preservation and editability better than UniEdit (which often under-edits) and FireFlow (which often over-edits).
• Image Reconstruction: SteerFlow demonstrated strong cross-architecture transferability, ranking consistently high (2nd place) on both models for reconstruction tasks, proving the efficacy of the AFP solver.
• Qualitative Results: Visual comparisons showed SteerFlow successfully preserved complex structures (poses, furniture, backgrounds) while accurately applying target prompts, whereas baselines suffered from structural collapse or background artifacts.
• Multi-Turn Editing: Demonstrated the ability to perform sequential edits (e.g., change object -> change attribute -> add object) without the cumulative drift seen in other methods.

5. Significance

SteerFlow represents a significant shift from heuristic-based editing (relying on manual truncation or feature injection) to principled, theory-driven editing.

• Efficiency: It achieves superior fidelity without the computational penalty of higher-order solvers.
• Generalizability: Being model-agnostic, it works effectively across different Rectified Flow architectures (FLUX, SD3.5).
• Robustness: It solves the fundamental issue of trajectory divergence, enabling reliable multi-turn editing, which is crucial for iterative creative workflows.
• Theoretical Grounding: By providing error bounds and proving convergence properties, it offers a rigorous foundation for future research in flow-based image editing.