FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing

FREE-Edit is a zero-shot image-driven video editing framework based on rectified flow models that utilizes an Editing-aware (REE) injection method to dynamically modulate attention injection intensity according to an optical flow-warped editing mask, thereby avoiding semantic conflicts and producing higher-quality results without fine-tuning.

The Gist

Imagine you have a home video of your family walking down a street. You want to edit the very first frame: maybe you want to swap your dad's boring gray jacket for a flashy superhero cape, or replace a boring gray car with a bright red sports car.

The goal of FREE-Edit is to take that single edited first frame and magically apply that change to the rest of the video, while keeping everything else (like the people walking, the cars moving, and the background scenery) exactly the same.

Here is the problem the paper solves, explained simply:

The Problem: The "Copy-Paste" Glitch

Existing video editing tools try to do this by looking at the original video and the new edited frame, then trying to blend them together. But they often use a "one-size-fits-all" approach.

Think of it like a bad photocopier.

• If you tell the photocopier to "copy the whole page," it copies the superhero cape and the background, which ruins the video because the background shouldn't change.
• If you tell it to "don't copy anything," the video loses its natural movement, and the people look like they are floating on a static image.

The old methods (called "vanilla injection") were like a photocopier that couldn't tell the difference between the part you wanted to change (the cape) and the part you wanted to keep (the street). This resulted in weird glitches where the superhero cape might suddenly disappear, or the street might start morphing into something else.

The Solution: The "Smart Mask" (REE Injection)

The authors of this paper created a new method called FREE-Edit (which stands for Free Rectified Edit, but you can think of it as Free Real Editing).

They introduced a clever trick called REE (Editing-awaRE) Injection. Here is how it works using a creative analogy:

1. The "Edit Map" (The Mask)

First, the computer looks at the original first frame and your new edited first frame. It draws a digital "mask" (like a stencil) over the parts that changed.

• Example: The mask covers the jacket but leaves the face and the background alone.

2. The "Flow Tracker" (Optical Flow)

The video is moving! The jacket moves as the person walks. The computer uses a technology called Optical Flow (which is like a super-smart tracker) to follow that mask through the entire video.

• Analogy: Imagine painting a red dot on a moving car. As the car drives, the red dot moves with it. The computer tracks that "red dot" (the edited area) frame by frame so it knows exactly where the edit is at any given second.

3. The "Smart Switch" (Modulation Weight)

This is the magic part. The computer creates a switch for every single pixel in the video.

• If the pixel is in the "Edit Zone" (the jacket): The switch is turned OFF. The computer says, "Do not touch this! Keep the new superhero cape exactly as you drew it."
• If the pixel is in the "Safe Zone" (the street): The switch is turned ON. The computer says, "Copy the movement and texture from the original video so the street looks natural."

Why is this better?

• Old Way: Like trying to paint over a moving wall with a roller that covers everything. You either mess up the wall or the paint doesn't stick.
• FREE-Edit: Like having a team of artists where one person holds a stencil perfectly over the moving wall, and another person paints only the exposed parts. The result is perfect: the wall moves naturally, but the new paint stays exactly where you put it.

The Result

The paper shows that this method works without needing to retrain the AI (it's "zero-shot," meaning it's ready to use immediately).

• Motion: The people and cars move naturally because the computer didn't mess with the background.
• Consistency: The superhero cape stays on the person the whole time and doesn't glitch out.
• Speed: Because it uses a fast model, it can edit videos in real-time.

In a nutshell: FREE-Edit is a smart video editor that knows exactly what to change and what to leave alone, using a digital tracking system to ensure your edits look natural and stay put, even when the video is moving.

Technical Summary

1. Problem Statement

Image-driven video editing aims to propagate modifications made to the first frame of a video (e.g., changing an object's color, removing an object, or adding a new one) to the subsequent frames while preserving the original motion and layout of the source video.

Current state-of-the-art methods typically follow an "inversion-then-editing" pipeline using pre-trained Image-to-Video (I2V) models. They invert the source video into noise and then guide the denoising process using the edited first frame. To preserve motion, these methods often employ vanilla feature injection, where internal representations (Query/Key attention maps) from the source video's reconstruction process are injected into the editing process.

Key Challenges Identified:

• Vanilla Injection Flaws: Using a uniform injection intensity for all tokens leads to sub-optimal results.
  • Excessive Injection: Injecting features from the source video into areas that were edited in the first frame causes semantic conflicts and "content leakage" (the edited object reverts to its original appearance).
  • Insufficient Injection: Failing to inject enough source features results in the loss of original motion dynamics.
• Rectified Flow (RF) Gap: While Rectified Flow models (like LTX-Video) have shown superior performance in I2V generation compared to traditional diffusion models, their application to zero-shot image-driven video editing remains largely unexplored.

2. Methodology: FREE-Edit

The authors propose FREE-Edit, a training-free framework built upon pre-trained Rectified Flow (RF) I2V models. The core innovation is the Editing-awaRE (REE) feature injection method.

A. The "Inversion-then-Editing" Pipeline

1. Inversion: The source video is inverted into a noisy latent space ($z_1$) using the pre-trained RF model, conditioned on the original first frame ($X_1$).
2. Editing: The model generates the edited video ($\tilde{z}_0$) by denoising from $z_1$, conditioned on the edited first frame ($\hat{X}_1$).

B. Editing-awaRE (REE) Injection

Instead of blindly replacing attention maps, FREE-Edit adaptively modulates the injection intensity for each token based on whether that token belongs to an editing area or a non-editing area.

1. Mask Generation:
   • Compute the pixel difference between the source first frame ($X_1$) and the edited first frame ($\hat{X}_1$).
   • Apply a threshold to create a binary editing mask ($M_1$), identifying pixels that have been modified.
2. Mask Propagation via Optical Flow:
   • Since the video has temporal compression, the mask must be propagated to all subsequent frames.
   • The authors use Optical Flow (RAFT) on the source video to warp the initial mask $M_1$ across the entire video sequence. This creates a sequence of tracking masks ($M$) that follow the motion of the edited regions.
   • Note: Occluded areas are treated as non-editing areas (value 0) to prevent artifacts.
3. Modulation Weight ($\lambda$):
   • The masks are downsampled and flattened to match the token dimension of the transformer blocks.
   • A modulation weight $\lambda$ is calculated as the inverse of the mask: $\lambda = 1 - M_{\text{compressed}}$.
   • Logic:
     • If a token is in an editing area ($M=1$), then $\lambda = 0$. No injection occurs, preserving the new edit.
     • If a token is in a non-editing area ($M=0$), then $\lambda = 1$. Full injection occurs, preserving the original motion and layout.
4. Adaptive Injection Formula:
   The Query ($Q$) and Key ($K$) vectors in the editing trajectory are linearly interpolated with those from the reconstruction trajectory:
   $$\bar{Q} = \lambda Q + (1 - \lambda) \tilde{Q}$$
   $$\bar{K} = \lambda K + (1 - \lambda) \tilde{K}$$
   Where $\tilde{Q}, \tilde{K}$ are from the editing branch and $Q, K$ are from the source reconstruction branch.

3. Key Contributions

1. FREE-Edit Framework: A novel, training-free image-driven video editing framework utilizing pre-trained Rectified Flow models (specifically LTX-Video).
2. REE Injection Mechanism: A method that adaptively modulates feature injection intensity using a modulation weight $\lambda$. It ensures that edited regions are protected from source feature leakage while non-edited regions retain original motion fidelity.
3. Optical Flow-Based Mask Tracking: A robust technique to propagate the first-frame editing mask throughout the video using optical flow, ensuring precise spatial alignment of injection suppression.
4. State-of-the-Art Performance: Demonstrates superior results in both editing consistency (appearance) and temporal coherence (motion) compared to existing diffusion-based and RF-based methods.

4. Experimental Results

The authors evaluated FREE-Edit on a custom benchmark (I2V-Edit-Bench) containing 60 videos with various editing types (object replacement, style transfer, adding/removing objects).

• Quantitative Metrics:
  • CLIP Score (Editing Fidelity): FREE-Edit achieved the highest score (0.937), indicating strong alignment with the edited first frame.
  • Warp Error (Temporal Consistency): FREE-Edit achieved the lowest error (0.595), significantly outperforming baselines (e.g., AnyV2V: 2.190, I2VEdit: 2.886).
  • Non-Editing Area Preservation: FREE-Edit scored highest on SSIM, PSNR, and LPIPS for areas not edited, proving it successfully preserves the original video content.
• Qualitative Results:
  • Vs. Vanilla Injection: Vanilla injection caused "content leakage" where edited objects reverted to their original state. FREE-Edit maintained the edit perfectly.
  • Vs. No Injection: Methods without injection failed to preserve motion, resulting in jittery or static videos. FREE-Edit maintained smooth, natural motion.
  • Vs. SOTA (Go-with-the-Flow, VideoShop, etc.): FREE-Edit produced more stable and visually coherent results, particularly in complex scenarios like adding objects or style transfer.
• User Study: In a two-alternative forced-choice test, FREE-Edit was preferred over all competing methods in both "Editing Consistency" and "Overall Quality" with win rates significantly above 50%.
• Efficiency: Leveraging the real-time capabilities of LTX-Video-2B, FREE-Edit is significantly faster than methods requiring fine-tuning (e.g., I2VEdit takes ~2 hours for a short clip; FREE-Edit takes ~1.5 minutes).

5. Significance and Limitations

Significance:

• Bridges the Gap: Successfully adapts Rectified Flow models for video editing, showing they are viable and potentially superior to diffusion models for this task.
• Training-Free: Eliminates the need for resource-intensive fine-tuning or training, making high-quality video editing accessible and efficient.
• Precision: The REE injection solves the fundamental trade-off between preserving motion and applying edits, a long-standing issue in video editing.

Limitations:

• New Object Motion: The method cannot generate natural motion for newly added objects (e.g., a turtle added to a scene) because the source video contains no motion trajectory for them. The added object may appear static or move unnaturally. This is identified as a limitation of current inversion-based approaches generally.
• Dependency on Optical Flow: The quality of mask propagation relies on the accuracy of the optical flow estimation, which can struggle with complex occlusions or rapid motion.

In conclusion, FREE-Edit represents a significant advancement in zero-shot video editing by introducing an adaptive, editing-aware injection mechanism that effectively balances appearance modification with motion preservation using Rectified Flow models.