Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning

1. Problem Definition

The paper addresses the limitations of current Text-to-Video (T2V) and Text-Image-to-Video (TI2V) generation models.

• Current Limitations: Existing TI2V methods typically rely on fine-tuning foundation models with paired text, image, and video data. This is computationally expensive, requires massive datasets, and risks introducing data bias. Furthermore, these models are rigid; they only support pre-defined conditioning settings (e.g., image animation where the image is the first frame, or frame interpolation where images are the first and last frames).
• The Goal: The authors define a generic TI2V problem where a model must synthesize a video conditioned on an arbitrary number of images ($N$) placed at arbitrary positions ($p$) within the video sequence, guided by a textual prompt. This unifies tasks like image animation, video rewinding, outpainting, and frame interpolation into a single framework.

2. Methodology: FlexTI2V

The proposed solution, FlexTI2V, is a training-free approach that modifies the inference process of off-the-shelf T2V diffusion models (both UNet and Transformer-based) without updating model weights. It operates in the latent space of the diffusion model.

The method consists of three core stages:

A. Condition Image Inversion and Initialization

1. Inversion: The input condition images are encoded into latent representations and then "inverted" into noisy representations ($\tilde{x}_t$) at every denoising step $t$ using a DDPM-based inversion process.
2. Noise Initialization: The final noisy representation of the condition image ($\tilde{x}_T$) is duplicated and stacked along the temporal dimension to serve as the initial noise ($z_T$) for the video generation. This preserves the global structure of the condition image.

B. Frame Replacement

At specific time steps corresponding to the desired positions of the condition images ($p$), the video frame embeddings in the latent space are directly replaced with the inverted condition image embeddings ($\tilde{x}_t$). This ensures the condition images appear exactly at the specified frames.

C. Random Patch Swapping (RPS)

This is the key innovation of the paper. To prevent the visual features of the condition images from being lost during the temporal propagation of the diffusion model, the method explicitly injects visual cues into every video frame.

• Mechanism: At each denoising step, a random portion of patches (spatial elements) in the current video frame embedding is swapped with patches from the nearest condition image embedding.
• Dynamic Control: To balance fidelity (sticking to the image) and creativity (generating smooth motion), the swapping percentage ($P$) and the number of steps ($t$) are dynamically adjusted based on the distance between the current frame and the condition image:
  • Near the condition image: High swapping percentage and more steps are used to enforce strong visual alignment.
  • Far from the condition image: Swapping is reduced or stopped to allow the model to generate creative motion and transitions without being "frozen" by the static image features.
• Mathematical Formulation: The swap is controlled by a binary mask generated based on a decay function relative to the frame index and the condition image position.

3. Key Contributions

1. Unified Problem Formulation: The paper formally defines a generic TI2V task that subsumes existing specific tasks (animation, interpolation, rewinding, outpainting) into a single flexible framework supporting arbitrary $N$ and $p$.
2. Training-Free Flexibility: FlexTI2V introduces a plug-and-play method that works with pre-trained T2V models without fine-tuning, eliminating the need for expensive retraining and large datasets.
3. Random Patch Swapping (RPS): A novel strategy to inject local visual features into the denoising process, effectively harmonizing the distinct denoising paths of text and image conditions.
4. Dynamic Control Strategy: A mechanism to adjust the strength of visual conditioning based on frame proximity, preventing video "freezing" while maintaining high fidelity.

4. Experimental Results

The authors evaluated FlexTI2V on the UCF-101 dataset and a synthetic dataset, comparing it against state-of-the-art training-based models (e.g., DynamiCrafter) and training-free baselines (e.g., TI2V-Zero, TRF).

• Quantitative Performance:
  • FVD (Fréchet Video Distance): FlexTI2V achieved significantly lower FVD scores (indicating higher video quality) across all tasks (Animation, Rewinding, Outpainting, Interpolation). For example, in Image Animation, it improved FVD from 210.77 (DynamiCrafter) to 125.49.
  • CLIP Metrics: It outperformed baselines in CLIP-T (text-video alignment) and CLIP-I (image-video similarity), demonstrating better adherence to both prompts and visual conditions.
  • User Study: Human evaluators preferred FlexTI2V videos over baselines in the majority of cases for alignment with prompts and images.
• Efficiency:
  • FlexTI2V is highly efficient, taking only 2.71 seconds to generate a 16-frame video on an NVIDIA H100 GPU.
  • In contrast, autoregressive methods like TI2V-Zero took 275.14 seconds due to sequential frame generation and multiple resampling steps.
• Generalization: The method successfully generalized to different architectures (UNet-based ModelScopeT2V and Transformer-based Wan2.1) and various conditioning settings (1 to 3 images at arbitrary positions).

5. Significance and Impact

• Democratization of Video Generation: By removing the need for fine-tuning, FlexTI2V lowers the barrier to entry for custom video generation, allowing users to adapt powerful foundation models to specific visual control needs with minimal computational resources.
• Flexibility: It solves the "rigidity" problem of current models, enabling complex scenarios like inserting an object in the middle of a video or rewinding a scene, which were previously impossible without retraining.
• Foundation for Future Work: The paper highlights that training-free approaches can achieve superior performance to specialized fine-tuned models in terms of generalization and efficiency, suggesting a new direction for controllable video generation research.

Limitations noted: The method still struggles with complex camera viewpoint transitions between condition images and inherits biases (like watermarks) from the underlying foundation model.