FlowC2S: Flowing from Current to Succeeding Frames for Fast and Memory-Efficient Video Continuation

1. Problem Statement

Video Continuation is the task of generating a plausible, temporally coherent sequence of frames given an input video segment. While critical for applications like long-horizon video generation, world models, and real-time AR/VR, current state-of-the-art (SOTA) methods face significant bottlenecks:

• High Memory Cost: Conventional approaches (e.g., LTXVCondition, CausVid) condition the model on input frames and Gaussian noise of the same dimensionality as the output. This doubles the input tensor size, leading to high GPU memory consumption.
• Slow Inference: These methods typically require many Neural Function Evaluations (NFEs) to converge, making them unsuitable for near-real-time applications like AR/VR.
• Quality vs. Efficiency Trade-off: Balancing temporal coherence (smooth motion) and visual fidelity (sharp details) while maintaining low computational cost remains a challenge.

2. Methodology: FlowC2S

The authors propose FlowC2S, a novel approach that redefines the video continuation task by training a flow matching model to flow directly from current (observed) frames to succeeding frames, rather than from noise to future frames.

Core Architecture & Training Strategy

The method fine-tunes pre-trained text-to-video flow models (specifically LTXV and Wan) using three key design choices:

1. Direct Current-to-Succeeding Flow (Dimensionality Reduction):

   • Standard Approach: Input = [Conditioning Frames (t=0) + Noise (t=1)].
   • FlowC2S Approach: Input = [Conditioning Frames (t=0)]. The model learns a vector field mapping the distribution of current frames ($p_{cfc}$) directly to succeeding frames ($p_{sfc}$).
   • Impact: This eliminates the need for noise input, halving the model input dimensionality. Consequently, it reduces peak GPU memory usage by approximately 50% compared to baselines like LTXVCondition.

2. Inherent Optimal Couplings (OC):

   • Concept: In flow matching, "optimal couplings" (pairs of $x_0$ and $x_1$ that minimize transport cost) lead to straighter flow trajectories, reducing the number of sampling steps (NFEs) needed.
   • Implementation: Instead of computing computationally expensive optimal transport (OT) plans, FlowC2S uses temporally adjacent chunks from the same video as source ($x_0$) and target ($x_1$) pairs.
   • Justification: Empirical analysis using OT heatmaps shows that consecutive chunks within a video naturally align as near-optimal couplings due to semantic and temporal continuity. This creates straighter flows without extra computational overhead.

3. Target Inversion (TI):

   • Problem: Fine-tuning a pre-trained model (trained on noise-to-data) to map data-to-data creates a distribution mismatch.
   • Solution: The method employs RF-Solver Inversion to recover an inverted latent representation ($\hat{x}_1$) of the target chunk. During training, with a certain probability, the input representation is augmented with this inverted latent.
   • Benefit: This biases the model to learn the residual shift-and-scale transformation required to move from the current state to the target, significantly improving visual fidelity and reducing artifacts.

3. Key Contributions

• Memory Efficiency: A novel input design that reduces input dimensionality by 2$\times$, resulting in ~50% lower peak GPU memory usage compared to SOTA baselines.
• Inference Speed: By leveraging inherent optimal couplings, the method achieves straighter flow trajectories, allowing for high-quality generation with as few as 5 NFEs (compared to 40+ in baselines).
• Target Inversion: A technique to bridge the distribution gap in fine-tuning, enhancing visual quality and temporal coherence.
• Generalization: The model demonstrates the ability to generalize to video durations longer than those seen during training (e.g., generating 113 frames when trained on 17/41-frame chunks).

4. Experimental Results

The authors evaluated FlowC2S on OpenVid and NuScenes datasets using LTXV (2B) and Wan (1.3B) backbones.

• Quantitative Performance:

  • FID/FVD: FlowC2S outperforms SOTA methods (Vista, GEM, LTXVCondition, CausVid) in Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD).
  • Efficiency: It achieves these superior scores with 4$\times$ fewer NFEs (5 vs. 20-50) and 2$\times$ lower memory scaling (slope $k$ in memory analysis).
  • VBench Metrics: It scores higher in subject consistency, background consistency, and motion smoothness compared to CausVid.

• Qualitative Performance:

  • Generated videos exhibit detailed motion (e.g., camera zooms, object trajectories) and maintain semantic consistency without explicit temporal conditioning.
  • Ablation studies confirm that removing either Inherent OC or Target Inversion leads to visual artifacts, color shifts, or temporal incoherence.

• Ablation Insights:

  • NFE: 5–10 NFEs are sufficient for high quality; 1 NFE results in severe blur.
  • Chunk Size: Training on multi-frame chunks (17/41) is essential; single-frame training fails to capture inter-frame structure.
  • Long Continuation: The model generalizes well to longer sequences (up to 113 frames) despite being trained on shorter segments.

5. Significance and Future Work

Significance:
FlowC2S represents a paradigm shift in video continuation by moving away from the "noise-conditioning" paradigm. It demonstrates that direct flow matching between observed and future states is not only more memory-efficient but also faster and higher quality. This makes real-time, high-fidelity video generation on embedded devices (crucial for AR/VR) more feasible.

Limitations & Future Directions:

• Complex Scenes: The current dataset construction filters out abrupt scene transitions, limiting the model's ability to handle complex, chaotic motions or scene cuts.
• Variable Lengths: The model is trained on fixed-length chunks. While it generalizes, performance degrades on very long sequences (>113 frames). Future work aims to train on heterogeneous chunk lengths.
• Controllability: The current framework lacks explicit control signals (e.g., camera trajectories, depth maps, text prompts). Integrating these for user-directed dynamics is a primary goal for future research.

In summary, FlowC2S offers a highly efficient, high-quality solution for video continuation, leveraging flow matching principles to drastically reduce computational costs while maintaining or improving generation quality.