FC-VFI: Faithful and Consistent Video Frame Interpolation for High-FPS Slow Motion Video Generation

The paper proposes FC-VFI, a novel video frame interpolation method that leverages latent temporal modeling, semantic matching lines, and a temporal difference loss to achieve high-fidelity, motion-consistent 4x and 8x frame rate upscaling from 30 FPS to 120/240 FPS at 2560×1440 resolution, overcoming the fidelity and consistency limitations of existing diffusion-based approaches.

The Gist

Imagine you have a movie that plays at a standard speed, like a normal TV show. Now, imagine you want to slow that movie down to show every tiny detail of a falling leaf or a splashing drop of water. This is called slow motion.

To make smooth slow motion, you need to invent new frames that didn't exist before, filling in the gaps between the original pictures. This is called Video Frame Interpolation (VFI).

The problem? Old methods are like a clumsy painter. If they try to guess what happens between two frames, they often mess up. They might make a car look like it's melting, or make a person's face flicker weirdly. They struggle to keep the details sharp and the movement smooth.

Enter FC-VFI, a new AI tool that acts like a master time-traveling artist. Here is how it works, explained simply:

1. The Problem: The "Melting Car" Effect

Imagine you have a photo of a car at the start of a second and another photo of the same car at the end of the second. You want to create 7 new photos in between to make it look like smooth slow motion.

• Old AI methods try to guess the movement by drawing invisible lines (optical flow) between the cars. But in complex scenes, these lines get confused. The result? The car in the middle frames might look squished, stretched, or ghostly.
• The Goal: We need an AI that remembers exactly what the car looked like at the start and end, and fills in the middle without losing those details.

2. The Solution: FC-VFI's Three Magic Tricks

The authors of this paper built FC-VFI using a "large pre-trained video model" (think of it as an artist who has already watched millions of movies). They taught this artist three specific tricks to fix the problems:

Trick #1: The "Bookends" Strategy (Temporal Fidelity Modulation)

Imagine you are trying to draw a scene that happens between a sunrise and a sunset.

• Old way: The artist looks at the sunrise and sunset, then closes their eyes and tries to guess the middle. They might forget the colors of the sky.
• FC-VFI way: The artist keeps the sunrise and sunset photos open on the table the entire time they are drawing the middle pictures. They constantly peek at the start and end to make sure the colors and shapes stay true.
• The Analogy: It's like holding the first and last pages of a book open while you write the middle chapters, ensuring the story never loses its plot. This prevents the "melting car" effect and keeps the details sharp.

Trick #2: The "Skeleton" Guide (Matching Lines)

Sometimes, objects move fast, and it's hard to tell where they are going.

• Old way: The artist tries to track every single pixel (like counting every grain of sand on a beach). This is messy and prone to errors.
• FC-VFI way: The artist only looks at the key outlines of the objects (like the edge of a license plate or the corner of a building). They use these "skeleton lines" to guide the movement.
• The Analogy: Instead of trying to guess how a dancer moves by watching their whole body, you just watch their feet and hands. If the feet and hands move correctly, the rest of the body follows naturally. This keeps the structure of the object solid, even when it's moving fast.

Trick #3: The "Smoothness" Check (Temporal Difference Loss)

Sometimes, AI generates frames that are so similar to each other that the video looks like a frozen statue, even though it's supposed to be moving.

• The Fix: FC-VFI has a built-in rule: "If the picture you just drew looks exactly like the one before it, you must make it move a little more."
• The Analogy: It's like a dance instructor telling a student, "Don't just stand there! Take a step!" This ensures the slow-motion video feels fluid and dynamic, not jittery or stuck.

3. The Result: High-Speed, High-Quality Magic

Because of these tricks, FC-VFI can take a video playing at 30 frames per second (standard TV speed) and turn it into 120 or 240 frames per second (super smooth slow motion) without losing quality.

• Resolution: It can do this even on huge, high-definition screens (like 1440p), which is a big deal because most AI struggles with that much detail.
• Speed: It's surprisingly fast. While other methods might take a long time to "re-draw" the video multiple times to fix mistakes, FC-VFI gets it right in just a few tries (10 steps).

Summary

Think of FC-VFI as a super-smart editor who doesn't just guess what happens between two moments in time. Instead, it:

1. Keeps the original photos in sight so it never forgets the details.
2. Follows the outlines of objects to keep them from breaking apart.
3. Forces the movement to be smooth and natural.

The result is a slow-motion video that looks so real and detailed, you'd swear the camera was filming at super-speed, even if it wasn't.

Technical Summary

1. Problem Statement

Video Frame Interpolation (VFI) aims to synthesize intermediate frames between a start and end frame to increase frame rates (e.g., 30 FPS to 120/240 FPS) for slow-motion generation. While traditional methods rely on optical flow, they struggle in complex scenes with occlusions or rapid motion. Recent diffusion-based methods offer better generative capabilities but suffer from two critical issues:

• Fidelity Loss: Generative priors often cause structural distortions, artifacts (e.g., ghosting), and flickering, where intermediate frames deviate significantly from the geometry of the start/end frames.
• Temporal Inconsistency: Interpolated sequences often exhibit inaccurate motion or "near-static" behavior between adjacent frames. Existing motion guidance (dense optical flow or sparse points) is either error-prone in complex scenes or lacks sufficient structural context.
• Efficiency: Many current diffusion approaches require bidirectional inference (generating forward and backward then fusing) or re-noising steps, leading to high computational overhead.

2. Methodology: FC-VFI

The authors propose FC-VFI, a framework fine-tuned from a pre-trained large-scale Image-to-Video (I2V) diffusion model (specifically HunyuanVideo-I2V). The core innovations are:

A. Temporal Fidelity Modulation Reference (TFMR)

Instead of concatenating start/end frame latents with intermediate latents via channels (which can perturb boundary information), FC-VFI concatenates them along the temporal dimension.

• Mechanism: The noisy latent of the intermediate frame ($z_n$) is concatenated with the latents of the start ($z_s$) and end ($z_e$) frames to form a sequence $Z = \{z_s, z_n, z_e\}$.
• Fidelity Modulation: To prevent the diffusion process from corrupting the clean boundary frames, the model assigns a fixed timestep $t^*=0$ (noise-free state) to $z_s$ and $z_e$ during the velocity prediction, while intermediate frames follow the standard noise schedule. This ensures the generation process consistently references the high-fidelity features of the endpoints.

B. Matching Lines Condition

To address structural instability during large camera motions or rapid object movement, the authors introduce a semantic matching line mechanism.

• Extraction: Semantic line pairs are extracted from start and end frames using GlueStick.
• Encoding: A lightweight ResNet-based encoder converts these lines into condition features ($c_s, c_e$).
• Injection: These features are added element-wise to the boundary latents ($z'_s = z_s + c_s$, $z'_e = z_e + c_e$). A replicated DiT block processes these enhanced latents to produce residual features, which are injected back into the main backbone.
• Advantage: Unlike dense optical flow (error-prone) or sparse points (lack structure), matching lines provide robust structural boundaries and object shapes without requiring frame-wise conditioning that conflicts with temporally compressed VAEs.

C. Temporal Difference Loss

To mitigate "near-static" artifacts where adjacent interpolated frames appear identical:

• The model introduces a loss term ($L_{temp}$) that explicitly aligns the predicted motion difference between consecutive frames with the ground truth motion difference.
• Objective: $L = L_{flow} + \omega L_{temp}$. This encourages dynamic distinctions and smoother motion transitions.

D. Training Strategy

• Architecture: Based on Flow Matching (FM) with a DiT backbone.
• Fine-tuning: Uses LoRA (Rank 64) to fine-tune the DiT backbone and a full training of the line encoder.
• Efficiency: The method eliminates the need for bidirectional inference or re-noising, requiring only 10 denoising steps.

3. Key Contributions

1. Faithful Interpolation Strategy: Proposes TFMR, a novel temporal modeling strategy that preserves boundary fidelity by modulating timesteps and concatenating latents temporally, enabling high-resolution (2560×1440) interpolation without structural distortion.
2. Robust Motion Guidance: Introduces a Matching Lines Condition that offers superior structural stability compared to optical flow or sparse points, specifically designed to work with temporally compressed video latents.
3. Temporal Consistency: Develops a Temporal Difference Loss to eliminate static artifacts and ensure smooth motion transitions.
4. Efficiency & Scalability: Achieves 4× and 8× interpolation (30 FPS $\to$ 120/240 FPS) with significantly fewer inference steps (10 steps) compared to SOTA diffusion methods, supporting resolutions up to 2560×1440.

4. Experimental Results

The model was evaluated on diverse datasets (REDS, Adobe240, X-Test, DAVIS-2017) against optical-flow-based (GIMM-VFI) and diffusion-based (GI, ViBiDSampler, FCVG) methods.

• Quantitative Performance:
  • At 1024×576, FC-VFI outperforms all diffusion-based SOTA methods across all metrics (PSNR, SSIM, FID, FVD, LPIPS) for both 4× and 8× interpolation.
  • At 2560×1440, it achieves competitive results against GIMM-VFI while maintaining the generative flexibility of diffusion models.
• Qualitative Performance:
  • FC-VFI preserves fine-grained details (text, license plates, building textures) and handles occlusions and high-contrast lighting better than competitors.
  • It avoids common artifacts like ghosting, flickering, and structural deformation seen in other methods.
• Efficiency:
  • Inference time is significantly lower than other diffusion methods (e.g., 16s vs. 606s for 4× at 1024×576) due to the single-pass, 10-step denoising process.
• Ablation Studies:
  • Removing TFMR or the Matching Lines condition leads to significant drops in PSNR and SSIM, confirming the necessity of temporal concatenation and structural guidance.
  • The Temporal Difference Loss effectively reduces FVD (video quality metric), indicating smoother motion.

5. Significance

FC-VFI represents a significant advancement in high-fidelity video generation. By shifting from channel-wise conditioning to temporal concatenation and introducing fidelity modulation, it solves the long-standing trade-off between generative creativity and structural faithfulness in VFI. Its ability to generate high-frame-rate slow-motion videos at 2K resolution (2560×1440) with low computational cost makes it highly practical for applications in animation, film production, and video enhancement, setting a new standard for diffusion-based interpolation.