💻 computer science

Multiple Hypothesis Flow Estimation for Video Frame Interpolation under Matching Ambiguity

This paper proposes a multiple hypothesis flow estimation framework that preserves top-K candidate correspondences and selects the most reliable one via a router to address matching ambiguities in video frame interpolation, thereby reducing artifacts like ghosting and blur while achieving state-of-the-art perceptual quality.

Original authors: Zibo Su, Jing Kong, Ruixing Wang, Zhanhe Zhang, Kun Wei

Published 2026-08-10

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Zibo Su, Jing Kong, Ruixing Wang, Zhanhe Zhang, Kun Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to create a smooth, slow-motion video from a standard clip. To do this, computers need to invent brand-new frames that fit perfectly between the ones you already have. This is like a magician trying to guess exactly what a dancer looked like in the split second between two photos. The trickiest part of this magic is figuring out how every single pixel moves from the first photo to the second. Usually, the computer tries to find just one "best guess" for where a pixel went. But sometimes, the world is confusing. If you have a spinning windmill with identical blades, or a pile of identical snowflakes falling, or a fast-moving fist that leaves a blurry trail, the computer gets stuck. It sees a pixel in the first frame and asks, "Did you move to this spot, or that spot, or that one?" In these tricky situations, there isn't just one right answer; there are several equally plausible ones. If the computer is forced to pick just one guess immediately, it often picks the wrong one or a messy compromise, resulting in a new frame that looks ghostly, blurry, or distorted.

This paper tackles that specific headache in the field of video frame interpolation (VFI). The authors propose a clever new way to handle these confusing moments. Instead of forcing the computer to make a single, desperate guess right away, they let it keep a shortlist of the top three best guesses (or "hypotheses") for every pixel. Think of it like a detective who doesn't arrest the first suspect they see, but instead keeps three suspects in the lineup. The computer then uses a special "reliability judge" to look at the clues—like how consistent the motion is and how sharp the image looks—and picks the single best suspect to be the final answer. By waiting to make the final choice until it has gathered more evidence, the method avoids the blurry, ghostly mistakes that happen when the computer is forced to guess too early. The result is a much clearer, sharper video, especially in those chaotic scenes where things spin, repeat, or move too fast to see clearly.

The Problem: When "One Right Answer" Doesn't Exist

Most video interpolation methods work like a strict teacher who demands a single answer to every question. They look at two frames of a video and try to calculate exactly how every pixel moved. In simple scenes, this works great. But in the real world, things get messy. The paper identifies three specific scenarios where this "one answer" rule breaks down:

Repetitive Textures: Imagine a field of identical flowers or a pile of snow. If you look at one snowflake in the first frame, it looks exactly like its neighbors. When the computer tries to match it to the next frame, it sees dozens of identical-looking snowflakes. It can't tell which one is the "real" match.
Symmetric Rotations: Think of a windmill or a helicopter propeller. If the blades are all the same, a blade in the first frame could have rotated to match any of the other blades in the next frame. There are multiple correct answers, not just one.
Fast Motion with Blur: When something moves super fast, like a punch in a martial arts movie, it leaves a blur. The computer sees a smear of pixels and can't pinpoint exactly where the object started or ended up.

In all these cases, the "ground truth" (the actual video frame that should appear in the middle) doesn't give the computer a unique clue. The computer might be able to blend the wrong pixels together and still make a picture that looks okay to a machine, but it will look weird to a human. Traditional methods force the computer to pick just one path immediately. If it picks the wrong one, or tries to average two different paths together, the result is "ghosting" (seeing double images) or "structural distortion" (objects looking melted or broken).

The Solution: The "Top-K" Shortlist

The authors of this paper, Zibo Su and colleagues, suggest a different approach. Instead of asking the computer to commit to one answer immediately, they let it keep a Top-K list of the best candidates. In their experiments, they found that keeping 3 candidates (K=3) worked best.

Here is how their system, called Multiple Hypothesis Flow Estimation (MHFE), works step-by-step:

The Coarse Search: First, the computer looks at the whole image and finds the top 3 most likely places a pixel could have moved to. These are called "anchors."
The Local Refinement: Instead of just guessing, the computer zooms in on each of those 3 anchors. It uses a special "local attention" tool to look closely at the neighborhood around each anchor. This helps it refine the guess, adding tiny details to make the motion path more precise.
The Reliability Judge: Now the computer has 3 refined options for every pixel. It doesn't just pick the first one. It uses a "reliability-guided router." This is a smart system that checks clues like:
- Consistency: If the pixel moves forward and then backward, does it end up where it started?
- Focus: Is the match sharp and clear, or is it fuzzy?
- Confidence: How sure is the initial search?
The Final Choice: Based on these clues, the router picks the single most reliable option for that specific pixel. Crucially, it picks one and discards the others. It does not blend them together.

Why "Picking One" is Better Than "Mixing Them"

A key finding in the paper is that mixing the different guesses is actually bad. Some older methods tried to average all the possible motions together to be "safe." The authors show that this averaging creates a "compromised prediction." It's like trying to blend a red car and a blue car together; you don't get a purple car that looks real; you get a muddy, ghostly mess.

By using a "hard routing" strategy (picking one winner), the computer ensures that the final image is built from a single, consistent story. If the computer is unsure, it waits until it has enough evidence to pick the best single story, rather than telling a confusing story that mixes two different plots.

The New Test: MA-HD

To prove their idea works, the researchers couldn't just use standard tests because those tests didn't have enough of these tricky "ambiguous" scenes. So, they built a new benchmark called MA-HD (Matching-Ambiguity High Definition).

This new test set includes 1,000 video clips specifically chosen for their confusion factors:

Dynamic Textures: Flames, water waves, snow, and rain.
Rotation: Windmills and airplane propellers.
Fast Motion: Martial arts punches and drifting cars.

They tested their method against the best existing video tools (like EMA-VFI, SGM-VFI, and others) on these difficult scenes.

The Results: Sharper, Cleaner, and Less Ghostly

The results were impressive, especially in the tricky categories:

Perceptual Quality: The authors used metrics called LPIPS and DISTS, which measure how much a computer thinks an image looks like what a human would see. On their new MA-HD test, their method scored 11.14 for LPIPS and 7.19 for DISTS. This was significantly better than the second-best method (PerVFI), which scored 13.69 and 7.60 respectively. In plain English, their videos looked much more natural and less blurry to human eyes.
Handling Ambiguity: In the visual comparisons, the old methods produced "ghosts" (faint double images) and "tearing" (parts of the image looking ripped) around spinning blades and fast fists. The new method kept the blades sharp and the fists solid.
Efficiency: The method is also reasonably fast. It runs at about 0.78 frames per second (FPS) on the MA-HD test, which is comparable to other high-quality methods, though not the absolute fastest. It uses about 6.91 GB of memory, which is much less than some other heavy-duty methods that require over 30 GB.

What the Paper Rules Out

The authors are very clear about what doesn't work:

Soft Fusion: They explicitly argue against averaging the different motion paths. Their tests showed that when they tried to blend the candidates (Variant B in their study), the image quality collapsed, creating heavy ghosting and washed-out details, even if the pixel-by-pixel numbers looked okay.
Confidence-Only Routing: They also tested a version where the computer just picked the candidate with the highest initial confidence score, without using the learned "reliability judge." This performed worse than their full system, proving that the smart judge is necessary to filter out bad guesses.

Conclusion

The paper suggests that when the world is ambiguous—when there are multiple ways a pixel could have moved—the best strategy isn't to guess randomly or to mix all the guesses together. Instead, the computer should keep a shortlist of the best possibilities, refine them with extra care, and then use a smart, learned system to pick the single most reliable story. This approach, they show, leads to cleaner, sharper, and more realistic slow-motion videos, especially in the chaotic, fast-moving, or repetitive scenes that usually break other video tools.

Technical Summary: Multiple Hypothesis Flow Estimation for Video Frame Interpolation under Matching Ambiguity

Problem Statement

Video Frame Interpolation (VFI) typically synthesizes intermediate frames by estimating optical flow, warping input frames, and blending the results. However, existing single-flow estimators struggle in regions exhibiting matching ambiguity. In such regions, a single query pixel in the source frame may correspond to multiple visually plausible locations in the target frame due to:

Repetitive or stochastic textures (e.g., flames, snow), where local patterns resemble multiple regions.
Rotating symmetric structures (e.g., windmill blades), where identical components appear at regular intervals.
Fast motion with blur, which smears contours and broadens matching evidence across multiple locations.

In these scenarios, the ground-truth intermediate frame provides only indirect, image-level supervision. This supervision is often insufficient to uniquely identify the correct latent correspondence, as different combinations of flow, visibility, and blending can yield similar reconstruction results. Consequently, deterministic single-flow estimators are forced to collapse multiple plausible peaks into a single, compromised prediction. This often leads to selecting an incorrect match or an inconsistent displacement relative to neighbors, resulting in ghosting, structural distortion, or blur in the synthesized frame.

Methodology

The authors propose a Multiple Hypothesis Flow Estimation (MHFE) framework designed to preserve ambiguity during the estimation phase and resolve it only after reliability assessment. The framework operates through three primary stages:

1. Top-K Optical Flow Candidate Endpoints

Instead of regressing a single flow vector, the model extracts features at 1/8 resolution for global matching. A differentiable matching layer computes a confidence matrix between the two input frames. For each query location $p$ , the system selects the top-K coarse matching anchors $\{q_k(p)\}_{k=1}^K$ . Crucially, this selection is row-wise, ensuring that $K$ alternative candidate motions are preserved for the same local content, rather than sparse matches from different queries.

2. Anchor-Centered Local Attention

Each of the $K$ coarse anchors is refined independently using anchor-centered local attention:

A local neighborhood is cropped from the fine-resolution feature map of the target frame around each anchor.
An attention map $\alpha_k$ is computed to weigh local residuals relative to the anchor.
The final motion $m_k(p)$ for the $k$ -th hypothesis is formulated as the sum of the coarse anchor displacement and the weighted local residual.
Simultaneously, appearance features $a_k(p)$ are aggregated using the same attention weights.

This coarse-to-fine strategy allows the model to refine $K$ distinct hypotheses separately without collapsing them into a single compromised estimate.

3. Reliability-Guided Router and Frame Synthesis

A reliability-guided router evaluates the $K$ flow–appearance hypotheses to select the most reliable one for synthesis:

Reliability Scoring: For each candidate, the router computes a reliability score based on attention concentration (how focused the local attention is) and bidirectional consistency (forward-backward flow error).
Hard Routing: The router selects the single most reliable hypothesis $k^*$ for each pixel. The system employs a straight-through estimator to ensure the forward pass during training and inference is identical: the generator synthesizes the frame using only the selected hard-routed flow and appearance features, avoiding soft blending of different trajectories which causes ghosting.
Gradient Estimation: During backpropagation, gradients for the routing network and unselected hypotheses are approximated using a soft surrogate (Gumbel-softmax) to enable end-to-end training.

Training Objectives

The model is trained with two complementary losses:

Reconstruction Loss ( $L_{rec}$ ): Computed on the single hard-routed output frame to ensure the selected hypothesis aligns with the ground truth.
Hypothesis Loss ( $L_{hyp}$ ): A "best-of-K" objective (soft-min) applied to all independently rendered hypotheses. This encourages the candidate set to contain at least one low-error hypothesis for every location, preventing the model from discarding correct matches early.

Key Contributions

Multiple Hypothesis Framework: The paper introduces a VFI framework that retains top- $K$ candidate flows as explicit motion hypotheses and selects one per pixel via learned reliability scores, explicitly avoiding the soft blending of different candidates during frame synthesis.
Coarse-to-Fine Refinement: The method initializes candidates from global matching anchors and refines them independently using local anchor-based motion. This enables the separate refinement of $K$ plausible correspondences rather than collapsing them.
MA-HD Benchmark: The authors constructed a new benchmark, MA-HD (Matching-Ambiguity HD), featuring 1,000 triplets specifically designed to test ambiguous motion scenarios, including dynamic textures, rotational symmetry, and fast blurred motion.

Experimental Results

The method was evaluated on standard benchmarks (SNU-FILM, Xiph) and the proposed MA-HD benchmark, comparing against state-of-the-art methods like EMA-VFI, SGM-VFI, PerVFI, AMT-G, and LC-Mamba.

Perceptual Quality: The proposed method achieved the best LPIPS and DISTS scores across all difficulty levels of SNU-FILM, the Xiph dataset, and the MA-HD benchmark. Notably, on the MA-HD benchmark (specifically designed for ambiguity), the method outperformed the second-best (PerVFI) by 18.6% in LPIPS and 5.4% in DISTS.
PSNR: While PSNR is a conventional metric, the method also achieved the best PSNR on the most ambiguous splits (Hard and Extreme) of SNU-FILM.
Efficiency: The method maintains a favorable quality-cost trade-off, requiring approximately 3.43 GB of GPU memory on SNU-FILM and 6.91 GB on MA-HD, with inference speeds of 1.66 FPS and 0.78 FPS respectively.
Qualitative Analysis: Visual comparisons demonstrate that the method effectively reconstructs sharp textures in stochastic regions, preserves rigid geometry in rotating symmetric structures, and avoids image tearing in fast motion scenarios where baseline methods produce ghosting or blur.

Significance and Claims

The paper argues that the limitation of current VFI methods lies not necessarily in the reconstruction loss averaging different flow modes, but in the fact that image-level supervision leaves latent correspondences under-constrained in ambiguous regions, while single-flow bottlenecks cannot retain the resulting multiple peak matching evidence.

By preserving multiple hypotheses until a reliability-aware routing decision is made, the proposed framework allows the model to commit to a single, consistent motion hypothesis per pixel rather than averaging competing motions. The authors claim this strategy is particularly effective in scenes where no unique ground-truth motion exists, as evidenced by the superior perceptual quality on the MA-HD benchmark. The work suggests that preserving multiple plausible motions and committing to one is a robust strategy for handling matching ambiguity in video frame interpolation.