💻 computer science

LatSearch: Latent Reward-Guided Search for Faster Inference-Time Scaling in Video Diffusion

This paper introduces LatSearch, an efficient inference-time scaling framework for video diffusion that utilizes a novel latent reward model to guide resampling and pruning during the denoising process, thereby overcoming the computational and signal limitations of prior noise-optimization methods to significantly improve video generation quality and controllability.

Original authors: Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Zhensong Zhang, Jifei Song, Jiankang Deng, Ioannis Patras

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zengqun Zhao, Ziquan Liu, Yu Cao, Shaogang Gong, Zhensong Zhang, Jifei Song, Jiankang Deng, Ioannis Patras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a masterpiece based on a description like "a butterfly changing from white to yellow."

In the world of AI video generation, the computer starts with a canvas covered in pure static (random noise). It has to slowly "denoise" this canvas, step by step, turning the static into a clear, moving picture.

The Problem: The "Blind" Painter
Current AI models are like painters who are blindfolded until the very last second. They make thousands of brushstrokes (calculations) to turn the static into a video.

The Issue: If the painter makes a mistake early on (like painting the wings blue instead of white), they keep painting over it for hundreds of steps, never realizing the error until the video is finished.
The Old Fix: Some researchers tried to find the "perfect starting static" (Golden Noise) or generate 100 different videos and pick the best one at the end. But this is like hiring 100 painters, waiting for all of them to finish their entire paintings, and then throwing away 99 of them. It's incredibly slow and expensive.

The Solution: LatSearch (The "Smart Guide")
The paper introduces LatSearch, a new method that acts like a smart art critic who walks into the studio while the painter is working, not just at the end.

Here is how it works, using a simple analogy:

1. The "Secret Scorecard" (Latent Reward Model)

Instead of waiting for the final video to be decoded (which is like waiting for the paint to dry and the frame to be built), LatSearch looks at the "raw canvas" (the latent state) while it's still wet and messy.

The Metaphor: Imagine a teacher grading a student's essay.
- Old Way: The teacher waits until the student finishes the whole 10-page essay, then grades it. If the first paragraph is terrible, the student wasted time writing the other nine.
- LatSearch Way: The teacher reads the first page, gives a quick score, and says, "Hey, this paragraph is off-topic. Let's try a different direction."
How it works: The AI has a special "reward model" that can look at the messy, half-finished video data and instantly say, "This looks good," or "This is going to fail." It checks three things: Does it look good? Is the motion smooth? Does it match the text prompt?

2. The "Smart Search" (Resampling and Pruning)

Once the AI has this "Smart Guide," it doesn't just follow one path. It runs a small race with multiple candidates.

The Race: Imagine the AI starts 6 different "painters" (candidates) working on the same prompt simultaneously.
The Checkpoints: Every few steps, the "Smart Guide" checks in on all 6 painters.
- Resampling: If one painter is doing a great job, the Guide says, "You're doing well, keep going! Let's make a copy of your work and send it on two more paths." If another painter is going off the rails, the Guide says, "Stop, you're wasting time."
- Pruning: This is the key to speed. Instead of waiting for all 6 painters to finish their entire 100-step journey, the Guide cuts off the bad ones early. It only keeps the best candidate to finish the job.

Why is this a Big Deal?

Speed: Because it stops the "bad painters" early, it saves a massive amount of computing power. The paper claims it can be up to 79% faster than previous methods that tried to do the same thing.
Quality: Because it can correct mistakes while the video is being made (not just at the end), the final result is much more accurate and follows the prompt better.
Efficiency: It's like finding the "Golden Noise" (the perfect starting point) on the fly, rather than guessing beforehand.

The Result

In the experiments, LatSearch took a standard video AI (Wan 1.3B) and made it generate videos that were:

Better: More creative, better motion, and more faithful to the text.
Faster: It achieved these results in roughly half the time (or less) compared to other "search" methods.

In a nutshell: LatSearch is like giving the AI a GPS with real-time traffic updates. Instead of driving a car blindly to a destination and hoping you didn't get lost, the GPS checks your location every few minutes, tells you if you're on the right road, and instantly reroutes you if you're going the wrong way, saving you time and gas.

1. Problem Statement

The paper addresses the challenge of inference-time scaling in text-to-video (T2V) diffusion models. While increasing the number of denoising steps improves fidelity, recent research suggests that the quality of the initial noise ("golden noise") and the ability to search for better trajectories are equally critical.

Existing approaches suffer from three main limitations:

Initial Noise Bias: Methods that optimize initial noise (e.g., FreeInit, FreqPrior) rely on priors injected at the very beginning. Once the denoising trajectory starts, they lack mechanisms to monitor or correct intermediate states, leading to error accumulation.
Delayed and Sparse Rewards: Search-based methods (e.g., Best-of-N, EvoSearch) typically generate multiple candidates and evaluate them only after full decoding into video space. This results in:
- High Computational Cost: Repeatedly decoding full videos is prohibitively expensive.
- Reward Delay: Feedback is only available at the end of the process, preventing early pruning of low-quality candidates.
Inability to Scale Search: Due to the high cost of full decoding, stronger search algorithms (like beam search or evolutionary algorithms) are often too slow to be practical for video generation.

2. Methodology: LatSearch

The authors propose LatSearch, a framework that enables efficient inference-time scaling by performing search directly in the latent space using a Latent Reward Model.

A. Latent Reward Model

Instead of evaluating decoded videos, this model scores partially denoised latents ( $z_t$ ) at arbitrary timesteps.

Input: A latent tensor $z_t$ , the text prompt $p$ , and the timestep $t$ .
Architecture: Based on a Vision Transformer (ViT) backbone (specifically Qwen2-VL-3B). The latent is patchified, and special query tokens ([VQ], [MQ], [TA]) are appended to predict scores for:
- Visual Quality (VQ)
- Motion Quality (MQ)
- Text Alignment (TA)
Training Strategy: Since ground truth rewards exist only for the final video, the authors use a similarity-grounded credit assignment:
- They calculate the cosine similarity ( $s_t$ ) between an intermediate latent $z_t$ and the final clean latent $z_0$ .
- The final video-level reward is scaled by $s_t$ to create a target for the intermediate latent.
- Loss Function: A combination of Regression Loss (to match absolute reward magnitudes) and Preference Loss (inspired by RLHF, to enforce relative ordering between candidates).

B. Reward-Guided Resampling and Pruning (RGRP)

LatSearch maintains a set of $N$ candidate trajectories during the diffusion process. The algorithm proceeds as follows:

Candidate Generation: At $t=T$ , $N$ candidates are generated by perturbing a base noise with isotropic noise.
Scoring & Resampling: At scheduled timesteps (e.g., $t=10, 15, 20$ $t = 10, 15, 20$ ):
- The Latent Reward Model scores all candidates.
- Probabilistic Resampling: Candidates are resampled based on reward-normalized probabilities (Softmax). This prevents over-reliance on a potentially imperfect reward model and encourages exploration.
- Uniqueness Pruning: Duplicate seeds (identical trajectories) are removed to save compute.
Final Pruning: At the final scheduled step, the candidate with the highest cumulative reward across all scoring steps is selected.
Decoding: Only the single surviving candidate is decoded into the final video.

3. Key Contributions

Latent Reward Model: A novel model capable of evaluating intermediate denoising latents for visual quality, motion, and text alignment without decoding to video space. It introduces a similarity-based credit assignment strategy to train on latent-level targets.
LatSearch Algorithm: An inference-time scaling method incorporating Reward-Guided Resampling and Pruning (RGRP). It balances exploration and exploitation via probabilistic resampling and reduces redundancy via uniqueness pruning.
Efficiency: By operating entirely in latent space and avoiding repeated full-video decoding, LatSearch achieves significant speedups compared to state-of-the-art search methods.

4. Experimental Results

The method was evaluated on the VBench-2.0 benchmark using the Wan2.1-1.3B and Wan2.1-14B models.

Quality Improvement: LatSearch consistently outperforms the baseline and other inference-time scaling methods across all dimensions (Creativity, Commonsense, Controllability, Human Fidelity, Physics).
- Compared to the baseline, it improved the average score by +1.94% (1.3B model) and +1.03% (14B model).
- It achieved comparable or better quality than EvoSearch (a strong evolutionary search baseline).
Efficiency Gains:
- Runtime: LatSearch is 4.77× faster than EvoSearch while achieving similar quality.
- Comparison: It reduces runtime by up to 79% compared to other search-based methods (e.g., VideoReward, EvoSearch) because it avoids the heavy cost of decoding multiple videos.
- Cost: It requires only 2.13× the computation of the baseline, whereas EvoSearch requires 10.15×.
Ablation Studies:
- Credit Assignment: Cosine similarity weighting was found to be superior to uniform or exponential weighting.
- Preference Loss: Including preference loss improved reward prediction accuracy by ~3% and downstream video quality.
- Search Budget: Performance scales with the number of candidates ( $N$ ), with $N=6$ offering a good balance between quality and cost.

5. Significance

Paradigm Shift: LatSearch moves inference-time scaling from "final-output evaluation" to "process-level supervision." This allows for early error correction and candidate pruning, which was previously impossible due to the lack of intermediate feedback.
Scalability: It demonstrates that complex search strategies (like beam search or evolutionary algorithms) can be applied to video generation efficiently if the evaluation metric is moved to the latent space.
Practicality: The method offers a "best of both worlds" scenario: it unlocks the quality gains of extensive search while maintaining inference times close to standard diffusion sampling, making high-quality video generation more accessible and efficient.

In summary, LatSearch solves the computational bottleneck of video search by introducing a latent reward model that provides real-time, fine-grained feedback, enabling efficient selection of high-quality generation trajectories without the prohibitive cost of decoding.