🤖 AI

Lookahead Sample Reward Guidance for Test-Time Scaling of Diffusion Models

This paper introduces LiDAR, an efficient test-time scaling method for diffusion models that computes closed-form expected future reward guidance using marginal samples and few-step lookahead sampling to achieve high human-aligned generation quality with a 9.5x speedup over existing gradient guidance approaches.

Original authors: Yeongmin Kim, Donghyeok Shin, Byeonghu Na, Minsang Park, Richard Lee Kim, Il-Chul Moon

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yeongmin Kim, Donghyeok Shin, Byeonghu Na, Minsang Park, Richard Lee Kim, Il-Chul Moon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented artist (a Diffusion Model) who can paint beautiful pictures from scratch. You give them a prompt like "a yellow bird on a black motorcycle," and they start painting. They begin with a canvas full of static noise (like TV snow) and slowly refine it into a clear image.

However, sometimes the artist gets a little lost. They might paint a bird that looks more like a chicken, or a motorcycle that looks like a toaster. They are technically following the rules of "painting," but they aren't perfectly matching your specific vision.

This paper introduces a new way to guide the artist while they are still painting, without needing to retrain them or hire a new teacher. The method is called LiDAR (Lookahead Sample Reward Guidance).

Here is how it works, broken down into simple concepts:

1. The Problem: The "Backward Rollout" is Too Slow

To fix the artist's mistakes, previous methods tried to be very thorough. They would say, "Let's pause right now, imagine 100 different ways this painting could finish, check which one looks best, and then steer the current painting toward that best version."

The Flaw: Doing this "100 different ways" for every single brushstroke takes forever. It's like asking the artist to finish the whole painting 100 times just to decide the next stroke. It's too expensive and slow.

2. The Old Shortcut: "Guessing the Future"

Other methods tried to be faster by using a shortcut. They would look at the current messy painting and guess, "If we just smooth this out a bit, what will the final picture look like?" Then they'd check that guess against a reward system (a score for how good the image is).

The Flaw: This guess is often wrong. If the artist is still in the "messy noise" phase, the guess is a bad approximation. If you push the artist too hard based on a bad guess, the painting can become weird or distorted. Also, this method often requires the artist to "think backward" through their own brain (neural backpropagation), which is computationally heavy.

3. The LiDAR Solution: The "Lookahead" Strategy

The authors of this paper came up with a clever trick. Instead of trying to predict the future from the current messy painting, they do something different:

Step A: The "Lookahead" Sketches (Phase 1)
Before the main painting starts, the artist quickly sketches a few rough drafts (let's say 50 of them) based on your prompt. These are "lookahead samples." They aren't perfect, but they give a hint of what the final image could look like.

The Reward: A judge (a reward function) looks at these 50 rough drafts and gives them scores. "This draft has a great bird, but the motorcycle is weird. This one has a perfect motorcycle."

Step B: The Main Painting with a Compass (Phase 2)
Now, the artist starts the real painting from scratch. As they work, they don't just look at their current messy canvas. They also look at those 50 rough drafts they made earlier.

The Magic: The painting process is guided by a simple rule: "Move your brush toward the rough drafts that got high scores, and push away from the ones that got low scores."

Why is this special?

No Backtracking: The artist doesn't need to re-simulate the whole painting 50 times. They just use the pre-made rough drafts as a map.
No Heavy Math: The paper claims this method calculates the guidance using simple math (like measuring distances) rather than complex neural network calculations. It's like using a compass instead of a supercomputer to find North.
Speed: Because they pre-made the rough drafts quickly (using a fast solver), the whole process is much faster than previous methods. The paper claims it is 9.5 times faster than the current best methods while achieving the same high quality.

The Analogy: Hiking with a Map

Old Method (Gradient Guidance): You are hiking in the fog. To find the best path, you try to imagine every possible path you could take, calculate the score of each, and then adjust your step. This is exhausting and slow.
LiDAR Method: Before you start hiking, you send a drone up to take 50 quick photos of the terrain ahead. You mark the best spots on a map. As you hike, you simply look at your map and walk toward the "good" photos and away from the "bad" ones. You don't need to imagine the future; you just follow the pre-made map.

The Results

The paper tested this on popular image generators (like SDXL).

Quality: The images were better at following the prompt (e.g., getting the number of objects right, matching colors) compared to previous methods.
Efficiency: It was significantly faster and used less computer memory.
Versatility: It worked well even when the "rough drafts" were very quick and simple, proving you don't need perfect previews to get a great final result.

In short, LiDAR is a way to give a diffusion model a "cheat sheet" of potential futures before it starts drawing, allowing it to steer itself toward the best outcome quickly and efficiently, without needing to do heavy calculations at every step.

Technical Summary: Lookahead Sample Reward Guidance for Test-Time Scaling of Diffusion Models

Problem Statement

Diffusion models have achieved remarkable generative performance across images, videos, and language; however, generated samples frequently fail to fully align with human intent. While reward functions capturing human preferences exist, integrating them into the sampling process presents significant challenges. The core objective of test-time scaling is to sample from a reward-tilted distribution $p_\theta^r(x_0|c) \propto p_\theta(x_0|c) \exp(\lambda \cdot r(x_0, c))$ .

Achieving this requires computing the Expected Future Reward (EFR), $r_t^\lambda(x_t, c)$ , for intermediate particles $x_t$ to guide the reverse diffusion process. Existing methods face critical limitations:

Backward Rollout: Directly sampling from $x_t$ to $x_0$ to estimate EFR is computationally prohibitive due to the need for iterative denoising at every timestep.
Tweedie-based Approximations: Methods like Gradient Guidance and Sequential Monte Carlo (SMC) rely on first-order Taylor approximations (Tweedie's formula) to estimate EFR. These introduce bias, particularly when the reward signal strength ( $\lambda$ ) is high, and often require backpropagation through neural networks (Gradient Guidance) or suffer from sample collapse in high-dimensional spaces (SMC).

Methodology: LiDAR Sampling

The authors propose LiDAR (Lookahead Sample Reward Guidance), a framework that decouples the neural dependency between the intermediate particle $x_t$ and the EFR, enabling closed-form, derivative-free guidance.

1. Reformulation of Expected Future Reward

The paper derives a new formulation for EFR (Theorem 3.1) that expresses the reward expectation using only marginal samples from a pre-trained diffusion model and the forward perturbation kernel, rather than conditioning on the current particle $x_t$ via neural networks:
$r_t^\lambda(x_t, c) = \log \mathbb{E}_{p_\theta(x_0|c)} \left[ \frac{p(x_t|x_0)}{\mathbb{E}_{p_\theta(x_0|c)}[p(x_t|x_0)]} \exp(\lambda \cdot r(x_0, c)) \right]$
This "forward rollout" approach allows the EFR to be computed for any $x_t$ using a pre-generated set of marginal samples $\{\hat{x}_0^i\}$ , eliminating the need for backpropagation through the score network $s_\theta$ or the reward model during the guidance step.

2. Efficient Lookahead Sampling

To mitigate the cost of generating marginal samples for every prompt, the authors introduce an efficient lookahead sampling strategy. Instead of using the full, slow diffusion process, they employ a fast generator $q(x_0|c)$ , such as a few-step ODE solver (e.g., DPM-Solver) or a distilled model, to generate a set of $n$ lookahead samples $\{\hat{x}_0^i\}_{i=1}^n$ . These samples are annotated with reward values $r(\hat{x}_0^i, c)$ .

3. Closed-Form Derivative-Free Guidance

Using the finite set of lookahead samples, the authors derive a closed-form expression for the gradient of the empirical lookahead reward (Theorem 3.3). The guidance term steers the current particle $x_t$ toward high-reward lookahead samples and repels low-reward ones:
$\nabla_{x_t} \hat{r}_t^\lambda(x_t, c) = \sum_{i=1}^n (w_i^r - w_i) \frac{\hat{x}_0^i}{\sigma_t^2}$
where $w_i^r$ and $w_i$ are softmax weights based on the reward values and the distance between $x_t$ and the lookahead samples $\hat{x}_0^i$ . This formulation requires no neural differentiation.

4. Theoretical Scaling Laws

The paper establishes scaling laws for the method:

Lookahead Accuracy ( $\delta$ ): The total variation distance between the target distribution and the LiDAR-generated distribution decreases as $O(1/\sqrt{\delta})$ , where $\delta$ is the number of discretization steps in the lookahead solver.
Sample Size ( $n$ ): The variance of the empirical reward estimate decreases as $O(1/\sqrt{n})$ , implying that increasing the number of lookahead samples improves approximation accuracy.

Key Results

Experiments were conducted on SD v1.5, SDXL, FLUX (flow matching), and UDLM (discrete diffusion) using ImageReward, CLIP, HPS, and GenEval metrics.

Performance vs. Efficiency: On SDXL, LiDAR achieves GenEval performance comparable to the latest gradient guidance methods (DATE) but with a 9.5× speedup in inference time.
Memory Efficiency: Unlike gradient guidance methods which require ~3× more memory due to backpropagation, LiDAR's memory usage is nearly identical to vanilla sampling, as the target sampling stage (Algorithm 2) does not require gradient computation.
Robustness to Reward Hacking: While gradient guidance methods often degrade in metrics like CLIP and HPS when optimizing for ImageReward (reward hacking), LiDAR improves all metrics simultaneously. This is attributed to the correlation between ImageReward and other metrics in the lookahead samples, which naturally guides the sampler in a balanced direction.
Orthogonal Improvements: LiDAR provides additional performance gains when applied to models already fine-tuned with Direct Preference Optimization (DPO), demonstrating its orthogonality to training-based alignment.
Scalability: The method scales effectively with the number of lookahead samples ( $n$ ) and solver steps ( $\delta$ ), and can be combined with other techniques like SMC and Best-of-N.

Significance and Claims

The paper claims that LiDAR systematically addresses the efficiency and reward-hacking limitations of existing test-time scaling methods. By reformulating the EFR computation to rely on forward rollout and marginal samples, the authors enable derivative-free guidance that is both computationally efficient and theoretically grounded.

The authors emphasize that LiDAR achieves a superior efficiency-performance trade-off compared to existing baselines. Specifically, it avoids the computational overhead of neural backpropagation and the sample collapse issues of SMC, while providing more accurate EFR estimates than Taylor-approximation-based methods. The work suggests that decoupling the reward signal from the intermediate particle's neural dependency is a viable path toward scalable, human-aligned generative modeling.