💻 computer science

Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioning

This paper introduces Equilibrium Forcing (EqF), a novel framework for autoregressive video generation that eliminates noise level conditioning to decouple denoising learning from sampling, thereby enabling adaptive, closed-loop inference that significantly improves video quality and consistency compared to standard methods.

Original authors: Hansen Jin Lillemark, Alex Rojas, Zachary Novack, Runqian Wang, Yilun Du, Yian Ma, Taylor Berg-Kirkpatrick, Rose Yu

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hansen Jin Lillemark, Alex Rojas, Zachary Novack, Runqian Wang, Yilun Du, Yian Ma, Taylor Berg-Kirkpatrick, Rose Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Creating a video that plays out over time, frame by frame, is one of the most demanding tasks for artificial intelligence. It requires a model to not only generate a single image but to imagine a sequence where every new frame follows logically from the last, maintaining a consistent world, lighting, and motion. This is the realm of autoregressive video generation, a technology that powers everything from interactive simulations to the next generation of digital storytelling. For years, the most successful methods for this task have relied on a specific, rigid strategy: they start with pure static noise and gradually remove it, step by step, to reveal the final image. To keep this process stable, these systems have traditionally been forced to follow a strict, pre-written schedule, telling the computer exactly how much noise to remove at every single moment, regardless of what the image actually looks like at that moment.

A team of researchers from UC San Diego, MIT, and Harvard University has challenged this long-held assumption. They propose that forcing a computer to follow a fixed script is actually the source of many errors, especially when generating long videos. In their new work, they introduce a method called Equilibrium Forcing. Instead of following a rigid schedule, their system learns to listen to the image it is currently creating. It estimates how much noise remains in the picture based on what it sees, and then adjusts its next move accordingly. This creates a closed loop where the generation process adapts to its own progress, much like a driver adjusting their speed based on the road ahead rather than sticking to a pre-set speed limit. The result is a system that produces longer, more consistent videos with fewer errors than previous methods.

The core problem the researchers identified is that the old way of generating video creates a mismatch between the plan and the reality. In standard methods, the computer is told to remove a specific amount of noise at step one, then a different amount at step two, and so on. This schedule is fixed before the video generation even begins. However, as the computer generates frames, small errors inevitably creep in. The image it is working on might not have the exact amount of noise the schedule predicted. Because the computer blindly follows the schedule, it continues to apply the wrong amount of correction, causing the errors to pile up. Over a long video, this divergence becomes severe, leading to flickering, distorted shapes, or a complete breakdown of the scene's logic. The researchers found that this gap between the scheduled noise level and the actual noise level in the image grows larger with every frame, eventually ruining the quality of the generation.

To solve this, the team developed a framework that removes the noise schedule entirely. They trained a new type of model that does not need to be told how much noise is present. Instead, the model learns a single, unified way to clean up an image, regardless of how noisy it is. During the generation process, the model looks at the current image and internally estimates how much noise is left. It then uses this estimate to decide how much to change the image in the next step. This allows the system to operate in a closed loop, constantly checking its own progress and adjusting its speed and direction. If the image is still very noisy, it takes a larger step; if it is nearly clean, it takes a smaller, more careful step. This flexibility prevents the errors from accumulating, allowing the video to stay stable for hundreds of frames.

The researchers tested this approach on several challenging datasets, including videos of a robot arm performing tasks, tours of real estate properties, and gameplay footage from the video game Minecraft. In every case, their new method outperformed the standard approaches. On the Minecraft dataset, for example, the new system generated 300-frame sequences with significantly higher visual quality and consistency than the previous best methods. The improvement was particularly noticeable in long sequences where the old methods would typically start to degrade. The researchers also showed that this new method works even when applied to very large, pre-existing models. By taking a massive video model that was originally trained with the old, rigid schedule and simply retraining it with their new, flexible objective, they were able to unlock the same high-quality, adaptive behavior without needing to build a new model from scratch.

A key part of this success is the ability to adapt the generation process to the available computing power. Because the system knows how much noise is left in the image, it can decide to stop generating early if the image is already clean enough, or speed up the process if there is a lot of noise to remove. This "budget-adaptive" capability means the system can produce high-quality videos even when given fewer computing resources, making the technology more practical for real-world applications. The researchers also demonstrated that this method allows the system to recover from mistakes more effectively. If the system generates a frame that is slightly off, it can detect the error and correct it in the next step, rather than being forced to continue down a flawed path dictated by a pre-set schedule.

The findings suggest that the rigid schedules that have dominated video generation for years are not necessary for stability. In fact, they may be the very thing that limits the quality and length of the videos we can create. By letting the model listen to its own progress and adapt in real time, the researchers have opened a new path for creating longer, more coherent, and more reliable video content. This shift from a fixed, open-loop process to a flexible, closed-loop one represents a fundamental change in how we think about teaching machines to imagine the future. It moves the field away from following a script and toward a more dynamic, responsive form of creation, where the computer learns to navigate the complex landscape of video generation on its own terms.

Technical Summary: Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioning

Problem Statement

Standard autoregressive video generation algorithms, particularly those based on Diffusion and Flow Matching, rely on rigid training objectives and static sampling schedules conditioned on explicit noise levels. These methods operate in an open-loop fashion, where the noise level passed to the model at each step is predetermined by an external schedule rather than derived from the current state of the generated sample.

The authors identify a critical flaw in this paradigm: during long autoregressive rollouts, the true noise level of a sample often diverges from the scheduled noise level. This "train-inference mismatch" causes errors to accumulate over time, degrading video quality and temporal consistency. Existing attempts to close this loop by conditioning on estimated noise levels only partially mitigate the issue, suggesting that the reliance on explicit noise conditioning itself is a fundamental bottleneck for stable, high-quality autoregressive generation.

Methodology: Equilibrium Forcing (EQF)

The paper introduces Equilibrium Forcing (EQF), a framework for training and inferring denoising generative video models without noise level conditioning. EQF decouples the learning of the denoising field from the sampling process, enabling data-adaptive inference.

1. Training: Unmodulated Equilibrium Objective

Unlike standard Flow Matching (FM), which learns a separate velocity field $f_{FM}(x_\sigma, \sigma)$ for each noise level $\sigma$ , EQF learns a single, unified velocity field $f_{EQF}(x_\sigma)$ over the data distribution.

Loss Function: The training objective removes the noise level conditioning term. The network is trained to predict the velocity $v = \epsilon - x$ directly from the noisy input $x_\sigma$ , without receiving $\sigma$ as an input.
$L_{EQF} = \mathbb{E}_{x, \epsilon, \sigma} [\|f_{EQF}(x_\sigma) - v\|_2^2]$
Theoretical Justification: The authors prove that under the assumption of noise level concentration in high dimensions (where the posterior $p(\sigma|x_\sigma)$ collapses to a single value), the Bayes-optimal EQF denoiser approximates the Bayes-optimal FM denoiser pointwise. Thus, EQF can learn the same denoising field as FM without explicit conditioning.

2. Inference: Closed-Loop Adaptivity

EQF leverages the internal state of the model to drive sampling dynamics, replacing fixed schedules with closed-loop feedback.

Noise Level Estimation: A lightweight readout model $h_\omega$ is trained to predict the mode of the noise level posterior from the activations $z$ of the EQF model. This provides an internal estimate of the sample's progress toward the data manifold.
$\eta$ Warp and Attractor Dynamics: The inference procedure uses a warp function $\eta$ to modulate the step size based on the estimated noise level $\hat{\sigma}$ . By choosing $\eta(\sigma) \to 0$ as $\sigma \to 0$ , the sampling landscape is shaped into a fixed-point attractor. This allows the use of gradient-based optimizers like Nesterov Accelerated Gradient (NAG) to accelerate convergence.
Budget-Adaptive Sampling: The framework supports algorithms that reallocate the remaining sampling budget dynamically. Instead of following a precomputed grid, the step size is adjusted based on the current estimated noise level and the number of steps remaining, preventing overshooting and improving efficiency.

Key Contributions

Challenge to Noise Conditioning: The paper challenges the prevailing assumption that explicit noise conditioning is necessary for stable autoregressive video generation, demonstrating that its removal actually improves robustness by eliminating schedule-sample divergence.
Modular Framework: EQF separates training (learning a unified equilibrium field) from inference (shaping the landscape via $\eta$ warps). This contrasts with prior noise-unconditional methods like Equilibrium Matching (EqM), which bake specific loss weightings and target modulations into the training objective.
Closed-Loop Inference: The introduction of a closed-loop sampling mechanism that uses internal noise level estimates enables adaptive algorithms (e.g., NAG, budget-adaptive sampling) that are incompatible with standard open-loop, noise-conditional models.
Scalability: The authors demonstrate that EQF can be applied by finetuning large-scale pretrained Flow Matching models (1.3B and 5B parameters) with a simple objective change, avoiding the need for massive pretraining compute.

Experimental Results

The authors evaluate EQF on three autoregressive video generation benchmarks: Minecraft (action-conditioned), RealEstate10K (camera-conditioned), and Droid (robot manipulation, language-conditioned).

Performance Gains: EQF consistently outperforms standard Flow Matching, Diffusion, and other noise-unconditional baselines (like EqM and NU-FM) in terms of Frechet Video Distance (FVD) and VBench scores.
- On Minecraft (300-frame rollouts), EQF with NAG and a c-function warp achieved an FVD of 64.34, significantly lower than Flow Matching (104.13) and EqM (77.02).
- On RealEstate10K and Droid, finetuned EQF models (EQF-FT) also surpassed their Flow Matching counterparts, confirming the method's scalability.
Ablation Studies:
- Closed-Loop vs. Open-Loop: Removing noise conditioning and using closed-loop feedback is essential for performance; simply removing conditioning without adaptive inference yields suboptimal results.
- Sampler Choice: Nesterov Accelerated Gradient (NAG) significantly outperforms standard Euler or Gradient Descent samplers, but only when paired with attractor-style $\eta$ warps (e.g., c-function or linear log-SNR) that vanish at the data manifold.
- Budget Adaptivity: Algorithms that dynamically adjust the sampling trajectory based on the remaining budget and current noise estimate achieve lower FVDs, particularly under tight computational budgets.
Finetuning Efficiency: The study shows that large pretrained models can be converted to EQF via finetuning, recovering the FM velocity field while gaining the ability to perform adaptive inference.

Significance and Claims

The paper claims that Equilibrium Forcing represents a paradigm shift in video generation by demonstrating that:

Explicit noise conditioning is not necessary for high-quality, stable autoregressive generation; in fact, it can be a hindrance due to the open-loop nature of fixed schedules.
Data-adaptive inference is the key to unlocking the potential of noise-unconditional models. By allowing the sampling procedure to adapt to the state of the generation, EQF achieves superior consistency and quality in long-horizon tasks.
Modularity between training and inference allows for the use of advanced optimization techniques (like NAG) that were previously incompatible with denoising models.

The authors conclude that while EQF currently focuses on video, its principles of decoupling training objectives from inference dynamics and leveraging equilibrium landscapes offer a promising direction for future world modeling and interactive content creation, moving beyond the constraints of rigid, pre-scheduled denoising paths.