💻 computer science

FreqForcing: Autoregressive Long Video Generation via Spectral Self-Anchoring

This paper introduces FreqForcing, a training-free framework that mitigates error accumulation in autoregressive long-video generation by employing Spectral Self-Anchoring to stabilize low-frequency components while preserving high-frequency motion, thereby enabling high-quality two-minute video synthesis from models pretrained on short clips.

Original authors: Jiatong Li, Leo Liang, Linghe Kong, Yulun Zhang

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiatong Li, Leo Liang, Linghe Kong, Yulun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can type a sentence and instantly watch a movie unfold, frame by frame, right before your eyes. This is the dream of "autoregressive video generation," a cutting-edge corner of artificial intelligence where computers don't just predict the next word in a story, but the next image in a movie. To do this, the AI acts like a storyteller who looks at the last few pictures it drew to decide what comes next. It's a bit like a game of "telephone" played with images: the computer takes the previous frame, adds a little bit of new detail, and passes it along. The catch? If the computer makes a tiny mistake in the first picture, that error gets passed down to the second, then the third, and so on. Over a long movie, these tiny mistakes pile up like a snowball rolling down a hill, eventually turning a beautiful scene into a blurry, color-drifting mess. Scientists have been trying to fix this "error accumulation" so we can generate long, stable videos without the picture falling apart.

This paper, titled "FreqForcing," tackles that exact problem by looking at video generation through a different lens: sound waves. The authors realized that when these AI movies start to go wrong, it's not just a visual glitch; it's a shift in the "frequency" of the image, much like how a song might drift out of tune. They found that previous attempts to fix this, like keeping a few "anchor" frames in memory to remind the AI of the start, helped a little but didn't stop the drift completely. The team proposed a new trick called "Spectral Self-Anchoring." Think of it as a dual-brain system for the AI: one part of the brain focuses on the fast, exciting details (like a character's quick movements), while the other part holds onto a steady, low-frequency "anchor" from the beginning of the video to keep the colors and layout from drifting away. By mixing these two signals together, they managed to stop the video from collapsing. Their method, which doesn't require retraining the AI from scratch, successfully extended a model trained on short 5-second clips to generate stable, two-minute videos—a 24-fold increase in length—without the visual quality falling apart.

The Problem: The "Telephone" Game of Video

Imagine you are playing a game of "telephone" with a friend, but instead of whispering a sentence, you are drawing a picture. You draw a cat, pass it to your friend, who adds a tail, and passes it back. If your friend accidentally draws the tail a little crooked, and you try to fix it but make it slightly bigger, and the next person makes it even bigger, soon you have a giant, distorted monster instead of a cute cat. This is exactly what happens in autoregressive video generation. The AI generates a video one chunk at a time, using the previous chunk to predict the next one. As the video gets longer, the tiny errors in color, shape, and motion accumulate. The result? The video starts to "drift." The colors might shift from blue to purple, the character might stop moving, or the whole scene might dissolve into static.

The Discovery: Listening to the "Hum" of the Video

The authors of this paper decided to investigate this drift not by looking at the pictures, but by listening to them. They used a mathematical tool called a Fourier Transform, which is like a prism for sound or images. Just as a prism splits white light into a rainbow of colors, this tool splits an image into different "frequencies."

High frequencies are like the sharp, crisp details: the flicker of a candle, the texture of fur, or a quick hand wave.
Low frequencies are like the deep bass notes: the overall shape of the room, the general color tone, and the big layout of the scene.

When they analyzed the videos that were failing, they found a strange pattern. As the video generation went on, the "energy" in the low-frequency part of the image started to drift away. It was as if the deep bass note of the song was slowly changing pitch, causing the whole video to lose its stability. The high-frequency details (the motion) also became jittery, but the root cause was this low-frequency drift.

The Old Fix: The "Anchor" That Wasn't Strong Enough

Before this paper, researchers tried to fix this by using something called an "attention sink." Imagine you are trying to remember a long story. If you only remember the last few sentences, you might forget the beginning. An "attention sink" is like keeping the first few sentences of the story in your mind permanently while you tell the rest. In video terms, the AI keeps a few of the very first, perfect frames in its memory and forces itself to pay attention to them.

The authors tested this and found that it did help. It was like having a stronger anchor; the video didn't drift as fast. However, it wasn't a perfect solution. Even with the anchor, the low-frequency energy still slowly drifted over time, and the video quality eventually suffered. The anchor was there, but it wasn't pulling hard enough to stop the current.

The New Solution: FreqForcing and Spectral Self-Anchoring

The team realized they needed a smarter way to use that anchor. They proposed a new framework called FreqForcing, which uses a technique they named Spectral Self-Anchoring (SSA).

Here is how it works, using a creative analogy:
Imagine the AI is a painter working on a massive mural.

The Local Painter (High Frequency): One painter is focused on the immediate action. They are painting the fast movements, the flickering lights, and the small details. They are great at capturing the "now," but they might get a little carried away and lose the big picture.
The Anchor Painter (Low Frequency): The second painter is a master who only looks at the very first, perfect sketch of the mural. They don't paint the new details; they just hold the steady, low-frequency "soul" of the image—the colors, the layout, and the identity of the characters.
The Mix: Instead of letting the Local Painter go rogue, the AI takes the Local Painter's work and gently mixes in the Anchor Painter's steady low-frequency signal. It's like a DJ mixing a live, energetic track with a steady, deep bassline to keep the dance floor from getting chaotic.

This "mixing" happens in the frequency domain. The AI takes the fast-moving parts from the current frame and the stable, low-frequency parts from the "anchor" frames and fuses them together. This stops the colors from drifting and keeps the layout stable, while still allowing the video to move and change naturally.

The Results: From 5 Seconds to 2 Minutes

The team tested this new method on a model that was originally trained to generate only 5-second video clips. Without their fix, the model would fall apart if you tried to make it go longer. With FreqForcing, they were able to extend the generation to two minutes. That is a 24-fold increase in length!

They compared their method to other top-tier techniques, including those that required expensive retraining of the AI. Their results showed that FreqForcing produced videos that were more consistent, had better motion, and fewer visual glitches than the competition. It managed to keep the video looking like a coherent movie rather than a melting dream.

Why It Matters

This is a big deal because it offers a way to create long, high-quality videos without needing to spend months retraining the AI models, which is incredibly expensive and energy-intensive. By simply changing how the AI "thinks" about its own history—using a frequency-based anchor—they unlocked the ability to generate much longer content. It suggests that we might not need bigger, more complex models to solve these problems; sometimes, we just need to listen to the music of the data a little more carefully.

Technical Summary: FreqForcing

Problem Statement
Autoregressive video diffusion models, such as Self-Forcing, enable real-time streaming video generation by sequentially generating frames while reusing cached key-value (KV) states. However, these models suffer from error accumulation over long horizons. As the model self-rollout over time, errors are repeatedly fed back, leading to visual collapse characterized by color drift, motion stagnation, and eventual degradation of visual quality. Existing remedies fall into two categories: training-based methods that improve stability through better distillation (computationally expensive) and training-free methods that intervene at inference via RoPE adjustments or KV cache management. However, current training-free approaches struggle to guarantee visual quality for minute-long video generation.

Methodology: Frequency-Domain Analysis and Spectral Self-Anchoring
The authors characterize the error accumulation phenomenon from a frequency-domain perspective. Through spectral analysis of both latent-space and pixel-space outputs, they observe that error accumulation manifests as a pronounced energy drift in the low-frequency bands (DC component, color tone, layout, and identity), while high-frequency components (local motion, fine details) become unstable.

While prior work established that "attention sink" (retaining initial tokens in the KV cache) alleviates this drift to some extent, the authors find it insufficient to fully resolve the issue, particularly for minute-scale generation.

To address this, the paper proposes FreqForcing, a training-free framework centered on Spectral Self-Anchoring (SSA). The methodology operates as follows:

Dual-Branch Attention: During inference, the model employs two parallel attention branches:
- Local Attention: Follows standard sliding-window causal attention with an attention sink to capture recent changes and high-frequency details.
- Anchor Attention: Maintains a separate cache of high-quality "anchor frames" generated within the model's pretrained horizon (e.g., the first 21 frames). This cache is frozen once filled and provides stable, low-frequency guidance.
Spectral Anchoring: The outputs of the local and anchor branches are fused in the frequency domain.
- The local attention output ( $A_{loc}$ ) and anchor attention output ( $A_{anc}$ ) are transformed via 3D Fast Fourier Transform (FFT).
- A spatial-temporal Gaussian low-pass filter ( $H_{lp}$ ) is applied to isolate low-frequency components.
- The fused output ( $A_{fused}$ ) is constructed by injecting the low-frequency residual from the anchor attention into the local attention:
  $\hat{A}_{fused} = \hat{A}_{loc} + \lambda H_{lp}(\hat{A}_{anc} - \hat{A}_{loc})$
- Here, $\lambda$ controls the strength of the anchoring. This process suppresses low-frequency spectral drift while preserving the dynamic motion captured in the high-frequency components of the local attention.
Temporal RoPE Alignment: To prevent unnatural artifacts caused by increasing temporal distances, the method shifts the temporal indices of both the sink frames and anchor frames to align with the current sliding window, keeping relative positions within the pretrained range.
Efficiency Optimization: To minimize computational overhead, SSA is applied only during the first two denoising steps (where global layout is established) and uses a small anchor cache size ( $N_{anc}=6$ ). This results in only a ~16.5% increase in inference latency.

Key Contributions

Frequency-Domain Characterization: The paper provides a novel analysis linking autoregressive error accumulation to spectral energy drift, specifically in the low-frequency bands.
Spectral Self-Anchoring (SSA): A training-free mechanism that fuses local and anchor attention outputs in the frequency domain to stabilize visual quality without retraining.
Extended Generation Capability: The framework extends Self-Forcing, pretrained on 5-second clips, to stable two-minute generation (a 24× extrapolation), outperforming existing training-free methods and remaining competitive with training-based approaches.

Results
Extensive experiments on the VBench-Long benchmark demonstrate the efficacy of FreqForcing:

Quantitative Performance: FreqForcing achieves state-of-the-art results among training-free methods and surpasses training-based methods in "Dynamic Degree" and "Overall Consistency" for both 60s and 120s generations. For instance, on 120s generation, it achieves a Dynamic Degree of 58.97 and Overall Consistency of 20.98, outperforming Self-Forcing (27.42 and 16.17) and Deep Forcing (41.78 and 20.80).
Qualitative Stability: Visual comparisons show that while Self-Forcing suffers from severe color drift and visual fading, and other methods exhibit motion jitter or repetitive scenes, FreqForcing produces visually stable videos with consistent color and preserved motion dynamics.
Ablation Studies: Removing SSA or the attention sink leads to significant performance drops. Hyperparameter analysis indicates that balancing the low-frequency injection strength ( $\lambda$ ) and filter width ( $\sigma$ ) is crucial to maintaining temporal consistency without inducing excessive visual repetition.

Significance
The paper claims that FreqForcing addresses a fundamental limitation in autoregressive video generation: the accumulation of errors over long horizons. By shifting the perspective to the frequency domain and introducing a mechanism to "anchor" low-frequency stability using high-quality historical frames, the method enables robust, training-free long-video generation. This bridges the gap between short-clip pretraining and the demands of applications requiring streaming, minute-scale video synthesis, such as interactive world models and embodied decision-making, without the computational cost of retraining.