Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation

1. Problem Statement

Autoregressive (AR) video diffusion models have emerged as a promising paradigm for generating long videos (minute-scale) by synthesizing frames causally. However, extending these models beyond their training horizon faces a critical challenge: progressive temporal degradation.

• Exposure Bias: Even with training strategies like Self Forcing (which conditions the model on its own predictions during training), inference still involves rolling out generation where residual errors accumulate over time.
• The Memory Bottleneck: Existing approaches attempt to mitigate this by increasing memory capacity (retaining more historical frames) or compressing memory. However, the authors argue that the core issue is not the quantity of memory, but how temporal memory is utilized.
• Key Observation: Empirical analysis reveals that:
  1. Simply increasing the number of historical frames (dense memory) does not consistently improve quality; it often introduces redundancy that constrains motion dynamics.
  2. The temporal placement of historical frames matters significantly. Different parts of the history serve different functional roles (e.g., global stability vs. short-term continuity), which are currently treated as a homogeneous buffer.

2. Methodology: Relax Forcing

The paper proposes Relax Forcing, a training-free inference strategy that replaces the standard dense chronological memory buffer with a structured, sparse KV-memory mechanism.

Core Concept: Decomposition of Temporal Memory

Instead of attending to all past frames, Relax Forcing decomposes the temporal context into three distinct functional roles:

1. Sink (Global Anchors): A fixed set of early frames (e.g., the first few frames) that provide long-term identity and scene stability.
2. Tail (Recent Context): A small set of the most recently generated frames that ensure short-term continuity and smooth transitions.
3. History (Dynamic Mid-Range): A dynamically selected subset of intermediate frames. These are chosen to provide structural motion guidance without introducing redundancy.

The Selection Mechanism

The "Relax" aspect refers to the scoring mechanism used to select the History frames from a candidate pool (typically the latter half of the middle region). For each candidate frame $h$, a relaxation score $r(h)$ is computed:
$$r(h) = S(h) - \lambda R(h)$$
Where:

• $S(h)$: Stability Score. Measures the similarity between the candidate's key prototype and the Sink prototype. High similarity ensures global consistency.
• $R(h)$: Redundancy Score. Measures the similarity between the candidate's key prototype and the Tail prototype. High similarity indicates redundancy with recent context.
• $\lambda$: A hyperparameter balancing the trade-off between global stability and local redundancy.

The top-$K$ frames with the highest scores are selected to form the History set. The final conditioning context is $M_i = S \cup H_i \cup T_i$.

Positional Embedding Adjustment

Since the selected memory is non-contiguous in time, standard sliding-window Rotary Position Embedding (RoPE) is unsuitable. The authors introduce a Hybrid Positional Indexing scheme:

• Tail: Uses absolute frame indices to preserve real temporal location.
• Sink & History: Assigned relative indices immediately preceding the Tail segment. This allows the model to leverage long-range anchors and mid-range structure without forcing them to appear temporally adjacent to the latest frames, preserving the "distance" in the attention mechanism.

3. Key Contributions

1. Systematic Analysis of Temporal Memory: The paper identifies that dense historical conditioning is a bottleneck for long-horizon motion evolution. It demonstrates that memory effectiveness depends on both quantity and temporal placement, not just capacity.
2. Relaxed KV Memory: A novel, training-free sparse memory mechanism that decomposes context into Sink, Tail, and dynamically selected History. This reduces over-reliance on rigid past frames and enables flexible long-term generation.
3. Performance Gains: The method significantly improves motion dynamics and temporal consistency while reducing computational overhead (attention length).

4. Experimental Results

The method was evaluated on the VBench-Long benchmark using 30-second and 60-second video generation tasks.

• Quantitative Performance:
  • Overall Score: Relax Forcing achieved the highest average score (80.87% for 30s, 80.88% for 60s), outperforming strong baselines like Deep Forcing and Rolling Forcing.
  • Dynamic Degree: The most significant improvement was in Dynamic Degree (65.67% vs. 57.56% for Deep Forcing), indicating much better motion evolution and less "frozen" or repetitive behavior.
  • Consistency: Subject and Background Consistency remained competitive, proving that increased motion does not come at the cost of visual identity drift.
• Efficiency:
  • By reducing the attention window from a dense 21 frames to a structured 7 frames (2 Sink + 1 History + 1 Tail + current block), the method achieved a 1.58x speedup in self-attention computation and a 1.26x end-to-end acceleration.
• Human Evaluation:
  • In a user preference study, Relax Forcing was preferred over baselines in Motion Quality (65.4%) and Overall Quality (53.8%), confirming that the structured memory produces more natural and coherent long-term motion.

5. Significance

• Paradigm Shift: The work shifts the focus from "how much memory to keep" to "which memory to keep and how to structure it." It proves that structured sparsity is superior to dense accumulation for long-horizon generation.
• Training-Free: Unlike many previous methods that require complex retraining or rollout-aware training objectives, Relax Forcing is a plug-and-play inference strategy compatible with existing Self-Forcing models.
• Scalability: By reducing attention overhead and mitigating error accumulation, this approach provides a scalable path toward generating videos of hour-long durations, addressing a fundamental limitation in current autoregressive video diffusion models.

In summary, Relax Forcing solves the long-video drift problem not by memorizing more history, but by intelligently curating a sparse, functionally diverse set of historical frames that balance global stability with dynamic motion evolution.