Technical Summary: TokenTrim

Problem Statement

Autoregressive video generation has emerged as a dominant paradigm for synthesizing long videos by iteratively conditioning new batches of frames on previously generated content. However, this approach suffers from temporal drift, a phenomenon where minor errors and inconsistencies introduced in early frames accumulate and amplify over long horizons. This accumulation leads to identity changes, structural degradation, loss of global coherence, and semantic collapse.

Recent attempts to mitigate drift have focused on strengthening long-range conditioning (e.g., temporal Key-Value caching, anchor tokens) or modifying training procedures (e.g., self-forcing with stochastic gradient truncation). The authors hypothesize that these methods remain brittle because they fail to address the root cause of inference-time error propagation: the uncontrolled reuse of corrupted latent conditioning tokens. Once a region of the latent state becomes corrupted, it is repeatedly reused in subsequent autoregressive steps, dominating attention and propagating errors forward. The system currently lacks a mechanism to assess which cached latent tokens are trustworthy.

Methodology: TokenTrim

The authors propose TokenTrim, a fully inference-time method designed to identify and remove unstable latent tokens before they are reused for conditioning. TokenTrim operates entirely in the latent space, requires no architectural changes, no retraining, and introduces negligible computational overhead.

The method consists of the following core components:

1. Motion-Stabilized Initialization

To reduce early-stage corruption, the first batch of frames is generated using FlowMo, an inference-time regularization technique that introduces motion-aware variance guidance. This produces a stable latent anchor. Subsequent batches are generated using the base autoregressive model.

2. Latent Summary Construction

At each autoregressive step $t$, the method encodes the previously generated batch ($X_{t-1}$) and the candidate current batch ($X_t$) into latent tokens. To enable direct comparison, it constructs a latent summary ($Z_{t-1}$ and $Z_t$) by averaging the spatial latent tokens across the temporal dimension (frames) of each batch.

3. Per-Token Drift Estimation

The method computes a drift score ($d_i$) for each spatial token index $i$ by measuring the Euclidean distance between the latent representations in the current and previous summaries:
$$d_i = \|Z_t(i) - Z_{t-1}(i)\|_2$$
High drift scores indicate potential corruption or semantic deviation at specific token locations.

4. Adaptive Trigger and Severity Assessment

Rather than pruning unconditionally, TokenTrim assesses whether the current batch exhibits abnormal drift.

• It identifies the top $p \cdot N$ tokens with the highest drift scores (where $p$ is a pruning fraction and $N$ is the total number of tokens).
• It calculates a drift severity score ($D_t$) as the mean drift of these top tokens.
• It maintains running statistics (mean $\mu_t$ and standard deviation $\sigma_t$) of drift severity from previously accepted batches.
• A pruning intervention is triggered only if $D_t > \mu_t + \lambda\sigma_t$, where $\lambda$ is a sensitivity hyperparameter. A warm-up period ($T_{warm}$) is used initially to accumulate reliable statistics.

5. Hard Pruning and Regeneration

When the drift criterion is exceeded:

• A spatial pruning mask is applied to the temporal Key-Value (KV) cache, removing the keys and values corresponding to the unstable token positions.
• The current batch is regenerated once, conditioned on this pruned cache.
• If the regenerated batch still violates the drift criterion, it is accepted as-is to prevent unbounded loops.
• If the drift criterion is not exceeded, the batch is accepted without modification.

TokenTrim is compatible with autoregressive frameworks utilizing self-attention with KV caching, specifically demonstrated on Self Forcing and Rolling Forcing.

Key Contributions

1. Inference-Time Drift Detection: The paper introduces a mechanism to explicitly detect unstable latent tokens during inference, shifting the focus from model capacity to information propagation control.
2. Hard Pruning of Latent Tokens: Unlike methods that modify entire spatial regions or model parameters, TokenTrim selectively removes specific corrupted tokens from the autoregressive context, preventing unreliable information from influencing future steps.
3. Training-Free Integration: The method operates without modifying model weights, training procedures, or leaving the latent space, making it compatible with existing state-of-the-art video generation frameworks.
4. Motion-Stabilized Initialization: The integration of FlowMo for the initial batch provides a stable anchor, complementing the pruning mechanism.

Experimental Results

The authors evaluated TokenTrim on the Wan2.1-1.3B text-to-video model using Rolling Forcing and Self Forcing strategies.

• Quantitative Performance (VBench):
  • On Rolling Forcing, TokenTrim improved the Final Score from 75.12% to 79.67% (+4.55%), with significant gains in Quality (+5.58%) and Semantic (+3.53%) scores.
  • On Self Forcing, the Final Score increased from 75.93% to 81.84% (+5.91%).
  • TokenTrim consistently outperformed FlowMo (an inference-time motion guidance method) and baseline autoregressive methods across all aggregated metrics.
• Human Preference Study (VideoJAM-Bench):
  • In head-to-head comparisons, TokenTrim was significantly preferred over baselines and FlowMo in temporal drift reduction (41.7% vs. 15.2% for Rolling Forcing baseline; 49.7% vs. 10.2% vs. FlowMo).
  • It also showed superior performance in motion coherence and aesthetic quality while preserving text-video alignment.
• Efficiency: TokenTrim increased wall-clock runtime by only 1.08x compared to the baseline, whereas FlowMo-adapted setups incurred a 2.18x slowdown.
• Ablation Studies:
  • Aggressive pruning (20%) severely degraded performance, while moderate pruning (10%) yielded optimal results.
  • Removing the FlowMo initialization reduced performance, confirming the complementary benefit of a stable start.

Significance and Claims

The paper claims that temporal drift is not solely a modeling or data issue but an inference-time information propagation problem. By treating the KV cache as an active, self-correcting memory mechanism rather than a passive history log, TokenTrim demonstrates that controlling the conditioning context at inference time alone can substantially reduce temporal drift.

The authors conclude that future progress in infinite video synthesis may depend not just on how much context a model can remember, but on its ability to selectively discard the noise that threatens to distort reality. They position TokenTrim as a distinct, underexplored mechanism that improves long-horizon temporal consistency without the computational cost of retraining or architectural modification.