Technical Summary: Instance-Specific Parametric Absorption (ISPA) for Memory-Efficient Video Generation

1. Problem Statement

Autoregressive (AR) streaming models have emerged as a leading paradigm for long video generation, enabling the synthesis of videos of arbitrary length by generating sequential chunks conditioned on historical context via a Key-Value (KV) cache. However, this approach faces a critical bottleneck: the KV cache grows linearly with the video duration, eventually saturating GPU memory and degrading inference throughput.

Existing solutions primarily rely on token dropping or pruning strategies (common in Large Language Models) to reduce cache size. The authors argue that directly applying these methods to video generation is unstable. Unlike text, where context can be sparse, video signals require dense temporal coherence. Discarding historical KV tokens breaks long-range dependencies, leading to severe artifacts such as background flickering, structural distortion, and identity loss (e.g., a character's face changing shape over time). The core challenge is to compress the KV cache without sacrificing the global consistency required for high-fidelity video synthesis.

2. Methodology: Instance-Specific Parametric Absorption (ISPA)

The paper proposes Instance-Specific Parametric Absorption (ISPA), a framework that shifts the paradigm from "discarding" tokens to "distilling" context into the model's weights. Instead of treating the KV cache as an external memory to be edited, ISPA internalizes a portion of this history into the model parameters via test-time adaptation.

Core Framework

ISPA operates through a three-stage pipeline during inference:

1. Dual-Stream Warmup:
   During the initial $T_{warm}$ frames (e.g., the first 12 latent frames), the model operates in a dual-stream configuration using Full-Attention Layers (F-Layers). It simultaneously computes:

   • $A_{full}$: The attention output utilizing the entire historical context (ground truth).
   • $A_{loc}$: The attention output utilizing only local context (current frame and "sink" tokens).
     To avoid the computational overhead of running two separate attention passes, ISPA employs a Decomposable Attention mechanism. Leveraging the Log-Sum-Exp (LSE) states tracked by hardware-aware kernels (e.g., FlashAttention), it decouples the attention into local and historical branches and fuses them post-hoc. This allows the collection of both signals with negligible FLOP overhead.

2. Layer Absorb (Weight Modulation):
   At the transition point ($T_{warm}$), the system identifies which layers can be converted to Local-Attention Layers (L-Layers). For these selected layers, the historical context is "absorbed" by solving a closed-form least-squares optimization problem. The goal is to find a weight modulation matrix $\Delta W$ such that the local stream output, when projected by the new weights, reconstructs the full stream output:
   $$\min_{\Delta W} || A_{loc}(W + \Delta W) - A_{full}W ||^2_2 + \lambda || \Delta W ||^2_2$$
   The optimal $\Delta W$ is derived analytically:
   $$\Delta W = (A_{loc}^\top A_{loc} + \lambda I)^{-1} A_{loc}^\top (A_{full} - A_{loc})W$$
   This process is non-iterative and requires no gradient-based optimization, preserving real-time inference latency.

3. Memory-Efficient Inference:
   For all subsequent frames ($t > T_{warm}$), the selected $K$ layers permanently transition to L-Layers. They evict their historical KV cache ($K_{hist}, V_{hist}$) and operate solely on local context, compensated by the updated weights $W_{new} = W + \Delta W$. The remaining $N-K$ layers remain as F-Layers to maintain global coherence.

Dynamic Layer Selection

ISPA employs Instance-Specific Selection. Rather than fixing which layers to compress, it ranks all layers based on their reconstruction error ($\epsilon$) after the warmup phase. The layers with the lowest error (those where linear weight modulation best approximates the missing history) are selected for conversion to L-Layers. This allows the model topology to adapt to the specific temporal dynamics of the video instance being generated.

3. Key Contributions

The authors identify three primary contributions:

1. Paradigm Shift: Introducing ISPA, which reframes KV cache compression from token pruning to parametric memory consolidation. It distills instance-specific context into weights, allowing layers to operate with minimal local cache while maintaining global coherence.
2. Non-Iterative Optimization: Proposing a closed-form, non-iterative solution for weight modulation. By utilizing a Decomposable Attention mechanism grounded in Online Softmax, the method collects necessary signals and updates parameters during the standard forward pass with negligible overhead.
3. Comprehensive Validation: Extensive experiments across diverse autoregressive architectures (ranging from 1.3B to 14B parameters) and tasks (text-to-video and speech-to-video).

4. Experimental Results

The paper evaluates ISPA on benchmarks including VBench-Long and MovieGen prompts, covering models like LongLive (1.3B), Reward-Forcing (1.3B), Krea-Realtime (14B), and LiveAvatar (speech-to-video).

• KV Cache Reduction: ISPA successfully removes up to 50% of the KV cache (converting half the layers to L-Layers) with near-lossless visual quality.
• Visual Quality: On 5-second and 30-second generation tasks, ISPA achieves metrics nearly identical to the full-cache baseline. Notably, on the 14B Krea model for 30-second generation, ISPA slightly improves aesthetic scores and temporal consistency, attributed to the "macroscopic regularization" effect of filtering high-frequency noise from massive historical tokens.
• Memory Efficiency: In the 14B Krea model, ISPA reduces peak GPU memory usage by 23.7GB, making long-video generation feasible on consumer-grade hardware.
• Speedup: When combined with post-training quantization (W8A8), ISPA achieves an aggregate 1.86× inference speedup compared to the full-precision baseline.
• Ablation Studies:
  • No Absorb: Removing the weight adaptation leads to catastrophic "subject mismatch" (morphing).
  • No Sink: Removing the sink frame (initial KV pairs) causes over-smoothed, low-fidelity outputs.
  • Warmup Duration: A warmup of $T_{warm} \ge 12$ frames is sufficient to capture temporal dynamics; shorter durations lead to information starvation.

5. Significance and Claims

The paper claims that ISPA offers a fundamental rethinking of how context is managed in streaming generative models. By moving beyond external token-level cache management, it demonstrates that parametric memory consolidation is a viable and efficient route for long-range dependency modeling.

The authors emphasize that their method is not merely an efficiency optimization but a mechanism that can act as a regularizer, filtering noise from long-range attention to improve stability in long-form generation. They position ISPA as a robust foundation for ultra-efficient streaming systems, compatible with quantization and adaptable to diverse autoregressive architectures, encouraging future work to explore internalizing memory rather than just managing external buffers.