💻 computer science

SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces

This paper proposes integrating bidirectional Structured State Space Models (SSMs) as efficient temporal feature extractors in video diffusion models to overcome the quadratic computational limitations of attention mechanisms, enabling high-quality long-term video generation with significantly reduced memory consumption.

Original authors: Yuta Oshima, Shohei Taniguchi, Masahiro Suzuki, Yutaka Matsuo

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuta Oshima, Shohei Taniguchi, Masahiro Suzuki, Yutaka Matsuo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Traffic Jam" in Video AI

Imagine you are trying to teach a robot to draw a movie, frame by frame. To make the movie look smooth and logical, the robot needs to remember what happened in the previous frames and predict what comes next.

Current AI models (called Diffusion Models) are great at this, but they have a major bottleneck when making long videos. They use a mechanism called Attention. Think of Attention like a librarian who has to read every single book in a library to find the one you need.

Short Video (16 frames): The library is small. The librarian finds the book quickly.
Long Video (256 frames): The library is massive. The librarian has to read every book against every other book to find connections. The work doesn't just double; it quadruples. This creates a "traffic jam" in the computer's memory and processing power, making it very expensive and slow to generate long videos.

The Solution: The "State-Space" Express Lane

The authors of this paper propose a new way to handle the "memory" part of the AI, swapping out the librarian for a State-Space Model (SSM), specifically a type called Mamba.

Think of the SSM as a high-speed train instead of a librarian.

Instead of re-reading the whole history every time, the train carries a "summary" of where it has been.
As it moves to the next station (the next video frame), it simply updates its summary based on the new station.
The Result: Whether the train travels 16 miles or 256 miles, the time and fuel (computational cost) it takes increases in a straight, predictable line. It doesn't explode in cost like the librarian method does.

What They Actually Did

The researchers built a video generator and swapped the "Attention" engine for this new "SSM" engine. They tested it on three different video datasets:

MineRL Navigate: A robot navigating a Minecraft-like world.
GQN-Mazes: A robot solving 3D mazes.
CARLA-Town01: A self-driving car simulation.

They tested videos ranging from short clips (16 frames) to long sequences (256 frames).

The Key Findings

1. The Long Video Winner
When generating short videos (16 frames), the new SSM method and the old Attention method were neck-and-neck. However, once they tried to generate long videos (256 frames), the SSM method pulled ahead significantly.

Analogy: It's like comparing a bicycle and a sports car on a short driveway (both are fine). But on a 200-mile highway, the sports car (SSM) gets you there faster and uses less gas, while the bicycle (Attention) gets exhausted and slows down.
The Proof: The SSM models produced higher-quality videos (measured by a score called FVD) while using less memory and less time than the Attention models.

2. The Secret Sauce: Two-Way Traffic and Smart Filtering
The researchers found that just swapping the engine wasn't enough; they had to tune it two specific ways to get the best results:

Bidirectionality (Two-Way Traffic): Standard SSMs only look forward (past to future). The researchers made the model look both ways (past and future).
- Analogy: Imagine driving a car. If you only look through the windshield, you might miss a car pulling out from the side. By looking in the rearview mirror (future context) as well, the AI understands the full scene better.
Selective Scans (Smart Filtering): The model learned to ignore boring, repetitive frames and focus on important changes.
- Analogy: Imagine reading a novel. If 10 pages describe the same empty hallway, you skim them. But if a character enters the room, you read carefully. The SSM does this automatically, keeping its "memory" fresh for the important moments.

The Bottom Line

This paper proves that for making long videos, using State-Space Models (SSMs) is a smarter, more efficient choice than the traditional Attention method. It allows computers to generate longer, smoother videos without needing super-expensive hardware, essentially giving researchers a "cheat code" to bypass the memory limits that have held back video AI for a while.

Note: The paper focuses strictly on the technical architecture and performance on specific datasets. It does not claim these models are currently ready for medical diagnosis, real-time security, or commercial movie production, but rather that they are a superior architectural choice for the specific task of long-term video generation.

Technical Summary: SSM Meets Video Diffusion Models

1. Problem Statement

The research community has increasingly focused on extending diffusion models from image generation to video generation. While recent Video Diffusion Models (VDMs) have demonstrated impressive capabilities, they predominantly rely on attention mechanisms to extract temporal features across video frames. A critical limitation of this approach is that the computational cost (memory and time) of attention layers scales quadratically ( $O(L^2)$ ) with the sequence length $L$ .

This quadratic scaling poses a significant barrier to generating long-term video sequences. Consequently, many state-of-the-art methods are restricted to short sequences (e.g., 16 frames, or roughly two seconds at 8 fps). While some recent large-scale models attempt to address longer sequences, practical memory and computational constraints often limit the feasible generation length. The paper identifies the need for a temporal feature extractor that maintains high generative quality while offering linear computational complexity to enable efficient long-term video generation.

2. Methodology

The authors propose replacing the temporal attention layers in standard Video Diffusion Models with Structured State Space Models (SSMs), specifically utilizing the Mamba architecture.

Core Architecture

Baseline: The study uses a U-Net based Video Diffusion Model (VDM) as a baseline. In standard VDMs, spatial attention layers are interleaved with temporal attention layers to capture frame-to-frame dependencies.
Proposed Modification: The authors replace the temporal attention layers with Temporal SSM Layers while keeping the spatial layers (convolutional and spatial attention) unchanged.
Bidirectional Mamba: Unlike standard SSMs which are inherently causal (unidirectional), the proposed temporal layer employs a bidirectional Mamba block. This consists of two parallel branches:
- A forward branch processing frames from start to end.
- A backward branch processing frames from end to start.
- The outputs of both branches are summed before a final linear projection. This allows the model to capture temporal dynamics dependent on both past and future frames.
Selective Scan (S6): The implementation utilizes the selective scan mechanism found in Mamba. Here, the parameters governing the state space ( $B$ , $C$ , and the discretization step $\Delta$ ) are input-dependent. This enables the model to selectively discard irrelevant information and retain essential information over extended periods, adapting the state update based on the content of each frame.
Complexity: By leveraging SSMs, the memory allocation and computational time for the temporal layers scale linearly ( $O(L)$ ) with the sequence length, in contrast to the quadratic scaling of attention.

3. Key Contributions

Architectural Proposal: The paper presents the first application of bidirectional Mamba blocks specifically to the temporal axis of video diffusion models for long-term generation (≥256 frames).
Efficiency Analysis: It demonstrates that SSM-based temporal layers can achieve comparable or superior generative performance to attention-based layers while significantly reducing computational costs (memory and inference time) for long sequences.
Ablation Insights: The study provides a detailed ablation analysis showing that:
- Bidirectionality is crucial; unidirectional SSMs underperform compared to attention-based baselines.
- Selective Scanning is essential; non-selective SSMs (like S4D) perform significantly worse than selective variants in capturing temporal dependencies over long sequences.
Comprehensive Evaluation: The method is evaluated across multiple datasets (MineRL Navigate, GQN-Mazes, CARLA-Town01) and various model sizes (ranging from ~14M to ~913M parameters) and sequence lengths (16, 64, and 256 frames).

4. Experimental Results

The experiments were conducted on three datasets:

MineRL Navigate: 1,186 videos (Minecraft environment).
GQN-Mazes: 108,200 videos (3D maze simulator).
CARLA-Town01: 508 videos (driving simulator).

Key Findings:

Short Sequences (16–64 frames): SSM-based models achieve generative performance (measured by Fréchet Video Distance, FVD) comparable to attention-based models under the same computational constraints.
Long Sequences (256 frames): SSM-based models significantly outperform attention-based models. Under identical memory and time constraints, SSM models achieve a lower FVD (better quality).
Resource Efficiency: For 256-frame sequences, the SSM-based approach demonstrated approximately 1.56x faster inference and 1.56x lower memory usage compared to attention-based counterparts of the same size.
Scaling Behavior: While memory and inference time for attention models increase quadratically with sequence length, SSM-based models exhibit only a linear increase.
Ablation Results: On the MineRL 256-frame dataset, the full bidirectional selective-scan SSM achieved an FVD of 381, compared to 458 for a non-selective SSM and 610 for a unidirectional SSM.

5. Significance and Claims

The paper claims that leveraging SSMs as temporal feature extractors allows for higher generative performance at the same computational cost compared to attention-based methods, particularly for long-term sequences.

Accessibility: By reducing the computational barrier for long-term video generation, this approach enables institutions with limited computational resources to participate in cutting-edge video generation research.
Architectural Superiority: The study suggests that the choice of temporal architecture is critical; simply replacing attention with a naive SSM is insufficient. The combination of bidirectionality and selective scans is necessary to match or exceed the performance of attention mechanisms in video generation.
Future Potential: The authors note that their temporal SSM block is a "drop-in replacement" for temporal self-attention. This implies it can be combined with other existing strategies (e.g., latent diffusion, cascaded pipelines, or autoregressive generation) to further improve long-sequence generation without modifying the underlying complexity of the temporal mixing layer.

The authors remain modest regarding limitations, acknowledging that their experiments were fixed at a spatial resolution of 32×32 to isolate temporal effects, and that evaluation metrics like FVD may become less reliable for very long sequences (≥512 frames) due to the pretraining of the I3D backbone on short clips. They identify scaling to higher resolutions and developing better long-horizon evaluation protocols as necessary future work.