💻 computer science

TS-Attn: Temporal-wise Separable Attention for Multi-Event Video Generation

The paper proposes TS-Attn, a novel training-free attention mechanism that dynamically rearranges attention distributions to resolve temporal misalignment and conflicting attention coupling, thereby significantly improving multi-event video generation quality and consistency in pre-trained models with minimal inference overhead.

Original authors: Hongyu Zhang, Yufan Deng, Zilin Pan, Peng-Tao Jiang, Bo Li, Qibin Hou, Zhiyang Dou, Zhen Dong, Daquan Zhou

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyu Zhang, Yufan Deng, Zilin Pan, Peng-Tao Jiang, Bo Li, Qibin Hou, Zhiyang Dou, Zhen Dong, Daquan Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to film a short movie based on a script that says: "A cat walks into a room, jumps on a table, knocks over a vase, and then runs out the door."

Current AI video generators are like talented but slightly confused actors. If you give them this whole script at once, they might get overwhelmed. They might show the cat running out the door before it even jumps on the table, or they might forget the vase entirely. They try to do everything at once, and the result is a chaotic mess where the timeline makes no sense.

Alternatively, if you try to film this scene by filming each action separately and stitching the clips together, the cat might look different in every clip, or the background might jump around. It's like trying to build a house by gluing together bricks from different houses; the structure falls apart.

Enter TS-Attn (Temporal-wise Separable Attention).

Think of TS-Attn as a super-smart film editor and script supervisor that sits inside the AI's brain. Its job is to make sure the AI pays attention to the right part of the script at the right time.

Here is how it works, using a simple analogy:

1. The Problem: The "Cafeteria Noise"

Imagine the AI is in a noisy cafeteria. The script (the text prompt) is a long list of instructions from different people.

Person A says: "Walk!"
Person B says: "Jump!"
Person C says: "Knock!"

Without TS-Attn, the AI hears all these voices at once. It tries to walk, jump, and knock all at the same time, or it gets confused and ignores some voices. The result is a video where the cat is doing everything simultaneously or nothing at all.

2. The Solution: The "Time-Traveling Spotlight"

TS-Attn acts like a spotlight that moves across the stage. It tells the AI:

Time 1: "Hey AI, ignore everyone else. Just listen to the person saying 'Walk'. Focus your energy on making the cat walk."
Time 2: "Okay, now the walking is done. Turn off the 'Walk' light. Now, focus only on the person saying 'Jump'."
Time 3: "Now, focus on 'Knock'."

It essentially untangles the knot of the instructions. It ensures that the visual action (the cat moving) is perfectly synced with the specific word in the script that caused it.

3. The "Motion Detective"

How does the AI know where to look?
TS-Attn has a little detective inside it. It looks at the video and says, "Okay, the cat is the thing moving right now." It puts a digital sticky note on the cat. Then, it tells the AI: "Only let the 'Jump' instruction talk to the cat. Don't let the 'Jump' instruction try to move the background or the table."

This prevents the AI from getting confused about which part of the image should change.

Why is this a big deal?

No Re-training: Usually, to fix a broken AI, you have to re-teach it from scratch (like sending a student back to school for four years). TS-Attn is like giving that student a cheat sheet or a new pair of glasses. You can plug it into existing AI models (like Wan or CogVideoX) and they instantly get smarter.
Speed: It doesn't slow the process down much. It's like adding a GPS to a car; the car drives just as fast, but it doesn't get lost.
Consistency: The cat looks like the same cat from start to finish, and the story flows logically.

The Result

With TS-Attn, the AI can finally handle complex stories. It can generate a video where a robot arm picks up a cup, pours water, and sets it down, all in one smooth, continuous shot without the cup disappearing or the water turning into fire.

In short, TS-Attn teaches the AI to listen to the story in order, rather than trying to hear the whole story at once. It turns a chaotic jumble of instructions into a clear, step-by-step movie.

1. Problem Statement

Generating high-quality videos from complex temporal descriptions containing multiple sequential actions remains a significant challenge in current text-to-video (T2V) and image-to-video (I2V) models. Existing approaches face an inherent trade-off:

Multi-prompt sequential generation: Decomposing a complex prompt into single-event prompts improves action fidelity but often leads to temporal inconsistencies, content drift, and high inference costs due to repeated model runs.
Single complex prompt: Feeding the entire description into a single pass preserves global consistency but often results in poor prompt-following, manifesting as event omission, temporal hallucinations, or the model failing to distinguish between sequential actions.

The authors attribute these failures to two root causes:

Temporal Misalignment: Video tokens (visual content) do not correctly align with their corresponding text conditions (verbs/actions) in the temporal sequence.
Conflicting Attention Coupling: Motion-related visual objects attend to multiple, unrelated event conditions simultaneously, causing interference and blurred transitions.

2. Methodology: TS-Attn

The authors propose TS-Attn (Temporal-wise Separable Attention), a training-free attention mechanism designed to be seamlessly integrated into pre-trained Diffusion Transformer (DiT) models. It operates during the early stages of the denoising process (e.g., the first 20% of steps for T2V) to inject motion information with temporal awareness.

The framework consists of three core components:

A. Motion Region Extraction (MRE)

To avoid disrupting background consistency, TS-Attn first identifies motion-related regions in the video.

It utilizes the semantic layout of the subject performing the action (derived from the prompt) to generate a semantic map.
An adaptive thresholding mechanism creates a binary mask ( $M_s$ ) that isolates the foreground subject's motion regions, filtering out scattered noise.
This mask ensures that attention modulation is applied only to the relevant moving objects, preserving background stability.

B. Event-Aware Attention Modulation

This is the core mechanism, divided into two sub-steps applied to the cross-attention layer:

Attention Rearrangement:
- The video tokens are temporally segmented (via user input, LLM planning, or uniform division) to correspond with specific events in the prompt.
- A bias function ( $B$ ) is applied to the attention matrix. It applies a positive bias to the text tokens corresponding to the current temporal event and a negative bias to tokens from other events.
- This forces motion-related video tokens in a specific time segment to focus exclusively on their corresponding action while suppressing interference from other events.
Attention Reinforcement:
- Recognizing that rearrangement alone may be insufficient if the initial attention distribution is too flat, a reinforcement factor ( $R$ ) is introduced.
- This factor adaptively scales the attention intensity based on the "sharpness" of the distribution. If the alignment between a video token and its event is weak, the reinforcement factor increases the weight, ensuring a strong temporal correspondence.

The final attention map is formulated as:
$A = \text{softmax}\left(\frac{QK^\top + M_s \odot R(Q, K) \odot B(Q, K)}{\sqrt{d}}\right)$
Where $Q$ and $K$ are query and key vectors, and $\odot$ denotes element-wise multiplication.

3. Key Contributions

Root Cause Analysis: The paper provides an in-depth analysis revealing that poor prompt-following in multi-event scenarios is caused by temporal misalignment and entangled attention correlations, proposing "temporal-wise separable grouping" as the solution.
Training-Free Framework: TS-Attn is a plug-and-play module that requires no fine-tuning of the underlying video generation model. It can be integrated into various architectures (e.g., Wan2.1/2.2, CogVideoX, Open-Sora).
Dual-Module Design: The combination of Attention Rearrangement (structural realignment) and Attention Reinforcement (adaptive scaling) effectively balances global coherence with precise event execution.
Versatility: The method supports both Text-to-Video (T2V) and Image-to-Video (I2V) tasks and handles complex scenarios involving multiple subjects and scene-level transitions.

4. Experimental Results

The authors evaluated TS-Attn on the StoryEval-Bench (a benchmark for multi-event T2V) and a newly constructed StoryEval-Bench-I2V.

Quantitative Performance:
- On Wan2.1-T2V-14B, TS-Attn improved the StoryEval-Bench score by 33.5% (from 37.6% to 50.2%).
- On Wan2.2-T2V-A14B, it achieved a 16.4% relative improvement (from 48.3% to 56.2%).
- It significantly outperformed existing multi-prompt methods like DiTCtrl and MEVG, which suffer from temporal drift and higher latency.
- Similar improvements were observed on CogVideoX-5B (57.3% relative improvement) and various I2V baselines.
Inference Efficiency:
- Unlike multi-stage methods that require $N$ inference passes for $N$ events, TS-Attn operates in a single pass.
- It incurs only a ~2% increase in inference latency compared to the baseline model, whereas multi-prompt methods often increase latency by 200-300%.
Qualitative Results:
- Visual comparisons show that TS-Attn successfully generates videos where distinct actions (e.g., "a cat watches a ball," then "dips the ball," then "takes it out") occur in the correct order without blending or omitting steps.
- It maintains physical plausibility and smooth transitions between events.

5. Significance

TS-Attn represents a significant step forward in scalable, high-quality video generation. By solving the "temporal conflict" problem without requiring expensive retraining or massive computational resources, it enables foundation models to handle complex, dynamic narratives.

Accessibility: As a training-free, plug-and-play solution, it allows researchers and developers to immediately upgrade existing open-source models (like Wan or CogVideoX) for multi-event tasks.
Future Applications: The method opens doors for interactive long-video generation, scene-level storytelling, and complex multi-subject interactions, addressing a critical bottleneck in the current generation of video AI.