💻 computer science

LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time

LiveAnimate is a novel real-time, billion-scale video generation system built on a 14B-parameter Diffusion Transformer that achieves stable, long-form streaming human animation with constant latency and high perceptual quality through a two-stage training pipeline and a specialized Pose-Retrieval Sink Attention mechanism.

Original authors: Yuxuan Zhang, Haozhong Xiong, Yubo Huang, Jiayi Song, Jinpeng Yu, Haofan Wang, Jiaming Liu, Ruihua Huang, Liwei Wang

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxuan Zhang, Haozhong Xiong, Yubo Huang, Jiayi Song, Jinpeng Yu, Haofan Wang, Jiaming Liu, Ruihua Huang, Liwei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to dance. You give it a photo of a person and a stream of dance moves, and you want the robot to instantly generate a video of that person performing the dance. This is the world of "pose-driven human animation." For a long time, the best robots could only do this in "offline mode." Think of it like baking a complex cake: you mix the ingredients, wait hours for it to rise, bake it, and then finally serve it. If you wanted to change the dance move, you'd have to start the whole baking process over again. This is great for making movies, but terrible for live streaming or video games where you need the robot to react right now. The big challenge has been making a robot that is both incredibly smart (to look real and stay consistent) and incredibly fast (to keep up with a live stream) without the video falling apart after a few minutes.

Enter LiveAnimate, a new system that finally bridges this gap. The researchers have built a "digital dancer" that can generate a video of a person dancing in real-time, block by block, while keeping the person's face and clothes looking exactly the same for the entire duration of the stream. It's like having a live puppeteer who never gets tired, never forgets what the puppet looks like, and can keep the show going for hours without the puppet's face melting or its clothes changing color. The paper shows that this system can run at about 20 frames per second (which feels like real-time motion) on powerful computers, maintaining high quality for at least three minutes straight—a feat that previous methods either couldn't do in real-time or would take hours to compute.

The Magic Trick: How It Works

To understand how LiveAnimate pulls off this trick, we have to look at the three main ingredients it mixes together: a super-smart brain, a special training routine, and a clever memory trick.

1. The Brain: A Giant Video Transformer
At the heart of the system is a massive AI model called a "Diffusion Transformer" (or DiT) with 14 billion parameters. Imagine this as a giant library of every possible way a human body can move and look. Usually, these libraries are "bidirectional," meaning they can look at the future and the past to understand a scene. But for a live stream, you can't look at the future; you can only react to what's happening now. The researchers had to retrain this giant brain to be "causal," meaning it only looks at the past and the present, just like a human watching a dance in real-time.

2. The Training: Two Stages of Learning
You can't just tell a 14-billion-parameter brain to "go live" and expect it to work. The paper describes a two-stage training process to get it ready:

Stage 1 (The Teacher): First, they teach the model using "Reference-Anchored Teacher-Forcing." Imagine a teacher holding a perfect script (the ground truth) and guiding the student (the AI) through the dance, block by block. The student learns to copy the moves perfectly while always keeping the reference photo in mind so the dancer looks like the right person.
Stage 2 (The Speed Run): The first stage is still too slow for real-time. So, in the second stage, they use a technique called "Block-wise Self-Forcing Distillation." This is like taking the student, letting them practice the dance on their own without the teacher, and then only correcting the current move they are doing right now. This allows the model to learn from its own mistakes without needing to remember the entire history of the dance in its memory at once. The result? The model learns to generate high-quality video in just 3 steps instead of the usual 50, making it fast enough to run in real-time.

3. The Memory Trick: The Pose-Retrieval Sink
Here is the most creative part. If you ask a computer to remember a 3-minute video, it usually runs out of memory or starts to get confused. If the dancer does a pose they did 2 minutes ago, a normal system might have forgotten what they looked like in that exact pose, leading to weird glitches or a face that looks slightly different.

LiveAnimate solves this with a clever memory system called Pose-Retrieval Sink Attention (PR-Sink). Imagine the AI has a small "sticky note" pad (a Rolling Window) that only holds the last few seconds of the dance. But it also has a special "Pose Library" (a Memory Bank) where it stores snapshots of specific poses it has seen before.

The Static Sink: This is a permanent anchor. It keeps the very first frame of the video locked in memory forever, ensuring the dancer's identity never drifts away.
The Dynamic Sink: This is the magic. When the dancer strikes a pose that matches one in the "Pose Library," the system instantly grabs the "sticky note" from that past moment and pastes it into the current view. It's like the dancer suddenly remembering, "Oh, I did this move 2 minutes ago, and I looked great then!" and instantly copying that exact look. This happens without the system needing to remember the entire 3-minute video, keeping the memory usage constant no matter how long the stream goes on.

The Results: Fast, Stable, and Real

The researchers tested LiveAnimate on a three-minute dance sequence. The results were striking. While other systems either took 2 to 5 hours to generate the same clip or started to degrade (the face would blur, the clothes would change color, or the identity would shift) after a minute, LiveAnimate kept the quality steady from the first second to the last.

Speed: It runs at roughly 19.63 frames per second on two high-end NVIDIA H100 GPUs. This is fast enough to feel like a live interaction.
Consistency: The system maintained a nearly perfect score for visual quality and identity consistency throughout the entire three minutes. In contrast, other methods saw their quality drop significantly as the video got longer.
Efficiency: The memory usage stays the same whether the video is 10 seconds or 10 minutes long, thanks to the clever "Pose Library" trick.

What It's Not (Yet)

The paper is careful to note what this system doesn't do. It currently generates video at a resolution of 480×480 pixels, which is good for a screen but not Hollywood cinema quality. It also focuses on single-person scenes and doesn't yet handle multiple people dancing together or complex camera movements. The authors suggest that pushing these limits to higher resolutions and more complex scenes is the next step for future work.

Why This Matters

LiveAnimate represents a new operating point for AI. It proves that we don't have to choose between "high quality" and "real-time speed." By combining a massive AI brain with a smart, bounded memory system, the researchers have created a tool that could power the next generation of interactive avatars, live virtual concerts, and telepresence systems where a digital human can dance, talk, and interact with you instantly, without ever forgetting who they are.

Technical Summary: LiveAnimate

Problem Statement

Pose-driven human animation aims to synthesize a video of a target person from a single reference image and a stream of driving poses. While video diffusion models (specifically Diffusion Transformers or DiTs) have achieved high fidelity in offline generation, they face three critical barriers to interactive applications like live streaming and telepresence:

Latency: Current billion-scale DiT systems require minutes to hours per clip, precluding real-time interaction.
Causality: Standard bidirectional DiTs cannot generate temporal blocks causally (one after another) without sacrificing quality or requiring full-sequence re-computation.
Long-Form Stability: Autoregressive generation over extended sequences often leads to identity drift, appearance degradation, and semantic inconsistency, particularly when poses recur after long intervals.

Existing methods either generate fixed-length clips, lack real-time capabilities, or fail to maintain stability over minute-scale rollouts.

Methodology

LiveAnimate addresses these challenges through a unified framework built on a 14B-parameter video Diffusion Transformer (DiT). The system employs a two-stage training pipeline and a novel attention mechanism to enable real-time, stable streaming.

1. Two-Stage Training Pipeline

Stage 1: Reference-Anchored Teacher-Forcing Adaptation
- Converts a pretrained bidirectional DiT into a block-causal autoregressive generator.
- Training involves partitioning videos into temporal blocks. Each block is denoised while attending to ground-truth clean history (teacher forcing) rather than previous student predictions to prevent error accumulation during adaptation.
- Introduces a Ref Sink: A global conditional cache that permanently anchors the reference image's latent representation, making it visible to all generated blocks to ensure identity consistency.
Stage 2: Block-wise Self-Forcing Distillation (BS-DMD)
- Reduces the sampling budget from 50 steps to 3 steps per block to meet real-time latency requirements.
- Builds on Self-Forcing to expose the model to its own autoregressive rollouts, reducing the train-inference gap.
- Utilizes a two-pass block-wise replay procedure:
  1. The student generates a full trajectory without gradient tracking.
  2. Blocks are replayed one at a time with gradient tracking to compute a distribution-matching loss (DMD) against the teacher.
- This approach allows the 14B model to be distilled on a single 8×80GB GPU node by avoiding the retention of the full autoregressive activation graph.

2. Pose-Retrieval Sink Attention (PR-Sink)

To maintain appearance consistency over arbitrarily long streams without linear memory growth, LiveAnimate introduces a bounded KV-cache mechanism:

Static Sink: Permanently anchors the first generated block, providing a time-invariant appearance context.
Dynamic Sink: Stores a historical block retrieved from a compact memory bank based on pose similarity. When a pose recurs, the system retrieves the relevant historical context to restore appearance details that may have left the immediate context window.
Rolling Window: A standard three-slot window (current block + two recent clean blocks).
Mechanism: The system extracts a "whole-body fingerprint" from pose signals. A memory bank (capacity $M=5$ ) is updated to maximize pose diversity. During inference, if a current pose matches a stored fingerprint, the corresponding Key-Value (KV) states are retrieved, re-indexed for position consistency (handling Rotary Position Embeddings), and injected into the Dynamic Sink.
Result: Memory and per-block latency remain constant regardless of stream duration.

3. System Optimizations

Ulysses Sequence Parallelism: Distributes attention computation across GPUs using an all-to-all pattern, enabling efficient scaling.
Operator Fusion & Compilation: Uses torch.compile with specific resolution buckets to maximize kernel efficiency.

Key Results

Evaluated on a three-minute benchmark (25 FPS) using two NVIDIA H100 GPUs:

Real-Time Performance: LiveAnimate achieves 19.63 FPS (approx. 20 FPS), completing a 3-minute sequence in ~4 minutes. In contrast, baseline offline methods require 2–5 hours.
Long-Form Stability:
- Identity: Maintains nearly constant identity consistency (DINO-S drops only from 0.833 to 0.818) over 180 seconds.
- Quality: Perceptual quality (IQA) remains stable (4.047 at start vs. 4.026 at end).
- Comparison: Baselines like One-to-All and Wan-Animate show significant degradation in identity and quality over time. EverAnimate maintains quality but lacks real-time capability.
Efficiency: The Pose-Retrieval Sink mechanism adds negligible overhead (<0.54% of runtime).

Significance and Claims

The paper claims to present the first animation system to combine real-time streaming with stable long-form generation at the billion-parameter scale.

New Operating Point: LiveAnimate establishes a new trade-off point among quality, latency, and duration, making billion-scale video diffusion models viable for interactive applications such as live streaming, telepresence, and virtual avatars.
Technical Novelty: It demonstrates that bidirectional DiTs can be effectively adapted to causal, block-wise generation with minimal quality loss through specific distillation and attention mechanisms (PR-Sink).
Scalability: The system proves that stable, minute-scale generation is possible with constant memory and latency, overcoming the linear scaling limitations of previous autoregressive approaches.

Limitations

The authors note current limitations:

Generation is limited to 480×480 resolution with 3 denoising steps.
The system does not yet support multi-person scenes or large camera motions.
Future work is required to extend these capabilities to higher resolutions and more complex dynamic scenes.