💻 computer science

InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars

InteractiveAvatar is a real-time, infinite-streaming video generation framework that leverages autoregressive distillation, a Long-Short Visual Memory mechanism, and a Reasoning-Reaction Module to achieve state-of-the-art visual consistency and intent-aware interactions for audio-driven avatars.

Original authors: Quanyue Song, Yishan He, Yanfei Zhang, Shihao Cheng, Zhixiang He, Zhizhi Guo, Chi Zhang, Xuelong Li, Caigui Jiang

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quanyue Song, Yishan He, Yanfei Zhang, Shihao Cheng, Zhixiang He, Zhizhi Guo, Chi Zhang, Xuelong Li, Caigui Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a digital friend on a video call. Usually, these digital friends are a bit like broken puppets: they might move their lips perfectly to match your voice, but if you talk to them for too long, their face might start to look weird, or they might forget what they looked like five minutes ago. Also, if you ask them to "check the time," they might just say the time out loud but keep staring blankly, unable to actually look at a watch.

The paper InteractiveAvatar introduces a new way to build these digital friends so they can talk to you forever without looking glitchy, and so they can actually do things you ask them to do.

Here is how they did it, explained with simple analogies:

1. The Problem: The "Amnesia" and the "Robot"

The authors say current digital avatars have two main problems:

The "Drifting" Face: If you keep the video running for a long time, the avatar's face slowly starts to change. Maybe the eyes get bigger, or the hair color shifts. It's like a painter who keeps adding new layers of paint without looking at the original sketch; eventually, the picture looks nothing like the person you started with.
The "Blind" Robot: Current avatars are mostly just "audio-driven." If you say "Look at the watch," they hear the sound and move their mouth, but they don't understand the meaning. They don't actually turn their head to look at a watch because they don't "think" about your request.

2. The Solution: A Two-Part Brain

To fix this, the team built a system with two special tools: a Memory System and a Thinking Brain.

The Memory System: "The Short-Term Notebook and the Long-Term Photo Album"

To stop the avatar's face from drifting, they created something called Long-Short Visual Memory (LSVM).

The Short-Term Notebook: Imagine the avatar keeps a notebook of the last few seconds of the video. This ensures that when it moves its head, the movement looks smooth and natural, not jerky.
The Long-Term Photo Album: This is the clever part. Instead of trying to remember every single frame (which would be too heavy and slow), the avatar has a "Photo Album" of the most important moments.
- How it works: As the video plays, the system constantly asks, "Is this new frame important?" If the avatar puts on a hat or picks up a coffee cup, the system takes a "snapshot" and puts it in the album. If the avatar is just sitting there talking, it might skip the snapshot.
- The Benefit: This acts like a anchor. Even after 10 minutes of talking, the avatar can look back at its "album" to remember, "Oh right, I'm wearing a hat," or "I have a coffee cup." This keeps the character looking consistent from start to finish.

The Thinking Brain: "The Director and the Stage Manager"

To make the avatar understand what you want, they added a Reasoning-Reaction Module (RRM).

The Director (LLM): When you speak, a powerful AI "Director" listens to you. It doesn't just hear words; it understands your intent. If you say, "Check the time," the Director realizes, "Ah, the user wants the avatar to look at a watch."
The State Cycling: The system switches between two modes:
1. Action Mode: The avatar is busy doing something (like looking at a watch or sitting down).
2. Stable Mode: Once the action is done, the avatar settles into a calm pose (like sitting quietly) so it doesn't keep moving unnecessarily.
The Fast Switch (Cache-Switching): Usually, when an instruction changes, the computer has to recalculate everything, which takes time. This system uses a "Fast Switch" trick. It keeps a backup of the calculations for the current scene. When the instruction changes (e.g., from "stand up" to "sit down"), it instantly swaps the old backup for the new one, making the reaction feel instant and smooth.

3. The Result: A Real-Time, Infinite Conversation

By combining these tools with a special training method (called Distillation, which is like teaching a student to solve a problem in one step instead of ten), they created a system that:

Runs in Real-Time: You can talk to the avatar, and it responds immediately, without long waiting times.
Lasts Forever: You can have a conversation for hours, and the avatar will still look like the same person with the same clothes and accessories.
Understands You: If you ask it to perform an action, it actually does it, rather than just talking about it.

In summary: InteractiveAvatar is like giving a digital puppet a long-term memory so it doesn't forget its face, and a brain that understands your jokes and requests, allowing for a smooth, endless, and realistic conversation.

Technical Summary: InteractiveAvatar

Problem Statement

Recent diffusion-based models have advanced audio-driven avatar generation, enabling realistic lip synchronization and appearance. However, existing approaches face two fundamental limitations in real-time streaming scenarios:

Visual Temporal Inconsistency: Current methods typically rely on causal attention with a limited receptive field (few previous chunks and a reference image). As generation extends over long durations, this leads to "temporal drift," where adjacent video chunks diverge, causing inconsistencies in identity, objects, and scene stability.
Lack of Intent-Aware Interaction: Most systems follow a simple audio-driven paradigm where Large Language Models (LLMs) generate speech, and the avatar animates lips and coarse gestures based on audio-motion correlations. These systems fail to explicitly perceive user intent to generate fine-grained, semantically aligned actions (e.g., looking at a watch when asked for the time) or to adapt the scene dynamically, limiting the realism of interactive behavior.

Methodology

The authors propose InteractiveAvatar, a real-time infinite-streaming video generation framework designed to address these challenges through three core components:

1. Long-Short Visual Memory (LSVM)

To maintain visual consistency over arbitrarily long durations, the authors introduce a memory mechanism that flexibly compresses historical visual information:

Dual-Memory Structure: The system maintains a Short-Term Memory ( $M_s$ ) containing dense representations of recent frames (e.g., the last 5 seconds) to ensure local temporal coherence. It also maintains a Long-Term Memory ( $M_l$ ) with a fixed capacity of compact tokens representing globally salient visual states.
Memory Compression: A lightweight model compresses video latents into compact tokens without temporal downsampling, preserving the ability to separate and update latents during streaming.
Dynamic Key-Frame Selection (DKFS): During inference, when a frame leaves the short-term window, it is evaluated for long-term retention. The system uses SigLIP2 to extract semantic features and calculates a global redundancy score. A candidate frame is inserted into long-term memory only if it reduces the overall semantic redundancy, ensuring the memory stores diverse and representative key frames rather than uniform time-sampled frames.

2. Reasoning-Reaction Module (RRM)

To enable intent-aware interactions, the framework integrates an LLM-based reasoning center:

State-Cycling Strategy: The LLM processes user input and the current stable state to output three components: an Action Prompt (guiding body motion/pose), Response Audio, and a Stable State Prompt (defining the static visual condition post-action). The diffusion model conditions on the action prompt and audio during the response, then switches to the stable prompt once the audio ends, ensuring the avatar returns to a consistent state without motion drift.
Cache-Switching Mechanism: To reduce latency during state transitions, the system re-encodes the updated text conditions and recomputes the Key-Value (KV) caches for the immediately preceding chunks. This allows the model to rapidly realign with new action prompts without waiting for full re-generation.

3. Autoregressive Distillation Adaptation

To achieve real-time performance, the authors adapt a pre-trained Diffusion Transformer (DiT) backbone (Wan 2.2 5B) through a four-stage training pipeline:

AI2V Pretraining: Training on large-scale audio-visual data with bidirectional attention.
Memory Compression Pretraining: Training the compression module on video reconstruction.
ODE Initialization: Training with block-wise causal attention to approximate the teacher model's ODE trajectories.
Self-Forcing DMD: Applying Distribution Matching Distillation (DMD) to distill the diffusion model into a few-step autoregressive generator. This minimizes the reverse KL divergence between the student and teacher distributions, enabling stable, low-step inference.

Key Contributions

InteractiveAvatar Framework: A novel real-time streaming audio-driven avatar generation system capable of long-duration synthesis with strong visual consistency and intent-aware interaction.
Long-Short Visual Memory (LSVM): A mechanism that preserves historical visual representations via dynamic key-frame selection, significantly improving visual coherence and preventing temporal drift in long sequences.
Reasoning-Reaction Module (RRM): A module equipped with State-Cycling and Cache-Switching that enables the avatar to interpret user intent, generate coordinated speech and actions, and execute commands with reduced latency.

Experimental Results

The authors evaluated InteractiveAvatar against state-of-the-art methods (e.g., OmniAvatar, WanS2V, LiveAvatar) across 500 video test cases involving diverse interaction scenarios.

Quantitative Performance: InteractiveAvatar achieved state-of-the-art results in Object-level consistency (OBJ) (85.2) and Text-Video alignment (TV) (25.93), with competitive Identity preservation (ID). While video quality metrics (IQA, FID) were comparable to baselines, the method demonstrated superior temporal stability.
Real-Time Efficiency: Leveraging the lightweight 5B model and distillation, the system achieved 26.68 FPS, outperforming all baselines (the next fastest, LiveAvatar, achieved 21.94 FPS).
Qualitative Analysis: Visual comparisons showed that while baselines suffered from object shape/color changes, abrupt camera shifts, or failure to follow action instructions, InteractiveAvatar maintained consistent object appearance (e.g., a watch) and correctly executed complex actions (e.g., sitting, checking a watch) aligned with user intent.
Ablation Studies: Removing the LSVM module caused a significant drop in object consistency (OBJ dropped from 85.2 to 78.4). Removing the RRM components resulted in a loss of intent-aware actions, reverting the system to simple verbal responses. Removing DMD distillation caused inference speed to plummet to 1.27 FPS.

Significance

The paper claims that InteractiveAvatar bridges the gap between high-quality avatar generation and realistic, interactive digital humans. By jointly solving the challenges of long-range visual consistency and intent-aware responsiveness, the framework provides a practical foundation for next-generation immersive applications. The authors emphasize that their approach enables avatars to not only speak but also perform contextually relevant actions and maintain visual stability over extended interactions, moving beyond simple reactive systems toward fully intent-aware digital agents.