Technical Summary: Wan-Streamer v0.1

Problem Statement

Current artificial systems for human interaction often rely on cascaded, modular pipelines (e.g., separate modules for Voice Activity Detection, ASR, Language Modeling, TTS, and video generation). These architectures introduce significant latency at module boundaries, accumulate recognition and synchronization errors, and struggle to learn complex behaviors like turn management, interruption handling, and cross-modal consistency as a unified behavior.

Furthermore, real-world human interaction is fundamentally streaming and full-duplex: perception and expression overlap in time. A system that waits to finish perceiving before reasoning, or finishes reasoning before generating a response, fails to replicate natural interaction. Existing multimodal models often treat modalities asymmetrically (e.g., accepting video input but only outputting text) or rely on text as a hidden intermediate representation between separately trained components. The core challenge is to create a system where incoming speech and video immediately affect outgoing speech and motion, and where generated audio and visual states are coupled before decoding, all within a single, low-latency causal process.

Methodology

Architecture: Native-Streaming End-to-End Model

Wan-Streamer is designed from the ground up as a native-streaming, end-to-end interactive foundation model. It models language, audio, and video as both inputs and outputs within a single Transformer.

• Unified Sequence: The model processes an interleaved sequence of visual, audio, and text tokens (both input and output).
• Causal Constraints: The entire stack is strictly causal to support streaming:
  • Strictly Causal VAEs: Audio and video Variational Autoencoders (VAEs) are trained for streaming latent coding.
  • Causal Encoders/Decoders: Audio and visual encoders and decoders operate causally.
  • Block-Causal Attention: The Transformer uses block-causal attention to coordinate incremental streaming generation.
• Training Objectives:
  • Language: Optimized via next-token prediction (cross-entropy).
  • Audio/Video: Represented in continuous latent spaces and generated jointly using conditional flow matching. The model estimates velocity fields for noisy audio and video latents conditioned on the clean streaming context (past user observations and committed agent responses).

Data and Training Strategy

The model is trained on a broad mixture of data across three stages:

1. Independent-Task Pretraining: The unified Transformer is initialized from a language model and trained on a mixture of understanding (image, audio, video, dialogue) and generation (image, audio, video, joint audio-visual) tasks. This aligns perception, reasoning, and latent generation in one sequence model.
2. End-to-End Interaction Training: The model is fine-tuned on duplex interaction data where user and agent inputs/outputs (text, audio, video) are interleaved in the same causal stream. This teaches the model to update its state based on current observations, generate synchronized responses, and manage response timing and interruptions.
3. Distillation for Low-Latency: A stronger teacher model (using Classifier-Free Guidance and more flow-matching steps) is distilled into an efficient student. Rolling distillation is employed to mitigate long-horizon degradation by training the student on its own generated history.

Inference: Thinker-Performer Pipeline

To maximize hardware utilization and overlap, the single end-to-end model is deployed as a Thinker-Performer pipeline:

• Thinker: Handles causal audio/video encoding, token-causal decoding for language and state updates, KV-cache construction, and causal decoding of previous latents.
• Performer: Handles only the expensive flow-matching solver for generating the next clean audio-visual latents.
• Overlap: The system pipelines current-frame perception, previous-frame decoding, KV/latent communication, and next-frame denoising. The Thinker updates the state and decodes the previous unit's output while the Performer generates the next unit's latent. This allows the system to run in real-time as long as the Performer's generation time fits within the streaming unit duration (160 ms).

Key Contributions

1. Wan-Streamer Model: A native-streaming, end-to-end foundation model supporting text, audio, and video as both inputs and outputs within a single Transformer, eliminating reliance on external VAD, ASR, TTS, or animation modules.
2. Fully Causal Multimodal Architecture: A novel architecture comprising strictly causal audio/video VAEs, causal encoders/decoders, block-causal multimodal attention, and full-history autoregressive streaming.
3. Low-Latency Inference System: A Thinker-Performer deployment strategy that preserves unified model state via KV-cache exchange while overlapping understanding and generation, achieving sub-second total interaction latency.

Results and Performance

• Latency:
  • Model-Side Response Latency: Approximately 200 ms (measured from receiving a 160 ms user streaming unit to decoding the corresponding audio-video response).
  • Total Interaction Latency: Approximately 550 ms when combined with a 350 ms bidirectional network budget.
  • Streaming Unit: Supports streaming units as short as 160 ms at 25 FPS.
• Naturalness and Behavior:
  • The model maintains identity, gaze, posture, and subtle facial motion during idle states.
  • It produces responsive non-verbal feedback (nods, micro-expressions) temporally coupled with user speech.
  • It supports full-duplex interruption: the model can stop, shorten, or redirect speech when the user interrupts, as it continues consuming user observations while generating its own response.
  • It enables proactive speaking, initiating comments based on visual events in the input stream without explicit user prompts.
• Validation: Current results (v0.1) are validated at a 192p output resolution, serving as a proof of concept for the end-to-end streaming design.

Significance and Claims

The paper positions Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model that addresses the limitations of cascaded systems. Its primary significance lies in demonstrating that streamability is a modeling constraint rather than just a serving optimization.

By learning perception, reasoning, generation, response timing, turn management, and cross-modal synchronization jointly within one persistent interaction state, Wan-Streamer achieves sub-second full-duplex audio-visual communication. The authors argue that real-time multimodal agents must be designed as native full-duplex systems where listening, seeing, speaking, and visible response are learned jointly, rather than assembled as post-hoc modules. This approach reduces pipeline latency, prevents error accumulation, and enables more human-like, continuous interaction.