💻 computer science

UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking

UniLS is the first end-to-end framework that generates unified speaking and listening avatar expressions driven solely by dual-track audio, overcoming the stiffness of previous listener models through a novel two-stage training paradigm that learns an internal motion prior before modulating it with speech cues.

Original authors: Xuangeng Chu, Ruicong Liu, Yifei Huang, Yun Liu, Yichen Peng, Bo Zheng

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuangeng Chu, Ruicong Liu, Yifei Huang, Yun Liu, Yichen Peng, Bo Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a lively dinner party. You have two goals: you want to tell a great story (speaking), but you also want to show your friends you are listening when they talk (listening).

For a long time, computer scientists could build digital avatars that were great at telling stories. But when it came to listening, these avatars were terrible. They would just sit there like statues, staring blankly, blinking once every ten minutes, and looking completely bored. They were "stiff."

This paper introduces UniLS, a new system that finally teaches digital avatars how to be good conversationalists—both speakers and listeners.

Here is how they did it, explained simply:

The Problem: The "Stiff Listener"

The researchers realized that teaching a computer to speak is easy because the sound of the voice directly controls the mouth. If you say "Hello," your lips move. It's a direct link.

But listening is different. When someone else is talking, your face doesn't just copy their words. You nod, you blink, you raise an eyebrow, or you smile. These reactions come from inside you, not just from the sound of their voice.

When previous AI tried to learn this by just listening to audio, it got confused. It thought, "If there is no sound coming from me, I should do nothing." So, the avatar froze.

The Solution: A Two-Step Dance Lesson

To fix this, the team created a two-step training process, like teaching a dancer in two different phases.

Step 1: The "Silent Rehearsal" (Learning the Inner Rhythm)

First, they taught the AI to move without any audio at all.

The Analogy: Imagine a dancer practicing alone in a room with no music. They aren't trying to match a beat; they are just practicing how to breathe, how to shift their weight, how to blink naturally, and how to nod when they are thinking.
What the AI learned: The AI learned a "motion library" of natural, spontaneous human movements. It learned that humans blink often, tilt their heads, and fidget slightly even when they aren't speaking. This is called the Internal Motion Prior.

Step 2: The "Duet" (Adding the Music)

Once the AI knew how to move naturally on its own, they introduced the audio.

The Analogy: Now, the dancer is on stage with a partner. The partner is speaking (the music). The dancer uses their "silent rehearsal" skills (the natural blinking and nodding) but adjusts them based on the partner's voice. If the partner laughs, the dancer smiles. If the partner pauses, the dancer leans in.
What the AI learned: The AI learned to take those natural, internal movements and gently nudge them to match the conversation. It didn't stop moving; it just made the movement react to the other person.

Why This Matters

Before this, digital avatars were like actors who could only deliver a monologue. If you asked them to listen, they turned into mannequins.

UniLS is the first system that is End-to-End. This means you just give it two audio tracks (Person A talking, Person B talking), and it instantly generates the faces for both people.

Person A talks with perfect lip-sync.
Person B listens with natural nods, blinks, and expressions.

The Results

The researchers tested this against other methods, and the results were huge:

Speaking: The avatars spoke just as well as the best existing systems.
Listening: The avatars improved by 44% in naturalness. They stopped looking like statues and started looking like real people having a conversation.

The Bottom Line

UniLS is like giving a digital human a "social brain." It understands that conversation isn't just about talking; it's about the silent, subtle dance of listening, too. Now, when you talk to these avatars, they don't just hear you; they see you, and they react like a real human would.

1. Problem Statement

The paper addresses the challenge of creating conversational avatars that can naturally both speak and listen in real-time. While existing methods excel at generating speech-driven facial motions (speaking), they fail significantly when generating listening behaviors.

The Core Issue: Direct end-to-end training on "speak-listen" data leads to stiff, static listening motions (a "poker-face" effect).
Root Cause: There is a fundamental imbalance in audio-motion correlation.
- Speaking: The speaker's motion is strongly driven by their own audio (phoneme/rhythm alignment).
- Listening: The listener's motion is only weakly correlated with the other person's audio. Natural listening involves internal dynamics (blinking, subtle nods, micro-expressions) that occur independently of the external speech signal.
- Result: Standard models collapse the listening branch into a safe, low-variance static state because the audio signal provides insufficient guidance.
Limitations of Prior Work: Previous attempts at joint generation (e.g., DualTalk) rely on the interlocutor's pre-computed facial sequence as an input. This makes the system non-end-to-end, requiring an extra motion generation pipeline and preventing real-time deployment.

2. Methodology: UniLS Framework

The authors propose UniLS, the first end-to-end framework driven solely by dual-track audio (Speaker A and Speaker B) to generate unified speak-listen motions. The core innovation is a two-stage training paradigm that decouples the learning of internal motion priors from external audio modulation.

A. Representation

Motion Model: Uses FLAME parameters (expression $\psi$ , pose $\theta$ , shape $\beta$ ) to represent 3D facial motion.
Codec: Employs a multi-scale temporal VQ-VAE to convert continuous motion into discrete tokens, ensuring high-fidelity synthesis and temporal coherence.

B. Two-Stage Training Paradigm

Stage 1: Audio-Free Generator Training (Learning Internal Priors)

Goal: Learn the "internal motion prior"—the spontaneous, natural dynamics of a face (blinking, head swaying, micro-expressions) independent of speech.
Data: Unpaired, multi-scenario video data (news, interviews, casual talk) from diverse identities.
Process: An autoregressive transformer generator is trained to predict future motion chunks based only on past motion and a style embedding (capturing speaker identity). No audio is used.
Outcome: The model learns to generate diverse, non-static facial behaviors.

Stage 2: Audio-Driven Fine-Tuning (Modulation)

Goal: Teach the model to modulate the learned internal priors based on external speech cues.
Data: Paired conversational data (synchronized audio/video of Speaker A and Speaker B).
Architecture:
- The Stage 1 backbone is fine-tuned using LoRA (Low-Rank Adaptation) to preserve the learned priors.
- Dual Cross-Attention: Two new cross-attention layers are added to each transformer block:
  1. One attends to Speaker A's audio (driving A's speaking motion).
  2. One attends to Speaker B's audio (driving A's listening motion).
Process: The model learns to combine the internal motion prior with the specific audio cues to produce context-aware listening reactions (e.g., nodding when the other person emphasizes a point) while maintaining natural spontaneity.

3. Key Contributions

First End-to-End Speak-Listen Framework: UniLS is the first system to generate unified speaking and listening motions driven only by dual-track audio, eliminating the need for external motion inputs and enabling real-time application.
Two-Stage Training Strategy: A novel paradigm that solves the "listening stiffness" problem by first learning internal motion dynamics (Stage 1) and then learning audio modulation (Stage 2).
Dual Cross-Attention Mechanism: A specific architectural design that allows the model to simultaneously process the speaker's own audio and the interlocutor's audio to drive distinct motion branches.
High-Fidelity Dataset: The authors tracked FLAME parameters on the Seamless Interaction dataset, providing high-quality 3D supervision for both speaking and listening behaviors.

4. Experimental Results

The method was evaluated on the Seamless Interaction dataset against baselines like DiffPoseTalk, ARTalk, and DualTalk.

Quantitative Performance:
- Speaking: Achieved State-of-the-Art (SOTA) accuracy in lip-sync (LVE) and facial dynamics (FDD, PDD, JDD).
- Listening: Delivered a massive 44.1% improvement in distributional listening metrics (FDD, PDD, JDD, FID) compared to prior methods. This confirms the generation of diverse, non-static listening behaviors.
Qualitative Results: Visual comparisons show UniLS produces vivid, expressive, and temporally diverse listening reactions (blinks, nods, eye shifts), whereas baselines exhibit stiff, frozen faces.
User Study: In a study with 25 participants, UniLS was preferred over all baselines in 91.35% of cases for listening reaction naturalness (vs. DualTalk).
Real-Time Performance:
- Runs at 560.6 FPS on a single GPU.
- Supports real-time interaction, unlike non-end-to-end methods (e.g., DualTalk) which cannot run in real-time.
- Smaller parameter count (421.3M) compared to some baselines.

5. Significance and Impact

Solving the "Stiffness" Problem: The paper provides a theoretical and practical solution to the long-standing issue of unnatural listening avatars by redefining listening as a combination of internal priors and external cues.
Enabling Real-Time Interaction: By removing the dependency on pre-computed motion sequences, UniLS makes truly interactive, bidirectional human-AI communication feasible in real-time applications (e.g., virtual assistants, gaming, telepresence).
Foundation for Digital Humans: UniLS establishes a new standard for conversational avatars, moving beyond one-way speech generation to holistic, reciprocal social interaction.

6. Limitations & Future Work

Lack of Semantic Understanding: Current listening behaviors are driven by acoustic cues (rhythm, emphasis) rather than conversational meaning. The model cannot yet nod to agree or shake its head to disagree based on semantic content.
Temporal Discontinuities: The chunk-based generation approach may occasionally cause subtle discontinuities between motion chunks.
Future Directions: Incorporating semantic information and developing continuous long-horizon models are identified as key next steps.