💻 computer science

CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation

CapTalk is a novel text-guided framework that enables real-time, synchronized 3D head animation with separate and dynamic control over speaking style and emotional expression by leveraging a large-scale dataset of textual descriptions alongside driving audio.

Original authors: Xuangeng Chu, Yuan Gan, Ziteng Cui, Shuhong Liu, Jian Wang, Bing Zhou, Tatsuya Harada

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuangeng Chu, Yuan Gan, Ziteng Cui, Shuhong Liu, Jian Wang, Bing Zhou, Tatsuya Harada

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a digital puppet, a 3D head that can talk. Usually, when you feed it audio (like a recording of someone speaking), the puppet just mimics the mouth movements. It's like a ventriloquist's dummy that only moves its lips. If you want the puppet to look happy, angry, or to bob its head in a specific way, you usually have to manually program those actions or feed it a video of a real person doing it first.

CapTalk is a new system that changes the game. Instead of just listening to the audio, it also listens to text instructions you type out. Think of it as a director giving notes to an actor. You can tell the digital head, "Speak this sentence, but do it with a nervous, fast-paced style and a happy expression," or "Say it slowly, with a serious tone and big head nods."

Here is a breakdown of how it works, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Puppet

Previous methods were like a music box. You put in a song (the audio), and it played the exact same tune (the facial movements) every time. If you wanted a different "style," you had to swap out the entire music box mechanism or find a specific recording of a person who already moved that way. This made it hard to create unique characters or change the mood of a conversation on the fly.

2. The Solution: A New "Script" and a New "Library"

The researchers built two main things to solve this:

The New Library (The Dataset): They created a massive collection of 200 hours of videos from YouTube. But they didn't just save the video; they used smart AI tools to write a "script" for every clip.
- One AI listened to the audio to guess the emotion (e.g., "Is this person angry or happy?").
- Another AI watched the video to describe the physical style (e.g., "Is the mouth opening wide? Is the head nodding a lot?").
- This created a huge library where every movement is tagged with both an emotion and a style description.
The New Director (The Model): They built a model called CapTalk that acts like a translator. It takes three things:
1. The Audio: What is being said.
2. The Style Caption: A text description of how to say it (e.g., "subtle head movements," "wide mouth opening").
3. The Emotion Caption: A text description of the feeling (e.g., "neutral," "excited").

3. How It Works: The "Time-Window" Puzzle

Imagine trying to draw a long, continuous cartoon animation. If you try to draw the whole thing at once, it gets messy. CapTalk breaks the animation into small "time windows" (like short clips of 4 seconds).

The Codec (The Shrink Ray): First, the system takes the complex 3D movements of a face and compresses them into a simple code, like turning a high-definition movie into a set of digital instructions (0s and 1s).
The Autoregressive Generator (The Storyteller): The model then predicts the next set of instructions based on the previous ones, the audio, and your text notes. It's like a storyteller who knows the plot (the audio) and the character's personality (your text notes), so they can improvise the next few sentences of the story (the facial movement) perfectly.
The Magic of Text: If you change the text note from "nervous" to "confident" while keeping the audio the same, the model instantly changes the head movements and mouth shapes to match the new instruction, even in the middle of a sentence.

4. The Results: A Better Performance

The researchers tested CapTalk against other methods and found:

Better Lip Sync: The mouth moves perfectly with the words.
Better Style Control: If you ask for "big head movements," the head actually moves a lot. If you ask for "subtle," it stays still.
Realism: In tests where humans watched the videos, they preferred CapTalk over other methods because the movements felt more natural and matched the instructions better.

In Summary

CapTalk is like giving a digital actor a script that includes not just the dialogue, but also the stage directions. You can type "Speak with a dramatic, wide-eyed style," and the 3D head will instantly perform that specific style, synchronized perfectly with the audio. It moves away from rigid, pre-programmed animations to flexible, text-guided performances.

Technical Summary: CapTalk

Problem Definition

Audio-driven 3D facial animation aims to generate synchronized lip movements and expressive facial expressions from arbitrary audio inputs. However, existing methods face significant limitations in controllability and adaptability:

Rigid Style Control: Current approaches typically rely on predefined identity latent features or require reference videos to extract style embeddings. This restricts users' ability to flexibly control speaking styles without providing specific motion sequences.
Lack of Dynamic Adaptation: Applying a fixed style or identity throughout an entire audio segment often results in animations that fail to adapt to the dynamic emotional content of speech.
Data Scarcity: There is a lack of large-scale datasets that provide textual descriptions of both speaking style and emotion, which are necessary for text-guided generation.

Existing solutions often struggle to generalize to unseen identities or speaking styles, and those requiring reference videos are cumbersome for end-users.

Methodology

The authors propose CapTalk, a framework that generates 3D head motions from audio and text captions, enabling real-time synthesis of realistic and stylized animations. The methodology consists of three main components:

1. Dataset Construction: CapTalkingHead

To address the data gap, the authors constructed CapTalkingHead, a large-scale dataset containing approximately 200 hours of video (24,441 clips).

Annotation Strategy: Unlike previous datasets relying solely on identity IDs, this dataset uses natural language annotations for style and emotion.
- Emotion: Extracted from audio using an Audio-Language Model (Qwen-Audio-Chat), mapping to a predefined set of emotions (e.g., angry, happy, neutral).
- Style: Extracted from video using a Vision-Language Model (fine-tuned Qwen2.5-VL). The model focuses on head shape, mouth opening size, and head movement amplitude, ignoring background context.
Motion Data: Each clip is paired with frame-level FLAME motion parameters (shape, expression, pose).

2. Multi-Scale Codec

The authors first train a codec to create a discrete representation of the motion space.

Architecture: A transformer-based encoder-decoder with binary spherical quantization.
Process: Motion sequences are encoded into a latent space and quantized into binary, multi-scale discrete codes ( $C_{lvl}$ ). The quantization is performed residually across different temporal scales (e.g., lengths of 1, 5, 25, 50, 100 frames).
Objective: The model minimizes a hybrid loss function combining motion accuracy ( $L_1$ ), mesh vertex error ( $L_2$ ), lip region error, and a codebook stability loss ( $L_{vq}$ ).

3. Speech-to-Motion Autoregressive Generator

The core generation model is an autoregressive Transformer that operates across both temporal windows and code scales.

Inputs: The model accepts driving audio, style text captions (from video), and emotion text captions (from audio).
Feature Extraction:
- Audio: Multilingual pre-trained wav2vec2.
- Text: T5 model for style and emotion features.
Fusion Mechanism:
- Audio Conditioning: Uses Rotary Position Embedding (RoPE) to ensure precise temporal alignment between audio features and motion codes.
- Text Conditioning: Text features are fused via cross-attention layers. Unlike audio, text is not temporally aligned to specific frames; instead, it influences all code levels within a time window uniformly, allowing for global style control.
Generation: The model generates motion codes autoregressively within and across time windows, leveraging historical action information to maintain continuity even when text descriptions change.

Key Contributions

CapTalk Framework: The first model to empower users with direct, fine-grained control over both speaking style and character emotion via textual descriptions, eliminating the need for reference videos.
CapTalkingHead Dataset: The public release of the first large-scale 3D facial motion dataset with rich, multi-modal annotations for speaking styles and emotions, derived from "in-the-wild" YouTube videos.
Novel Annotation Pipeline: A method combining VLMs and ALMs to generate holistic style descriptions (mouth amplitude, head movement, emotion) that neither modality could capture alone.

Experimental Results

The method was evaluated on the CapTalkingHead test split and the MEAD dataset.

Quantitative Performance

Lip Synchronization: CapTalk achieved a Lip Vertex Error (LVE) of 6.44, outperforming baselines like ARTalk (7.71) and DiffPoseTalk (11.38).
Style Control: The model demonstrated superior performance in style control metrics, including Mean Head Distance (MHD: 1.80), Face Dynamic Deviation (FFD: 25.14), Lip Open Dynamic Deviation (LODD: 58.27), and Head Pose Dynamic Deviation (HPDD: 7.59).
Generalization: On the MEAD dataset (where the model was not trained and only received emotion labels, not style captions), CapTalk remained competitive, achieving an LVE of 8.07, comparable to state-of-the-art methods that use reference action sequences.

Qualitative and User Study

Visual Quality: Qualitative comparisons showed better alignment with ground truth in expression style, mouth dynamics, and head movements, particularly in capturing stress and accents.
User Preference: In a user study involving pairwise comparisons, CapTalk was preferred over baselines in lip sync (up to 96% preference over some baselines), style consistency, expression consistency, and pose consistency.
Ablation: Removing text captions resulted in significant performance drops in style control metrics (FFD, LODD, HPDD), confirming the necessity of textual guidance for expressive control. Style captions were found to be more critical than emotion captions for controlling motion amplitude.

Significance and Claims

The paper claims that CapTalk represents a significant step forward in making speech-driven 3D animation more accessible and controllable. By shifting from identity-based or reference-video-based control to text-guided control, the framework allows for:

Flexible Style Manipulation: Users can dynamically alter the speaking style and emotion of a character during inference using natural language.
Scalability: The model generalizes to unseen identities and styles without requiring additional training data or reference videos.
Real-time Synthesis: The autoregressive approach supports real-time generation of synchronized and expressive animations.

The authors position CapTalk as a foundational solution for applications in virtual avatars, language training, and game/film animation, while acknowledging limitations in fully continuous motion generation and cultural context understanding as areas for future work.