🤖 AI

CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation

This paper introduces CETalk, an audio-driven 3D talking head generation framework that leverages continuous Valence-Arousal representations and a multi-scale temporal modeling architecture to achieve fine-grained emotional control and accurate lip synchronization while overcoming the limitations of discrete emotion categories.

Original authors: Peng Jia, Li Dai, Zhen Xiao, Xueliang Liu, Jia Li

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peng Jia, Li Dai, Zhen Xiao, Xueliang Liu, Jia Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of digital media, the ability to create lifelike virtual humans has moved from the realm of science fiction into practical application. These digital avatars are increasingly used as virtual assistants, in virtual reality communication, and for interactive entertainment. A central challenge in this field is making these characters speak with their faces, not just their mouths. While early systems could successfully match lip movements to spoken words, they often struggled to convey the subtle, shifting emotions that make human conversation feel real. Traditional approaches treated emotions as distinct, separate boxes, such as "happy," "sad," or "angry." This method worked for broad strokes but failed to capture the fluid, continuous nature of human feeling, where a smile might slowly fade into a look of concern or where excitement might simmer quietly before bursting forth. Furthermore, the timing of speech and the timing of emotion are fundamentally different; the mouth must move quickly to match the rapid sounds of words, while emotional expressions often evolve more slowly across the entire face.

Researchers at the Hefei University of Technology have developed a new system called CETalk to solve these specific problems. Instead of forcing emotions into rigid categories, this system uses a continuous two-dimensional map to describe feelings, tracking how positive or negative a feeling is and how active or passive it feels. This approach allows the computer to generate facial animations that shift smoothly, mirroring the natural ebb and flow of human conversation. The team built a large new dataset to train their system, reconstructing 3D facial movements from existing video recordings and automatically estimating these continuous emotional values for every single frame. This data provided the necessary foundation for teaching the computer to distinguish between the fast, precise movements required for speech and the slower, sweeping changes that convey mood.

The core of the new system lies in how it handles time. The researchers realized that speech and emotion operate on different speeds, much like how a drummer keeps a steady, fast beat while a singer holds a long, slow note. To manage this, the system splits the processing into two parallel paths. One path focuses on the high-speed details of the mouth and jaw, ensuring that every syllable is synchronized perfectly with the audio. The other path focuses on the slower, broader movements of the face that express emotion, such as a furrowed brow or a widening of the eyes. These two streams of information are then brought back together by a smart integration system that decides, moment by moment, how much weight to give to the speech movements versus the emotional expression. This allows the final animation to be both accurate in its lip-syncing and rich in its emotional depth.

To test their work, the team compared their system against several existing methods using three different sets of video data. The results showed that their approach produced significantly more accurate lip movements and more realistic facial expressions than the previous best methods. In tests on the MEAD dataset, their system reduced lip movement errors to approximately 7.48 millimeters and overall facial movement errors to about 9.91 millimeters, outperforming all other tested technologies on that specific benchmark. Beyond just accuracy, the system demonstrated a superior ability to follow continuous emotional instructions. When asked to generate a face that gradually shifted from a negative state to a positive one, the system followed the path with high precision, matching the intended emotional trajectory far better than its competitors.

The researchers also demonstrated that the system allows for fine-grained control over the intensity of an emotion. By adjusting the input values, they could make a character's expression subtly more intense or more subdued without breaking the synchronization with the voice. This level of control is crucial for creating digital humans that feel truly alive, capable of the nuanced, continuous emotional shifts that define real human interaction. By moving away from discrete categories and embracing the continuous nature of human affect, this work offers a significant step forward in making virtual communication feel less like watching a machine and more like connecting with a person.

Technical Summary: CETalk – Continuous Valence–Arousal Control for Audio-Driven 3D Talking Head Generation

1. Problem Statement

Audio-driven 3D talking head generation aims to synthesize temporally coherent and visually realistic facial animations from speech. While recent methods have achieved accurate lip synchronization, they face two primary limitations in generating emotionally expressive avatars:

Discrete Emotion Limitations: Existing approaches often rely on discrete emotion categories (e.g., happy, sad). This fails to capture the continuous, subtle evolution of affect and smooth transitions that occur naturally in human conversation.
Temporal Frequency Mismatch: There is a fundamental discrepancy in temporal scales between audio articulation and emotional expression. Lip movements require high-frequency, frame-level synchronization with audio, whereas emotional expressions evolve more slowly over longer temporal spans. Existing models often struggle to decouple these dynamics, leading to animations that are either rigid or lack emotional nuance.

2. Methodology: The CETalk Framework

The authors propose CETalk, a framework conditioned on continuous Valence–Arousal (VA) representations to enable fine-grained emotion control. The system predicts a sequence of FLAME parameters ( $P$ ) given input audio ( $A$ ), speaker identity ( $S$ ), and a continuous emotion condition ( $E_{cond}$ ). The architecture consists of three core components:

Dynamic Emotion Modulation Module (DEMM):
This module bridges the gap between static emotional specifications and dynamic facial expressions. It takes the static VA condition ( $E_{cond}$ ) and modulates it using audio-derived cues.
- An expression encoder converts VA signals into a base emotional representation ( $E_{VA}$ ).
- An audio–emotion encoder (based on Emotion2Vec) extracts frame-level affective cues from the audio ( $E_{audio}$ ).
- A dynamic intensity adjustment module predicts temporal scaling coefficients ( $\alpha$ ) based on $E_{audio}$ .
- The final dynamic embedding ( $E_{mod}$ ) is obtained via element-wise scaling: $E_{mod} = \alpha \odot E_{VA}$ .
Multi-Scale Temporal Modeling (MSTM):
To address the temporal frequency mismatch, CETalk employs parallel branches to decouple high-frequency (HF) articulatory motions from low-frequency (LF) emotional dynamics.
- HF Branch: Focuses on fine-grained articulation tightly synchronized with audio. It uses cross-attention between the speaker embedding and high-resolution audio features (extracted via HuBERT).
- LF Branch: Models slowly varying emotional dynamics. It applies temporal downsampling to both audio features and the dynamic emotion embedding, performs cross-attention at a coarse scale, and then upsamples the result to the original temporal resolution.
Dynamic Fusion Mechanism (DFM):
This module adaptively integrates the HF and LF features. A lightweight gating network computes channel-wise adaptive weights ( $g$ ) based on the concatenation of both feature sets. The final unified representation ( $\tilde{F}$ ) is a weighted sum: $\tilde{F} = g \odot F_{HF} + (1 - g) \odot F_{LF}$ . This representation is then projected to predict expression and jaw coefficients.

3. Dataset Construction: 3D-VA-MEAD

To support training and evaluation, the authors constructed 3D-VA-MEAD, a large-scale dataset derived from the MEAD dataset.

Process: They utilized a pretrained emotion recognition model to estimate frame-level VA coefficients from videos and a monocular 3D face reconstruction method (EMOCA) to recover FLAME parameters.
Refinement: To mitigate temporal instability, a 1D moving average filter was applied to smooth the regressed emotion vectors.
Outcome: The dataset provides reconstructed 3D facial motions paired with frame-level continuous VA annotations.

4. Experimental Results

The model was evaluated on MEAD, RAVDESS, and HDTF datasets against state-of-the-art baselines (including FaceFormer, CodeTalker, EMOTE, and UniTalker).

Facial Motion Accuracy: CETalk achieved the lowest errors across all datasets. On MEAD, it recorded a Lip Vertex Error (LVE) of 7.4772 mm and Mean Vertex Error (MVE) of 9.9059 mm, outperforming all baselines. It also demonstrated strong generalization on the in-the-wild HDTF dataset.
Emotion Controllability: Using Root Mean Square Error (RMSE) and Sign Agreement (SAGR) metrics for VA prediction, CETalk achieved the best performance (RMSE: 0.1223 for Valence, 0.0747 for Arousal; SAGR: 0.9247 for Valence, 0.9906 for Arousal).
Qualitative Analysis: Visual comparisons showed that CETalk produces richer, more coherent expressive dynamics and smoother transitions compared to methods that exhibit over-smoothing or rigid animations.
Ablation Study: Removing the Dynamic Emotion Modulation Module (DEMM) caused the most significant performance degradation (EVE increased from 1.3256 to 2.4367), confirming the necessity of dynamic intensity modulation.

5. Significance and Claims

The paper claims that CETalk represents a significant step forward by moving beyond discrete emotion categories to a continuous affective space. Its primary contributions are:

Continuous Control: It enables fine-grained, temporally coherent emotion control via Valence–Arousal conditioning, allowing for smooth transitions between emotional states.
Temporal Decoupling: The Multi-Scale Temporal Modeling mechanism explicitly disentangles the heterogeneous temporal dynamics of audio articulation and emotional expression, addressing a core challenge in the field.
Resource Availability: The construction of 3D-VA-MEAD provides a necessary resource for future research in emotion-aware talking head generation, filling a gap in publicly available datasets with continuous VA annotations.

The authors conclude that by jointly capturing audio-driven articulation and emotional dynamics through adaptive fusion, CETalk achieves superior lip-sync accuracy and emotional expressiveness, validating the effectiveness of continuous affective modeling for 3D talking head generation.