SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization

1. Problem Statement

Speech-driven 3D facial animation has advanced significantly in lip synchronization and identity preservation, but it struggles with emotional expressiveness. Existing methods face two primary limitations:

• Coarse Temporal Granularity: Most approaches rely on utterance-level emotion labels (a single label for the entire sentence). This fails to capture the dynamic nature of human speech, where emotions shift, blend, and vary in intensity within a single utterance, resulting in "flat" or abruptly changing facial animations.
• Data Scarcity: High-quality datasets containing aligned emotional speech and corresponding 3D facial motion are extremely rare. Most available datasets (like EmoVOCA) provide emotional facial meshes but only neutral speech, preventing models from learning direct acoustic-to-emotion correlations.

2. Methodology

SEDTalker proposes a decoupled training strategy that separates emotion signal extraction from animation generation, utilizing Frame-Level Speech Emotion Diarization (SED).

A. Core Architecture

The framework consists of two main stages:

1. Speech Emotion Diarization (SED) Model:

   • Goal: Predict emotion categories and intensities at a fine temporal resolution (20ms) directly from raw audio.
   • Backbone: Based on SpeechBrain, utilizing a pre-trained WavLM-Base-Plus feature extractor.
   • Fine-Tuning: Employs selective fine-tuning where the first 6 Transformer layers of WavLM are frozen, and the upper 6 layers are fine-tuned for emotion recognition. This balances feature adaptation with computational efficiency.
   • Classification Head: A Multi-Layer Perceptron (MLP) with dropout and layer normalization outputs a 7-class probability distribution (Angry, Disgust, Fear, Happy, Neutral, Sad, Upset).
   • Training Strategy: Trained on a massive aggregated corpus of 58,834 utterances from 9 public datasets. To handle severe class imbalance (e.g., "Fear" is rare), the model uses inverse frequency weighting in the loss function.

2. 3D Facial Animation Model:

   • Backbone: A hybrid Transformer-Mamba architecture (based on JambaTalk). This combines Mamba layers for efficient linear-complexity sequential modeling ($O(T)$) with Transformer layers for global context.
   • Input: Raw audio (encoded via Wav2Vec 2.0) + Emotion Conditioning.
   • Conditioning Mechanism: Instead of training on emotional speech, the model is trained on neutral speech paired with emotional facial meshes. At inference, the SED model generates a temporally dense stream of emotion labels and intensity levels. These are encoded as learned embeddings and added to the audio features ($H_{cond} = W_{proj}H_{audio} + C$).
   • Output: Generates 3D facial vertex displacements (FLAME topology) at 30 FPS, ensuring lip synchronization and emotion consistency.

B. Training Strategy

• Decoupling: The animation model learns the mapping from neutral audio + emotion signals to emotional motion. This eliminates the need for paired emotional speech-face data.
• Loss Function: A composite loss includes vertex reconstruction ($L_{vert}$), velocity constraints for smoothness ($L_{vel}$), lip synchronization ($L_{lip}$), and text-level supervision ($L_{ctc}$).

3. Key Contributions

1. Frame-Level Emotion Conditioning: Formulates 3D animation as a frame-level emotion-conditioned task, enabling continuous, intra-utterance modulation of expressions, a capability absent in utterance-level approaches.
2. Data-Efficient Decoupled Training: Solves the data scarcity bottleneck by training the animation model on neutral speech while injecting emotion signals at inference time. This allows the system to adapt to new emotions without retraining.
3. Large-Scale SED Corpus: Constructed and released a frame-level emotion diarization corpus with 58,834 utterances covering 149 speakers, 4 accents, and diverse recording conditions, achieving 78.9% weighted F1 at 20ms resolution.
4. Hybrid Architecture & Efficiency: Extended the JambaTalk backbone with an additive emotion-intensity module, achieving high fidelity while reducing GPU memory usage to 13 GB for 19 states (18 emotion-intensity + neutral).

4. Experimental Results

A. Speech Emotion Diarization (SED) Performance

• Dataset: Tested on a held-out set of 7,007 utterances (1.85M frames).
• Accuracy: Achieved 78.92% frame-level accuracy and a 78.9% weighted F1-score.
• Per-Emotion Performance:
  • Disgust: Highest F1 (87.4%) due to effective class weighting.
  • Angry/Happy: Strong performance (F1 ~81%) due to distinct acoustic features.
  • Fear: Lowest performance (F1 62.0%) due to data scarcity (2% of frames) and acoustic similarity to "Sad."
• Confusion: The model primarily confuses high-arousal emotions (Happy $\leftrightarrow$ Angry) and low-arousal negative emotions (Fear $\leftrightarrow$ Sad), reflecting valence/arousal dynamics.

B. 3D Facial Animation Performance

Evaluated on the EmoVOCA dataset (neutral audio, emotional faces).

• Geometric Accuracy: Achieved a Mean Vertex Error (MVE) of $5.976 \times 10^{-3}$, outperforming the baseline FaceFormer ($6.006 \times 10^{-3}$).
• Temporal Stability: Low Motion Offset Deviation (MOD) and Acceleration Error (AE) indicate smooth, coherent motion.
• Ablation Study:
  • Removing Mamba layers degraded reconstruction accuracy and frequency alignment.
  • Removing Transformer layers slightly improved emotion-specific error but hurt global context modeling (MVE/LVE).
  • The hybrid approach proved superior to using either architecture alone.

C. Qualitative Analysis

• Continuous Control: The model successfully generates smooth transitions between emotions (e.g., Angry $\to$ Neutral $\to$ Happy) without abrupt artifacts.
• Intensity Modulation: Unlike EmoTalk (which uses static utterance-level labels), SEDTalker captures fine-grained intensity changes, producing more pronounced and synchronized facial movements (mouth, brows, jaw) that match the temporal dynamics of the speech.

5. Significance

SEDTalker represents a paradigm shift in speech-driven animation by moving from static, utterance-level emotion labeling to dynamic, frame-level emotion diarization.

• Scalability: By decoupling emotion extraction from animation training, the framework can leverage vast amounts of neutral speech data while still generating highly expressive, emotion-aware avatars.
• Realism: It addresses the "uncanny valley" of flat emotional expression, enabling digital humans to exhibit the subtle, shifting emotional nuances found in natural human conversation.
• Resource Efficiency: The hybrid Mamba-Transformer design offers a computationally efficient solution for high-fidelity generation, making it viable for real-time applications like virtual assistants and telepresence.

The code and datasets are publicly available at https://SEDTalker.github.io/.