💬 NLP

From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation

This paper evaluates four families of discrete speech representations for 3D facial animation, demonstrating that encoding phonetic classes yields accurate predictions and enabling the introduction of an Audio Visual Text-to-Speech (AVTTS) pipeline that utilizes shared discrete representations to simultaneously decode speech and 3D facial motion.

Original authors: Pedro Correa, Olivier Perrotin, Samir Sadok, Paula Costa, Thomas Hueber

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pedro Correa, Olivier Perrotin, Samir Sadok, Paula Costa, Thomas Hueber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to talk and move its face at the same time. You have the robot's voice, but how do you tell its face what to do? Do you give it a detailed script of every muscle movement? Do you give it a raw audio file? Or do you give it a set of simple, coded instructions?

This paper, titled "From Tokens to Faces," is essentially a taste test. The researchers wanted to find out which type of "instruction code" works best for making a 3D animated face look natural when it speaks.

The Four Types of "Instruction Codes"

The researchers tested four different ways to translate human speech into data (called "representations") that a computer can understand:

The "Semantic" Code (HuBERT): Think of this like a translator who understands the meaning of the sentence. It knows what words are being said and the context around them. It's great at understanding the "idea" of the speech.
The "Acoustic" Code (WavTokenizer): This is like a high-speed audio compressor. It focuses purely on the sound waves and how they feel, ignoring the meaning or the specific letters. It's very efficient at recreating the sound, but it doesn't really "know" about words.
The "Hybrid" Code (SpeechTokenizer): This is a mix of the two. It tries to understand both the meaning of the words and the texture of the sound at the same time.
The "Label-Based" Code (CosyVoice2): This is like a strict teacher using a checklist. It breaks speech down into specific, discrete labels (like specific letters or sounds) and focuses on the structure of the language rather than the fluid sound.

The Experiment: The Face Decoder

The researchers took these four different codes and fed them into two different "face drivers" (decoders):

The GRU: A step-by-step driver that looks at one frame of the face at a time.
The Transformer: A driver that looks at the whole sentence at once, like reading a paragraph to understand the flow.

They then measured how well the resulting animated faces matched the real human faces using three methods:

Math: Measuring the distance between the robot's lips and the human's lips.
Smoothness: Checking if the face was jittery or moving smoothly.
The "Human Eye" Test: They had real people watch the videos and rate how realistic they looked, similar to a blind taste test.

What They Found

Here are the surprising results, explained simply:

Meaning Matters: The "Semantic" code (the translator) and the "Label-Based" code (the checklist) were the winners. They produced the most realistic faces. It turns out that for a face to move naturally, it needs to know what is being said (the phonetic classes), not just how the sound waves vibrate.
Too Much Sound is Bad: The "Acoustic" code (pure sound) and the "Hybrid" code (sound + meaning) performed poorly. It seems that if the computer is too focused on the raw sound details, it gets confused about how to move the lips. The "noise" of the sound actually got in the way of the facial animation.
The Best Driver: The "Transformer" driver (the one that looks at the whole sentence) worked much better with the discrete codes than the step-by-step driver.
The "Label-Based" Surprise: The "Label-Based" code, which was originally designed for text-to-speech, worked almost as well as the complex "Semantic" code. This is a big deal because it's simpler and more structured.

The Big Idea: The "All-in-One" Machine

The most exciting part of the paper is a new idea they call AVTTS (Audio Visual Text-to-Speech).

Usually, to make a talking head, you have to do it in two steps:

Turn text into audio.
Turn that audio into a moving face.

The researchers showed that because the "Label-Based" code is so good at representing speech, you can use it as a shared language. You can feed text into the system, and it can spit out both the audio and the moving face at the exact same time, perfectly synchronized, without needing a middle step. It's like a conductor who can direct both the orchestra (the voice) and the dancers (the face) from a single sheet of music.

The Bottom Line

To make a robot face look real, you don't need to feed it every tiny detail of the sound wave. You need to feed it a clear, structured understanding of the sounds and letters being spoken. If you give the computer a clear "map" of the speech (like a checklist of sounds), it can figure out how to move the lips perfectly, even if it doesn't have the raw audio file to start with.

Technical Summary: From Tokens to Faces

Problem Statement
Speech-driven 3D facial animation requires selecting an appropriate speech representation to drive temporal decoders. While recent state-of-the-art (SOTA) systems predominantly rely on continuous hidden representations from self-supervised learning (SSL) models (e.g., wav2vec 2.0, HuBERT), it remains unclear which specific properties of speech representations are necessary for accurate and natural facial animation. Furthermore, the suitability of discrete speech representations (tokens) for this task is largely unexplored. Existing discrete approaches often utilize hybrid semantic-acoustic representations (e.g., SpeechTokenizer) but have not systematically compared fully acoustic, fully semantic, or label-based discrete representations against continuous SSL baselines for 3D facial synthesis.

Methodology
The authors present a comparative experimental framework evaluating four distinct families of speech representations across two temporal facial decoder architectures:

Speech Encoders:
- HuBERT (HB): A continuous SSL representation (Semantic).
- SpeechTokenizer (ST): A hybrid representation combining semantic distillation (HuBERT) with acoustic residual vector quantization (Semantic+Acoustic).
- WavTokenizer (WT): A highly compressed, single-codebook representation focused on acoustic reconstruction (Acoustic).
- CosyVoice2 (CV2): A supervised encoder trained with an Automatic Speech Recognition (ASR) pretext task using character and prosodic labels (Label-based).
Facial Decoders:
- GRU: A frame-by-frame recurrent network (used in the FaceDiffuser baseline).
- Transformer: A non-causal architecture processing full sequences via self-attention.

The models were trained on the BEAT2 dataset (27 hours of English speech aligned with FLAME parameters, converted to 51-dimensional ARKit blendshapes). The encoders were frozen using pre-trained weights, while decoders were trained from scratch.

Evaluation Strategy
The study employed a multi-faceted evaluation approach:

Objective Metrics: Lips Vertex Error (LVE) for geometric accuracy, Jitter Score for temporal smoothness (acceleration), and a novel Bilabial Closure Score (BCS) to measure articulatory accuracy during bilabial phonemes (/b/, /p/, /m/).
Perceptual Evaluation: A MUSHRA-like protocol with 30 participants rating animation realism and lip-speech coherence against a reference video.
Probing Analyses: To understand the information content of the representations, the authors conducted:
- Phonetic Probing: Measuring the co-occurrence entropy between token IDs and phoneme labels.
- Viseme Probing: Measuring the co-occurrence entropy between token IDs and discretized facial poses (visemes).
- Continuous Probing: Using Ridge regression to predict continuous blendshape values from token sequences ( $R^2$ metric).

Key Results

Performance: The Label-based representation (CosyVoice2) paired with a Transformer decoder ([CV2+T.]) achieved the best objective scores and was perceptually indistinguishable from the Semantic baseline (HuBERT + GRU).
Decoder Impact: Discrete representations performed significantly better when paired with a Transformer decoder compared to a GRU. Conversely, the continuous HuBERT representation showed equivalent performance across both decoders.
Hybrid Limitations: The Semantic+Acoustic representation (SpeechTokenizer) failed to provide satisfactory facial animation, performing comparably to purely acoustic representations. This suggests that low-structured acoustic information may be detrimental to facial prediction, even if phonetic information is present.
Probing Insights:
- Phonetic Encoding: Semantic+Acoustic representations encoded phonetic classes most efficiently, followed by Semantic, Label-based, and Acoustic.
- Viseme/Continuous Encoding: Label-based representations showed the strongest correlation with discrete visemes. However, all tokenized representations exhibited low $R^2$ scores for predicting continuous blendshapes compared to continuous representations, reflecting the inherent difficulty of mapping discrete tokens to continuous motion.
Correlation: Perceptual ratings correlated more closely with the Bilabial Closure Score (BCS) than with traditional reconstruction metrics like LVE.

Key Contributions

Systematic Comparison: A comprehensive evaluation of four speech representation families (Semantic, Semantic+Acoustic, Acoustic, Label-based) across two decoder architectures for 3D facial animation.
Probing Analysis: An investigation linking tokenized representations to both phonetic units and articulatory deformations (visemes and blendshapes), revealing that phonetic class encoding is beneficial but must be paired with the absence of low-structured acoustic noise.
AVTTS Pipeline: The introduction of a proof-of-concept Audio Visual Text-to-Speech (AVTTS) pipeline. Leveraging the fact that TTS systems (like CosyVoice2) already generate discrete tokens, the authors demonstrated a unified framework where a single shared token sequence is decoded simultaneously into speech audio and 3D facial motion, eliminating the need for a separate audio-driven animation stage.

Significance
The paper claims that phonetic class information is a necessary condition for accurate facial animation, provided it is not contaminated by low-structured acoustic data. It demonstrates that label-based discrete representations are competitive with established SSL continuous baselines. Furthermore, the work establishes that discrete speech tokens can serve as a shared latent space for both speech and facial synthesis, paving the way for unified multimodal generation pipelines that bypass the conventional two-stage (Text $\to$ Audio $\to$ Face) paradigm.