🤖 AI

Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents

This article proposes a Speech-Gesture-GAN framework that employs a conditional Generative Adversarial Network to generate realistic co-speech gesture sequences for embodied agents by learning the relationships between speech text, audio features, and joint angles from a public dataset.

Original authors: Carson Yu Liu, Gelareh Mohammadi, Yang Song, Wafa Johal

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Carson Yu Liu, Gelareh Mohammadi, Yang Song, Wafa Johal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to hold a conversation with a human. You can program the robot to speak, yet if it stands there like a statue while talking, the conversation feels awkward and robotic. Humans, on the other hand, constantly move their hands, shrug their shoulders, and point with their fingers while speaking. These movements are called gestures, and they help us express feelings, emphasize points, and make our words sound more natural.

This work presents a new "brain" for robots (and virtual characters) that learns how to move its hands in sync with what it is saying. Here is how they did it, explained simply:

The Problem: The "Uncanny Valley" in Speech

When a robot speaks without moving, it looks strange. If it moves its hands randomly, it looks like a glitched video game character. The goal is to align the robot's hand movements with the meaning of the words (such as holding up two fingers when saying "two") and the rhythm of the voice (such as a quick hand jerk when saying a short, sharp word).

The Solution: A "Copycat" Game (The GAN)

The researchers developed a system based on a concept called a Generative Adversarial Network (GAN). Imagine this as a game between two students:

The Forger (The Generator): This part of the AI tries to generate fake hand movements based on what the robot is saying. It wants to fool the teacher by making these appear as genuine human movements.
The Detective (The Discriminator): This part of the AI examines the movements and tries to figure out: "Is this a real human gesture, or did the robot just invent it?"

They play this game over and over. The forger gets better at producing realistic movements, and the detective gets better at spotting the fakes. Eventually, the forger becomes so skilled that the movements are indistinguishable from those of a real human.

The Secret: Listening and Reading

Most previous robots listened only to the sound of the voice or read only the text. The robot in this work does both, much like a musician who reads the sheet music and simultaneously listens to the conductor's baton.

The Sound (Audio): The robot hears the pitch contour and rhythm of the voice to know when to move.
The Meaning (Text): The robot reads the words to know what to move (for example, when the word is "big," the hands might spread wide).

By combining both aspects, the robot can generate gestures that fit both the rhythm of the speech and the actual meaning of the story.

The Training Camp

To teach this robot, the researchers used a special dataset called the Trinity Dataset. Imagine a professional actor standing in a room full of 20 high-speed cameras. They spoke for hours about movies, hobbies, and daily life while moving their hands naturally. The cameras recorded every joint angle of their body.

The researchers fed this data into their AI.
They removed the legs and fingers (since many robots, such as the popular NAO or Pepper, do not have complex fingers or legs that can be moved in the same way).
They focused on the spine, neck, shoulders, and arms.

Did It Work?

The researchers tested their robot in two ways:

The Math Test (Objective): They measured the smoothness and speed of the robot's hands compared to the real human actor. The robot's movements were much closer to those of the real human than those of other robots it was compared with.
The Human Test (Subjective): They showed videos of the moving robot to real humans and asked: "Does this look natural? Does it fit the speech?"
- The Result: The people could not tell the difference! Statistically, the robot's gestures were just as natural, well-timed, and meaningful as those of the real human actor.

The Conclusion

This work proves that by using a "forger-versus-detective" game and teaching the robot to listen to both the sound and the meaning of speech, we can create robots that do not just speak, but actually communicate with human body language. It is a major step toward making our digital and robotic friends appear less like machines and more like natural conversation partners.

1. Problem Statement

The paper addresses the challenge of generating natural, speech-coordinated gestures for embodied agents (such as social robots and virtual avatars). Current systems face several limitations:

Rule-based approaches: These rely on hand-defined mappings between words and gestures, leading to repetitive, monotonous interactions that lack variety and cannot adapt to physical robot constraints.
Single-modality data-driven approaches: Existing deep learning models often rely exclusively on speech text (semantic) or speech audio (acoustic). This results in information loss; for instance, text-based models miss rhythmic "beat" gestures, while audio-based models overlook semantic gestures (e.g., iconic or metaphorical gestures).
Lack of alignment: A lack of synchronization between verbal and non-verbal signals can lead to discomfort in human-agent interactions.

The core objective is to develop a framework that generates continuous, diverse, and synchronized gesture sequences from multimodal inputs (both speech text and audio) to enhance the naturalness of embodied agents.

2. Methodology

A. Dataset and Preprocessing

Dataset: The authors used the Trinity dataset, containing 244 minutes of speech and motion capture data from a single native English male speaker, recorded with 24 Vicon cameras.
Data selection: Lower body and finger data were removed to focus on upper body speech-coordinated gestures and to accommodate robots with limited dexterity (e.g., NAO, Pepper).
Feature extraction:
- Gesture: 45 features per frame (Euler angles, converted to exponential maps to avoid discontinuities) at 20 FPS.
- Audio: 26-dimensional MFCCs (Mel-Frequency Cepstral Coefficients), extracted at 20 FPS.
- Text: Semantic features extracted using BERT (768-dimensional vectors). Text features were upsampled to match the audio frame rates.

B. Model Architecture: Conditional GAN

The proposed framework is a Conditional Generative Adversarial Network (cGAN) designed to map speech inputs to gesture outputs.

Generator ( $G$ ):
- Input: Concatenation of text embeddings ( $s_t$ ), audio MFCCs ( $s_a$ ), and random noise ( $n$ ).
- Structure: A two-layer bidirectional GRU (Gated Recurrent Unit), followed by linear layers.
- Continuity mechanism: The generator integrates FiLM (Feature-wise Linear Modulation) layers, which use previously generated gesture frames as conditioning information to ensure temporal smoothness and continuity.
- Output: A sequence of joint angles (Euler angles) representing the gesture.
Discriminator ( $D$ ):
- Input: Concatenation of generated (or real) gestures, text embeddings, and audio features.
- Structure: A series of linear layers and 1D convolutional blocks (with Leaky ReLU and layer normalization) for analyzing temporal patterns.
- Output: A scalar value (0 to 1) indicating whether the gesture sequence is real or fake under the given speech condition.

C. Training Strategy

The model is trained using a combination of loss functions:

MSE Loss ( $L_{mse}$ ): Minimizes the difference between generated and ground-truth joint angles.
Continuity Loss ( $L_{continuity}$ ): Minimizes the difference in velocity between generated and real gestures to ensure smooth motion.
WGAN Loss ( $L_{WGAN}$ ): Uses the Wasserstein GAN loss to stabilize training and improve the quality of generated samples compared to traditional GANs.

3. Main Contributions

Multimodal input space: Unlike previous single-modality models, this framework simultaneously utilizes semantic features (from text) and acoustic features (from audio) to generate a diverse range of gestures (Iconic, Deictic, Metaphoric, and Beat).
Conditional GAN with noise injection: The model can generate multiple diverse gesture variations for the same speech input by varying the random noise vector ( $n$ ), thereby addressing the problem of repetitive outputs in deterministic models.
Comprehensive evaluation: The paper provides a rigorous evaluation, including:
- Objective metrics: Acceleration, Jerk, and RMSE.
- Subjective user study: Evaluation of naturalness, temporal consistency, and semantics.
- Ablation studies: Analysis of the impact of specific audio features and architectural components (e.g., GRU, FiLM, multimodal inputs).

4. Results

A. Objective Evaluation

The proposed model was compared with the state-of-the-art Gesticulator model:

RMSE: The proposed model achieved a significantly lower Root-Mean-Square Error (4.21 cm) compared to Gesticulator (13.0 cm), indicating higher geometric accuracy.
Dynamics: While the ground truth exhibited the highest values for acceleration and jerk, the proposed model (Acc: 94.48, Jerk: 2187.76) was much closer to the ground truth than Gesticulator (Acc: 63.8, Jerk: 1332), suggesting more natural motion dynamics.

B. Subjective Evaluation (User Study)

A study with 20 native English speakers evaluated the generated videos compared to the ground truth:

Criteria: Naturalness, temporal consistency, and semantics.
Statistical significance: A two-tailed t-test showed no statistically significant differences ( $p > 0.05$ ) between the proposed model and the ground truth across all three criteria.
Conclusion: Participants perceived the AI-generated gestures as natural and semantically appropriate, comparable to human performance.

C. Ablation Studies

Input features: Models using MFCCs performed best. Combining MFCCs with prosodic features offered marginal improvements, while using only text or only audio led to degraded performance (higher jerk or lower acceleration).
Architecture: Removing the FiLM conditions (previous gesture context) caused a drastic increase in jerk (5327 vs. 2187), proving its necessity for motion smoothing. Removing the GRU layer had only minor effects, suggesting that linear layers captured sufficient patterns, yet the full model remained optimal.

5. Significance

This work demonstrates that multimodal conditional GANs represent a practical and effective solution for generating speech-coordinated gestures for robots.

Practical application: The system can generate continuous, diverse, and physically plausible gestures for embodied agents, surpassing the limitations of rule-based or single-modality systems.
Human-Robot Interaction (HRI): By achieving statistical parity with human ground truth in human perception, the framework significantly enhances the potential of robots to engage in natural, non-verbal communication and reduces the "Uncanny Valley" effect caused by misalignment between verbal and non-verbal signals.
Efficiency: The system is computationally efficient and capable of generating a 30-second gesture sequence on standard hardware in approximately 12.3 seconds.