💬 NLP

Modeling Turn-Taking with Semantically Informed Gestures

This paper introduces the DnD Gesture++ dataset with 2,663 semantic gesture annotations and demonstrates that a Mixture-of-Experts framework integrating these semantically informed gestures significantly improves multimodal turn-taking prediction compared to baselines relying solely on text and audio.

Original authors: Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Varsha Suresh, M. Hamza Mughal, Christian Theobalt, Vera Demberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting around a table with friends playing a board game. The conversation is flowing fast. Someone is telling a story, and you can feel the moment coming when they are about to stop and let someone else speak. How do you know?

You don't just listen to their words. You watch their face, you notice their hand movements, and you sense their body language. Maybe they pause, look at you, and raise a hand as if to say, "Okay, your turn." Or maybe they keep talking while waving their hands wildly to emphasize a point, signaling, "I'm not done yet!"

This paper is about teaching computers to understand these invisible signals, specifically hand gestures, to predict who should speak next in a conversation.

The Problem: Computers are "Deaf" to Hands

Most computers that try to predict when a conversation will switch speakers only listen to words (text) and tone of voice (audio). They are like people wearing noise-canceling headphones and blindfolds; they can hear the words, but they miss the visual cues that humans rely on heavily.

The researchers asked: What if we taught the computer to "see" the gestures?

The Solution: A New Dataset (DnD Gesture++)

To teach the computer, they needed a massive library of examples. They took an existing dataset of people playing Dungeons & Dragons (a role-playing game where people talk a lot) and added a special layer of detail.

Think of this like taking a movie and adding subtitles that describe the actors' emotions and hand movements.

They watched 6 hours of video.
They labeled over 2,600 specific hand movements.
They categorized them into four "flavors" of meaning:
1. Iconic: Drawing a shape in the air (e.g., "It was this big!").
2. Metaphoric: Showing an abstract idea (e.g., "The future is a path").
3. Deictic: Pointing at something (e.g., "Look at that!").
4. Discourse: Rhythmic movements that organize the story (e.g., "First, then, finally").

The Brain: The "Mixture of Experts"

The researchers built a smart computer model called a Mixture of Experts (MoE). Imagine a team of three specialists working together to decide who speaks next:

The Linguist: Reads the words.
The Musician: Listens to the tone and pauses.
The Choreographer: Watches the hand gestures.

In the past, these specialists might have argued or worked in isolation. This new model uses a "Gating Network" (think of it as a smart manager). The manager looks at the situation and decides: "Right now, the words are confusing, so let's listen to the Musician more. But wait, the Musician is quiet, so let's ask the Choreographer what the hands are saying."

The Big Discovery

The results were exciting:

Gestures help: Adding hand movements made the computer much better at predicting turn-taking, especially in tricky moments where the words or voice weren't clear.
Meaning matters: The computer performed even better when it understood what the gesture meant (e.g., knowing a "pointing" gesture is different from a "storytelling" gesture) rather than just seeing the raw movement.
Different gestures do different jobs:
- Discourse gestures (organizing the story) often signaled, "I'm wrapping up, you can speak now" (Yield).
- Metaphoric gestures (showing abstract ideas) often signaled, "I'm still explaining, keep listening" (Hold).

Why This Matters

This isn't just about making better chatbots. It's about making machines that understand humans the way humans understand each other. By teaching computers to "read" the hands, we can create:

Better Virtual Assistants: That know when to interrupt or when to wait.
More Natural Avatars: That move their hands in ways that match their speech perfectly.
Smarter Meeting Tools: That can tell if a meeting is flowing smoothly or if someone is being talked over.

In short: Humans are multimodal creatures; we speak with our mouths, but we also speak with our hands. This paper teaches computers to finally listen to the whole conversation, not just the words.

1. Problem Statement

In multi-party conversations, effective turn-taking (managing who speaks next, when to continue, and when to yield) relies on multimodal cues. While existing computational models primarily utilize textual (lexical content) and acoustic (prosody) features, they often overlook non-verbal behaviors, specifically semantic gestures.

The Gap: Current turn-taking models treat gestures as raw motion or ignore them entirely. However, semantic gestures (iconic, metaphoric, deictic, discourse) are shaped by conversational meaning rather than just prosodic rhythm, offering complementary signals that could improve prediction accuracy, especially in ambiguous transitions.
The Challenge: There is a lack of large-scale, densely annotated datasets for multi-party interactions that link specific gesture types to turn-taking dynamics. Most existing datasets are either scripted, monadic (single speaker), or lack semantic gesture labels.

2. Methodology

A. Dataset Construction: DnD Gesture++

The authors extended the existing DnD Gesture corpus (a multi-party tabletop game dataset with synchronized 3D motion, audio, and transcripts) to create DnD Gesture++.

Annotation: They manually annotated 6 hours of conversation (2,663 gesture instances) with semantic gesture types based on McNeill's taxonomy:
- Iconic: Depicting concrete objects.
- Metaphoric: Representing abstract concepts.
- Deictic: Referential gestures (e.g., pointing).
- Discourse: Structuring dialogue (e.g., signaling topic shifts).
- Note: Beat gestures were excluded as they overlap with prosodic cues.
Turn-Taking Labeling: Continuous conversations were segmented into Inter-Pausal Units (IPUs) ending at Transition Relevance Places (TRPs). Turns were labeled as "Hold" (speaker continues) or "Yield" (speaker changes), resulting in 12,000 turns (7k Hold, 5k Yield).
Density: The dataset offers ~~444 labels/hour, significantly denser than comparable datasets like BEAT (~~288 labels/hour).

B. Modeling Framework: Mixture-of-Experts (MoE)

The authors propose a multimodal turn-taking prediction model using a Mixture-of-Experts (MoE) architecture with a gating network.

Input Modalities:
1. Text: Sentence embeddings from Mistral (mxbai-embed-large-v1).
2. Audio: Features from Wav2Vec2.
3. Gesture: 3D upper-body motion sequences.
Gesture Encoding Strategy:
- Base Encoder: A VQ-VAE (Vector Quantized Variational Autoencoder) compresses motion into discrete tokens.
- Semantic Enrichment: Unlike standard approaches, the gesture encoder is trained with semantic supervision. The VQ-VAE tokens are processed by a Transformer encoder, and the semantic annotations guide the learning of gesture representations to align motion cues with linguistic meaning.
Fusion Mechanism:
- Three dedicated "Experts" process Text, Audio, and Gesture inputs.
- A Gating Network computes dynamic weights ( $w_m$ ) for each modality based on the context.
- The final representation is a weighted sum: $h_{fused} = \sum w_m \cdot f_m(x_m)$ .
- A binary classifier predicts "Hold" or "Yield."

3. Key Contributions

DnD Gesture++ Corpus: The creation of the first densely annotated, multi-party English dataset specifically designed for studying gesture-informed turn-taking, containing 2,663 semantic gesture annotations.
Semantic Gesture Modeling: A novel approach to encoding gestures where semantic type annotations are used to supervise the gesture representation learning, ensuring motion cues are aligned with conversational meaning.
MoE Framework for Turn-Taking: Demonstrating that a Mixture-of-Experts framework effectively fuses text, audio, and semantically enriched gestures, outperforming standard concatenation or fixed-weight fusion methods.
Empirical Validation: Comprehensive analysis showing that semantic gestures provide distinct, complementary cues for different turn-taking actions (e.g., discourse gestures for yielding, metaphoric for holding).

4. Experimental Results

Performance Metrics

The model was evaluated on Accuracy, Macro-F1, and per-class F1 scores.

Best Model: Text + Audio + Gesture (with semantic supervision) achieved the highest performance:
- Accuracy: 71.5%
- Macro-F1: 69.9%
- Hold F1: 76.7%
- Yield F1: 63.1%
Comparison to Baselines:
- Text-only: 68.8% Acc / 66.6% F1.
- Audio-only: 67.1% Acc / 63.2% F1.
- Gesture-only: 66.2% Acc / 61.2% F1 (performing poorly alone, confirming gestures are complementary).
- Text+Audio: 69.7% Acc / 67.9% F1.
- Text+Audio+Gesture (Non-Semantic): 70.4% Acc / 68.7% F1.
- Text+Audio+Gesture (Semantic): 71.5% Acc / 69.9% F1.
- Significance: The inclusion of semantically supervised gestures provided a statistically significant improvement ( $p < 0.05$ ) over non-semantic gesture models.

Ablation & Analysis

Fusion Strategy: The MoE approach outperformed other fusion techniques (ConcatFusion, LMF) by producing more balanced predictions across the "Hold" and "Yield" classes, reducing the bias toward the majority class.
Gesture Type Specificity:
- Discourse Gestures: Strongly correlated with Yield transitions (F1: 74.1).
- Metaphoric Gestures: Strongly correlated with Hold behavior (F1: 89.5).
- Deictic & Iconic: Showed mixed but distinct contributions.
Latent Space: Visualization showed that semantically aligned embeddings formed more distinct clusters corresponding to gesture types, facilitating better cross-modal alignment with text and audio.

5. Significance and Future Work

Theoretical Impact: The study confirms that semantic gestures are not merely decorative but are functional, meaning-bearing signals that regulate conversation flow. They provide critical information when acoustic or lexical cues are ambiguous.
Practical Application: The DnD Gesture++ dataset and the MoE framework can be applied to:
- Improving conversational AI and virtual agents for more natural turn-taking.
- Co-speech gesture generation (synthesizing gestures that match semantic intent).
- Listener backchannel prediction.
Limitations: The dataset is domain-specific (Dungeons & Dragons), which may limit generalizability to formal meetings or task-oriented dialogues. Additionally, inter-annotator agreement for "discourse" gestures was moderate due to the subjective nature of their interpretation.

Conclusion: By integrating semantically informed gestures into a multimodal MoE framework, the authors successfully demonstrated that understanding the meaning of a gesture significantly enhances the ability of AI systems to predict human turn-taking dynamics.