💬 NLP

Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

This paper introduces a Multimodal Voice Activity Projection (MM-VAP) framework that leverages pretrained audio-visual encoders and Low-Rank Adaptation to predict future turn-taking dynamics in social robots, demonstrating improved performance over baselines on multiple human-robot interaction datasets.

Original authors: Antonio Cano, Guillermo Pérez, Luis Merino, Randy Gomez

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Antonio Cano, Guillermo Pérez, Luis Merino, Randy Gomez

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a social robot named Haru acting as a quiet, helpful host at a dinner party. Its job isn't to be the main guest talking, but to stand in the background, listening to two humans chat, and knowing exactly when to step in to keep the conversation flowing smoothly.

The problem is that current robots are terrible at this. They usually just wait for a long silence (like a pause in the music) before they think, "Okay, my turn!" But in real human conversation, people don't wait for silence. They predict when the other person is about to stop talking and jump in almost instantly. If a robot waits for the silence, it feels awkward and slow.

This paper presents a new "brain" for the robot called MM-VAP (Multimodal Voice Activity Projection). Here is how it works, using simple analogies:

1. The "Crystal Ball" vs. The "Stopwatch"

Old robots use a stopwatch. They count the seconds of silence. If the silence is too long, they assume the person is done.
The new MM-VAP system uses a crystal ball. Instead of waiting for the silence, it looks at what is happening right now and predicts what will happen in the next few seconds. It asks, "Is that person about to finish their sentence? Is the other person about to jump in?" It projects the future of the conversation frame-by-frame.

2. Seeing and Hearing (Multimodal)

Previously, these robots only had ears. They listened to the audio. But humans also use their eyes (nodding, looking at someone, facial expressions) to signal they are done talking or want to speak.
This new system gives the robot both eyes and ears. It looks at the video of the speakers' faces and listens to their voices simultaneously. It's like having a conversation where you can hear the words and see the body language, making the prediction much more accurate.

3. The "Expert Intern" Strategy (LoRA)

The researchers didn't build a brain from scratch. That would be like teaching a robot to speak and understand faces from day one. Instead, they took two "expert interns" that were already trained on massive amounts of speech and video data:

TalkNet: An expert at spotting who is speaking.
WhisperFlamingo: An expert at understanding speech and faces together.

These experts are very smart but specialized in their original jobs. To teach them how to predict turn-taking, the researchers used a technique called LoRA (Low-Rank Adaptation).

The Analogy: Imagine you have a master chef (the pre-trained model). You don't want to retrain them on everything from scratch. Instead, you give them a small, specialized recipe card (the LoRA adapter) that teaches them exactly how to make this specific dish (predicting turn-taking). You keep the chef's original skills frozen and only train the small recipe card. This is fast, efficient, and keeps the chef's original knowledge intact.

4. The "256-Step" Dance Floor

The system doesn't just guess "Yes/No." It breaks the next 2 seconds of conversation into tiny slices. It calculates the probability of 256 different scenarios happening at once (e.g., "Speaker A keeps talking," "Speaker B jumps in," "Both talk," "Both stop").
It then looks at this complex map of possibilities to decide: "Ah, the pattern suggests Speaker B is about to take the floor."

5. The Results

The team tested this on real conversations in many languages (English, French, German, Japanese, Chinese) and specifically in a setting where the robot acts as a mediator (like Haru).

The Outcome: The new system was better at predicting when a conversation would switch speakers than previous methods. It was especially good at spotting "Backchannels" (like saying "uh-huh" or nodding) and predicting when someone is about to shift the floor, even before they actually speak.
The Verdict: By using these "expert" audio-visual brains and the efficient "recipe card" training method, the robot can now anticipate human conversation dynamics much better, making it a more natural and less awkward mediator.

In short: The paper shows that by giving a robot the ability to "see and hear" like a human, and by using a smart, efficient way to teach it how to predict the future of a conversation, the robot can finally understand the rhythm of human talk without having to wait for an awkward silence.

Technical Summary: Multimodal Voice Activity Projection for Turn-Taking in Social Robots

Problem Statement
Effective Human-Robot Interaction (HRI), particularly in social mediation roles where a robot facilitates human-human dialogue, requires accurate turn-taking prediction. Current systems often rely on reactive, heuristic rules based on silence duration (e.g., detecting an End-of-Turn after ~700ms). These approaches fail to capture the predictive nature of human conversation, where speakers anticipate Transition-Relevant Places (TRPs) and coordinate responses with minimal delay (Floor Transfer Offset). Furthermore, existing Spoken Dialogue Systems (SDS) frequently lack multimodal perception, leading to errors such as responding to background speech or misidentifying the active speaker. While Voice Activity Projection (VAP) has emerged as a robust framework for predicting future voice activity (VA) in dyadic interactions, most implementations remain audio-only or rely on generic multimodal encoders that lack task-specific alignment with speech dynamics.

Methodology
The authors propose a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs. The core objective is to project the future VA of two speakers over a temporal horizon ( $\Delta t$ ) by modeling their relational dynamics.

Architecture: The model utilizes a transformer-based architecture with five stages: preprocessing, backbone encoding, inter-speaker attention, prediction heads, and post-processing.
- Backbones: Instead of generic encoders, the system employs pretrained audio-visual backbones specifically optimized for speech-related tasks: TalkNet (for Active Speaker Detection) and WhisperFlamingo (for Audio-Visual Speech Recognition). These backbones provide latent representations that are inherently aligned with speech activity and preserve audio-visual temporal coordination.
- Adaptation (LoRA): To efficiently adapt these large pretrained models to the specific turn-taking task without full fine-tuning, the authors employ Low-Rank Adaptation (LoRA). This strategy freezes the original weights ( $W^0$ ) and introduces trainable low-rank matrices ( $A$ and $B$ ) to approximate the weight updates ( $\Delta W$ ), allowing the model to specialize while maintaining computational efficiency.
- Inter-Speaker Attention: After independent encoding of each speaker's audio-visual stream, a cross-attention mechanism models the interaction between speakers, conditioning the prediction of one speaker's future state on the other's current multimodal state.
Output and Inference: The model projects a 2-second future window divided into four temporal bins per speaker, resulting in a joint 256-state space ( $2^{2 \times 4}$ ). Turn-taking events (Shift, Hold, Backchannel, etc.) are inferred in a zero-shot manner by aggregating probabilities of specific microstates, eliminating the need for manual turn-taking annotations during training.
Loss Function: To address the imbalance of rare dialogue states and ensure semantic consistency, a semantic consistency loss ( $L_{sem}$ ) is added to the standard cross-entropy losses. This term encourages the model to assign probability mass to all microstates that share the same semantic dialogue state (e.g., "Speaker 1 active"), improving training stability and alignment with event inference.

Key Contributions

MM-VAP Architecture: An audio-visual framework that integrates VA-related pretrained backbones, preserving their native representational and fusion capabilities for improved voice activity projection.
Parameter-Efficient Adaptation: The application of LoRA to specialize large, speech-activity-focused multimodal encoders for the turn-taking task, avoiding the computational cost of full fine-tuning.
Mediation-Oriented Evaluation: The first application of MM-VAP in a robotics context, specifically evaluating its efficacy for predictive turn-taking in the Haru social mediator setting.
Semantic Regularization: The introduction of a semantic consistency loss to regularize the 256-state output space according to higher-level dialogue patterns.

Results
Experiments were conducted on the NoXi and NoXi+J corpora (multilingual dyadic interactions) and the Haru EDR corpus (mediator-oriented interactions).

Performance: The proposed MM-VAP models (using TalkNet and WhisperFlamingo with LoRA) consistently outperformed non-pretrained baselines (CPC + 3DResNet) and previous state-of-the-art methods across multiple languages (English, German, French, Japanese, Chinese).
Specific Metrics: The models showed significant improvements in Shift prediction (S-pred) and Shift/Hold (S/H) events. For instance, on the NoXi+J dataset, WhisperFlamingo achieved an S-pred F1-score of 0.97 for English and German, while TalkNet achieved 0.94 for German.
Multilingual Robustness: The approach demonstrated strong generalization across languages, with joint multilingual training yielding competitive results comparable to language-specific implementations.
Robotics Validation: On the Haru EDR dataset, the models achieved high accuracy (e.g., 0.92 for Hold and Shift events with WhisperFlamingo), validating the approach's suitability for managing conversational flow in mediation scenarios.

Significance and Claims
The paper claims that the combination of specialized, speech-activity-related pretrained encoders and LoRA adaptation provides a more effective framework for multimodal VAP than using generic encoders or hand-crafted features. The authors assert that this approach successfully transfers the predictive dynamics required for turn-taking, particularly benefiting mediation-oriented HRI where the robot must anticipate floor changes rather than merely react to silence. While the study is limited to offline evaluation and acknowledges the computational challenges of real-time deployment with large encoders, the results suggest that this direction holds strong potential for improving social robot behavior in human-human communication settings. The work positions MM-VAP as a viable step toward fluid, predictive, and multimodal turn-taking management in social robotics.