💬 NLP

Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialogue

This paper investigates how gestures and speech jointly convey referential information in video-mediated dialogue under varying visibility conditions, demonstrating that gesture alone is predictive, multimodal fusion is most beneficial when speech is ambiguous, and partner visibility influences both gesture production and interactional entrainment.

Original authors: Esam Ghaleb, Hugh Mee Wong, Kristina Kobrock

Published 2026-08-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Esam Ghaleb, Hugh Mee Wong, Kristina Kobrock

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess a secret object your friend is thinking of, but you can only hear their voice. Sometimes they say, "It's the one with the pointy bit," which is pretty vague. But if you can also see them, they might point at the air or wiggle their fingers to show the shape. This is the magic of human conversation: we don't just use words; we use our whole bodies to fill in the gaps. Scientists who study how computers understand language (a field called Natural Language Processing) have been great at reading transcripts of what people say, but they often struggle to understand the messy, unspoken parts of a conversation, like hand gestures. The big question is: Can a computer learn to "read" those hand movements to figure out what someone is talking about, especially when the words alone aren't clear enough? And does it matter if the computer can "see" the person making the gestures, or just hear them?

This paper dives into that exact mystery by setting up a digital game. The researchers created a scenario where two people had to identify one of 16 strange, alien-like shapes (called "Fribbles") that have no real names. They had to describe and pick the right one out of a lineup. The team tested this in three different "visibility" settings: where the partners could only hear each other, where they could see each other from one side, and where they could see each other fully. They built computer models to act as the guesser, feeding them either just the spoken words, just the skeleton-like movement of the hands, or a mix of both.

Here is what they found: The hand gestures alone were actually surprisingly good at guessing the right object. The computer models could pick the correct Fribble just by watching the hand movements about 20% of the time, which is way better than random guessing (which would only get it right 5.9% of the time). However, the words were still the star of the show, getting it right about 46.5% of the time. The real magic happened when the computer combined both. The hand gestures didn't just add a little bit of help; they acted like a safety net. When the computer was confused by the words and didn't know what to guess, the hand movements provided the extra clue needed to solve the puzzle.

The researchers also discovered that the "visibility" of the partners changed how the humans acted. When the partners could see each other, they used more gestures, and those gestures were packed with more useful information. But when they couldn't see each other, the gestures became a bit more like self-soothing movements and less helpful for the computer to guess the object. Interestingly, as the partners played the game over and over again, they got better at describing the objects with words, but their hand gestures didn't get any better or more specific over time.

Finally, the team tried a fancy trick to help the computer understand gestures better. They taught the model to mentally "match" the shape of a hand movement to the picture of the object it was describing. This didn't make the hand gestures better on their own, but it did make the combination of words and gestures even sharper, pushing the success rate up a few more percentage points. In short, the paper suggests that while computers are getting good at listening, they still need to learn how to watch us move to truly understand what we mean, especially when our words get a little fuzzy.

Technical Summary: Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialogue

Problem Statement

Human communication is inherently multimodal and embodied, where gestures often carry information absent or underspecified in speech. However, contemporary dialogue models typically rely solely on text transcripts, failing to leverage the full spectrum of communicative cues. While previous work has established that gestures can aid reference resolution, it remains unclear how much referential information gestures carry independently, how they complement speech, and how these dynamics shift under different partner visibility conditions (e.g., audio-only vs. video-mediated). Furthermore, existing benchmarks often lack the granularity to disentangle when gesture provides unique value versus when it merely reinforces speech. This study addresses these gaps by quantifying the informativity of iconic gestures and their fusion with speech in a controlled referential communication task.

Methodology

Dataset and Task

The study utilizes the video-mediated extension of the CABB dataset (Akamine et al., 2025a,b), featuring 90 Dutch-speaking participants organized into 45 dyads. The task is a referential communication game where a "Director" describes one of 16 novel visual objects (Fribbles) to a "Matcher," who must identify the target.

Conditions: Dyads interacted under three visibility conditions: Audio-Only (AO), Asymmetric Video (ASYM), and Symmetric Video (SYM).
Data: The dataset includes high-quality audio, time-aligned orthographic transcripts (Dutch/English), multi-angle video, and motion-tracking data.
Task Formulation: The problem is framed as a referent classification over 16 candidates (plus an "OTHER" class). Models predict the target object given a speech transcript, a skeletal gesture sequence, or both.

Modeling Approach

The authors developed a pipeline comparing unimodal and multimodal architectures:

Gesture Representations: Two skeletal layouts were compared:
- MediaPipe-3D: 51 keypoints (3D coordinates).
- MMPose-2D: 69 keypoints (2D coordinates including face).
- Encoder: Both layouts were processed using Spatio-Temporal Graph Convolutional Networks (ST-GCN) to capture anatomical and temporal dependencies.
Speech Representations: Transcripts were encoded using a frozen CLIP ViT-L/14 text projection.
Referent Representations: Images of the 16 Fribbles were encoded using the frozen CLIP image projection to serve as class prototypes.
Fusion Mechanism: A gated residual fusion module integrates gesture and speech representations. The fusion is speech-anchored, allowing the model to selectively gate gesture information based on the speech signal.
Training-Only Alignment: A novel component involves Pose-to-Image (P→V) alignment. During training, a contrastive loss (InfoNCE) aligns the learned gesture representations with the frozen CLIP image representations of the target objects. This alignment is discarded at inference time.
Baselines: The study compares against a general-purpose Vision-Language Model (Qwen3-VL) and unimodal baselines.

Evaluation Protocol

Metrics: Top-k accuracy ( $k \in \{1, 2, 3, 4, 5\}$ ), Macro-F1, and a utility metric $U_{1:5}$ measuring ranking quality.
Validation: Five-fold cross-validation, grouped by trial to prevent data leakage. Face-to-face data was used only for auxiliary training.
Statistical Analysis: Bayesian binary logistic regression was employed to analyze the effects of visibility and interaction rounds on model performance and human gesture production.

Key Results

1. Gesture as a Genuine Referential Channel

Gesture-only models achieved significantly above-chance performance:

Top-1 Accuracy: ~19.5–20.4% (Chance: 5.9%).
Top-5 Accuracy: ~55.5–56.4%, placing the correct referent in a shortlist for over half the instances.
This confirms that iconic gestures alone contain recoverable referential information, independent of speech.

2. Selective Complementarity of Gesture

While speech-only models performed strongly (46.5% Top-1), adding gesture provided consistent, albeit modest, improvements (~1.6–1.7 percentage points).

Uncertainty-Driven Gain: The benefit of fusion was selective. In the least confident quartile of speech predictions (where speech accuracy was ~20%), adding gesture improved accuracy by +7.5 percentage points. In high-confidence speech segments, the gain was negligible.
This indicates that gesture primarily aids reference resolution when the verbal signal is ambiguous or underspecified.

3. Impact of Pose-to-Image Alignment

The training-only P→V alignment further improved multimodal performance:

Top-1 accuracy increased by 1.7 percentage points (from 48.1% to 49.8%) for the best-performing fusion model.
Crucially, this alignment did not improve gesture-only performance, suggesting its primary role is facilitating the integration of gesture into the speech representation rather than enhancing gesture discrimination in isolation.

4. Effects of Partner Visibility and Entrainment

Visibility: Gesture production and informativity were significantly higher when interlocutors could see each other (Symmetric/Asymmetric video) compared to audio-only. However, speech and fusion model performance remained stable across visibility conditions.
Entrainment: Across six rounds of interaction, speech accuracy improved significantly (from 40.3% to 60.4%), reflecting lexical entrainment (conventionalization of referring expressions). In contrast, gesture-only accuracy remained flat, and no independent gesture-entrainment effect was observed.

Significance and Contributions

The paper makes contributions to both technical modeling and the analysis of human interaction:

Technical Modeling: It demonstrates that gesture is a recoverable and complementary information channel in multimodal dialogue. The study establishes that fusion gains are concentrated in regions of speech uncertainty, providing a more nuanced view of multimodal integration than aggregate performance metrics. The introduction of training-only semantic alignment (P→V) offers a method to improve fusion without altering inference-time inputs.
Analysis of Human Interaction: By connecting model performance to human data, the study quantifies pragmatic effects. It confirms that partner visibility modulates gesture production and informativity, while repeated interaction drives lexical entrainment in speech but not necessarily in gesture.
Methodological Rigor: The use of controlled skeletal representations (rather than raw video) allows for a direct assessment of gesture informativity, disentangling it from the complexities of general-purpose vision-language models which, as shown in the Qwen3-VL baseline, may fail to effectively utilize gesture frames without specific architectural constraints.

The authors conclude that while gestures are informative, their utility in computational models is most pronounced when they resolve ambiguity in the verbal signal, and their production is heavily influenced by the visual context of the interaction.