💬 NLP

Multimodal Conversation Structure Understanding

This paper introduces the TV-MMPC dataset and a suite of tasks to evaluate multimodal LLMs on conversation structure understanding, revealing that while models outperform heuristics, they struggle with anonymized identities, alongside a sociolinguistic analysis of TVQA showing female characters are disproportionately cast as addressees or side-participants, shifting the conversational register toward social interaction.

Original authors: Kent K. Chang, Mackenzie Hanh Cramer, Anna Ho, Ti Ti Nguyen, Yilin Yuan, David Bamman

Published 2026-01-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kent K. Chang, Mackenzie Hanh Cramer, Anna Ho, Ti Ti Nguyen, Yilin Yuan, David Bamman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of friends having a lively dinner. It's not just about what they are saying; it's about the invisible dance of who is talking to whom, who is listening, and who is just watching. Sometimes, one person speaks to the whole table, sometimes they lean in to whisper to just one friend, and sometimes a third person chimes in on a topic they weren't directly invited to.

This paper is like a new set of glasses designed to help computers understand that complex dinner party dance, specifically when watching TV shows.

Here is a breakdown of what the researchers did, using simple analogies:

1. The Problem: Computers Are Bad at "Who's Talking to Whom"

Current AI models (the "smart" computers) are great at watching a video and saying, "Oh, that's a car," or "That's a dog." They can even read subtitles. But when it comes to a multi-person conversation, they often get lost. They might know who is speaking, but they struggle to figure out:

The Addressee: Is the speaker talking to the person next to them, or shouting across the room?
The Side-Participant: Who is sitting there, listening, and part of the group, even if no one is looking directly at them?
The Thread: Is this new sentence a reply to the sentence spoken 10 seconds ago, or is it a brand-new topic?

Think of it like trying to follow a game of catch in a dark room. You can hear the ball being thrown, but without seeing who is throwing it to whom, you can't understand the game.

2. The Solution: A New Dataset (TV-MMPC)

To teach computers this skill, the researchers created a new "training manual" called TV-MMPC.

The Source: They took clips from popular TV shows (like The Big Bang Theory and Friends).
The Work: Humans watched these clips and meticulously labeled every single line of dialogue. They marked:
- Speaker: Who is talking?
- Addressee: Who is being spoken to?
- Side-Participant: Who is listening but not being spoken to?
- Reply-to: Which previous line is this answering?

It's like having a human director sit down and draw arrows on a script showing exactly who is looking at whom and who is replying to whom.

3. The Experiment: Testing the AI

The researchers then asked several "smart" AI models (like Gemini, LLaMA, and others) to look at these TV clips and guess the roles and threads, just like a human would.

The Results:

AI is getting better: The AI models did much better than a simple computer program that just guessed based on who was talking the most.
The "Anonymity" Trap: Here is the big surprise. When the researchers hid the characters' names (e.g., changing "Sheldon" to "Character A"), the AI's performance crashed.
- The Metaphor: It's like the AI was cheating. Instead of actually understanding the social cues (like eye contact or body language), it was memorizing the names. "Oh, if I see Sheldon, I know he usually talks to Leonard." Once you took the names away, the AI didn't know how to figure out who was talking to whom just by looking at the video.

4. The Sociolinguistic Discovery: What the Data Revealed

Because they had this massive, perfectly labeled dataset, the researchers could also look at the TV shows themselves to see what they say about society. They analyzed over 350,000 lines of dialogue and found some interesting patterns:

The "Listener" Bias: While women in these shows speak about as much as men (proportional to their screen time), they are 1.2 times more likely to be cast as the listener (the addressee or side-participant) rather than the one starting the conversation.
- The Metaphor: Imagine a party where men are mostly the ones throwing the ball (starting the conversation), and women are mostly the ones catching it (listening), even if they are both playing the game equally.
The "Side-Participant" Effect: When there are extra people in the room (side-participants) just listening, the conversation changes. It shifts from being personal (like talking about your day) to social (like making polite small talk or performing for the group).
- The Metaphor: When you are alone with a friend, you might say something rude or intimate. But if a third person walks in and starts listening, you suddenly start talking about the weather or being extra polite. The presence of an audience changes the "vibe" of the chat.

Summary

In short, this paper built a new tool to teach computers how to understand the complex social dance of TV conversations. It found that while AI is getting good at this, it still relies too much on knowing character names rather than truly "seeing" the social dynamics. Finally, by using this tool, the researchers discovered that TV shows often subtly position women as the audience rather than the leaders of the conversation, and that the mere presence of extra listeners changes how people speak.

Technical Summary: Multimodal Conversation Structure Understanding

Problem Statement

While multimodal Large Language Models (LLMs) have demonstrated proficiency in dialogue generation and reasoning over complex video-text pairs, their ability to parse the underlying structure of conversation remains underexplored. Specifically, it is unclear whether these models can adequately resolve conversational roles (speakers, addressees, side-participants) and threading (reply-to relationships) in multi-party settings. Existing work in conversation disentanglement and addressee recognition has largely focused on text-based or controlled audio-only datasets, often neglecting the rich non-verbal cues (gaze, posture, turn-taking) that govern face-to-face interactions as described in sociolinguistics (Goffman, 1981; Goodwin, 1981; Clark and Carlson, 1982).

Methodology

1. Dataset Construction: TV-MMPC

To address the lack of annotated data for multimodal conversation structure, the authors constructed TV-MMPC (Multi-modal Multi-Party Conversation), a human-annotated dataset derived from the TVQA benchmark.

Source: 200 randomly sampled clips (60–90 seconds each) from four TV series: The Big Bang Theory, Friends, House M.D., and How I Met Your Mother.
Scale: The dataset contains 4,378 speaker and reply-to annotations, 5,599 addressee labels, and 3,412 side-participant tags across 350,842 total utterances in the broader TVQA corpus used for sociolinguistic analysis.
Annotation Framework: Grounded in conversation analysis and sociolinguistics, the annotation task requires predicting four elements for each utterance:
- Speaker: The source of the utterance.
- Addressee(s): Participants explicitly addressed, ratified as group members, and known to be attending.
- Side-participants: Participants who are ratified and perceptually engaged but not explicitly addressed.
- Reply-to: The parent utterance the current line responds to (or "self" if initiating a new thread).
Quality Control: Inter-annotator agreement improved significantly from a pilot round to the main round (e.g., Speaker Accuracy rose from 86.20% to 97.58%), achieved through a dedicated annotation interface and rigorous guidelines.

2. Task Formulation

The authors define conversation structure understanding as a structured prediction problem. Given a video clip $V$ and a sequence of utterances $U$ , the goal is to learn a function $G$ that maps an utterance $u_i$ and its context to structural elements $\{u_p, S_i, A_i, E_i\}$ , where $u_p$ is the reply-to parent, $S_i$ is the speaker, $A_i$ are addressees, and $E_i$ are side-participants.

3. Experimental Setup

The study evaluates six models on TV-MMPC:

Vision-Language Models (VLMs): LLaMA 4 Scout, GPT-4.1 mini, o4-mini, and Gemini 2.0 Flash (processed with sampled frames and text).
Audio-Visual LLMs: Reka-Flash, Qwen 2.5-Omni 7B, and Gemini 2.0 Flash (processed with full reconstructed video clips including audio).
Baselines: A heuristic baseline using Whisper for transcription and pyannote for speaker diarization, combined with face frequency heuristics for role assignment.
Fine-tuning: A resource-efficient supervised fine-tuning (SFT) experiment using LoRA on Qwen 2.5-Omni 7B with a leave-one-show-out evaluation strategy.
Anonymization Test: An experiment where character names and identities are replaced with random labels to test model reliance on memorized cultural knowledge versus multimodal reasoning.

Key Results

Model Performance

Superiority of Audio-Visual Models: Among all models tested, Gemini 2.0 Flash (audio-visual) achieved the highest performance, significantly outperforming the heuristic baseline and vision-only models. It achieved 78.60% accuracy in speaker recognition and 68.11% Set F1 for addressees.
Modality Impact: An ablation study on Gemini 2.0 Flash revealed that while text alone performs poorly, adding visual information improves role attribution. However, full audio-visual context is required for high accuracy, particularly in distinguishing speakers and addressees.
Fine-tuning Gains: LoRA SFT on Qwen 2.5-Omni 7B yielded statistically significant improvements over its zero-shot performance, particularly in side-participant recognition (increasing Set F1 from 25.94% to 57.69%).

Anonymization and Reliability

Significant Performance Drop: When character identities were anonymized, the performance of the best model (Gemini 2.0 Flash) dropped drastically: Speaker Accuracy fell by 64.92 points (from 78.60% to 13.68%), and Addressee Set F1 dropped by 52.37 points.
Implication: This suggests that current multimodal LLMs rely heavily on parametric knowledge of specific characters (memorization) rather than purely reasoning about conversational dynamics from visual and acoustic cues.

Sociolinguistic Analysis

Using the annotated data, the authors analyzed 350,842 utterances to uncover cultural patterns:

Gender and Floor Claiming: While female characters initiate conversations at rates proportional to their speaking time, they are 1.2 times more likely than male characters to be cast as listeners (addressees or side-participants) rather than primary speakers.
Register Shift: The presence of side-participants shifts the conversational register from "personal/transactional" (e.g., use of "I've," "job," "idiot") to "social/performative" (e.g., use of "lovely," "happened," "how's").

Significance and Contributions

The paper claims three primary contributions:

Framework and Tasks: It introduces a unified framework for multimodal conversation structure understanding, treating role attribution and thread disentanglement as complementary subtasks grounded in sociolinguistic theory.
TV-MMPC Dataset: It releases a high-quality, human-annotated dataset that enables the evaluation of multimodal models on complex interactional dynamics, moving beyond simple dialogue QA.
Critical Evaluation of LLMs: It provides evidence that while state-of-the-art models can outperform heuristics, their performance is fragile when stripped of character identity, raising questions about model reliability and the nature of "understanding" in multimodal systems.

Furthermore, the work demonstrates the utility of structural conversation analysis for characterizing cultural representation on screen, revealing implicit social hierarchies and gendered patterns in dialogue that are otherwise difficult to quantify at scale. The authors conclude that robust structural understanding is a necessary scaffold for analyzing power dynamics and cultural representation in media.