Technical Summary: TIDES – A Longitudinal Bilingual Dataset for Modeling Multi-Party Social Dynamics

Problem Statement

Standard large language models (LLMs) have demonstrated significant progress in dyadic (two-party) interactions but continue to struggle with the complexities of multi-party group conversations. Successful group collaboration requires understanding latent social dynamics, managing turn-taking among multiple speakers, and adapting to continuously evolving interpersonal dependencies. Existing multi-party datasets are often insufficient for capturing these dynamics because they are limited to:

• Short-term lab settings: Failing to capture the long-term evolution of team relationships.
• Contrived or scripted tasks: Missing the real stakes, shared history, and emergent roles found in authentic teamwork.
• Predefined roles: Overlooking the emergent functional roles that arise naturally from ongoing interaction and changing task demands.

Consequently, there is a lack of high-quality, longitudinal resources that capture authentic social dynamics as they naturally unfold, hindering the development of socially aware agents for group conversation.

Methodology

Data Collection (TIDES Dataset)

The authors introduce TIDES (Team Interaction and Dynamic Emergent Social-roles), a longitudinal bilingual dataset tracking 12 university project teams over a full semester (Fall 2025).

• Participants: 50 students in 12 teams (3–5 members each) across diverse disciplines (Design, CS, Industrial Engineering).
• Context: Teams formed organically, collaborated on self-managed projects with real academic stakes (grading), and met repeatedly over 6–12 weeks.
• Scale: 88 dated meetings, 104 transcript files, and 75,971 utterances (approx. 110 hours). The dataset includes both English and Korean (7 teams Korean, 5 English).
• Privacy: All personally identifiable information (PII) was removed using Microsoft Presidio (English) and a local Qwen3-30B-A3B model (Korean), with speakers unified across sessions using WeSpeaker embeddings and the Hungarian algorithm.

Annotation Layers

To capture social dynamics beyond raw transcripts, TIDES includes three layers of socio-structural annotations:

1. Utterance-level Interaction Types: Based on a modified act4teams-SHORT taxonomy (15 categories), including "Giving Information," "Active Listening," and "Social Negative." Labels were generated via human-labeled gold data (5,705 utterances) and fine-tuned Gemma-3-12B for the remaining 70,000+ utterances.
2. Emergent Roles: Derived from post-meeting peer surveys using the TRIAD model (Dominance, Sociability, Task Orientation). Participants rated teammates on 9 Likert items, which were normalized and mapped to 13 emergent roles (e.g., Problem Solver, Coordinator, Critic) based on Euclidean distance to prototypes.
3. Team Development Stages: Meetings were retrospectively classified into Tuckman's stages (Forming, Storming, Norming, Performing, Adjourning) via team consensus at the end of the semester.

Experimental Setup

The authors evaluated TIDES through two primary experiments using Gemma-3-12B-IT (fine-tuned with LoRA) and Llama-3.1-8B-Instruct, benchmarked against proprietary zero-shot models (GPT-5.4, Opus 4.6, etc.).

1. Prediction Tasks:

   • Next-Speaker Prediction: Evaluated under three context conditions: Speaker + Utterance (SU), Speaker + Role + Stage (SRU), and Structure Only (S+R, text removed).
   • Next-Intention Prediction: Predicting the interaction type of the next turn.
   • Data Efficiency: Analyzed performance gains as models were trained on incrementally more meetings from single teams.
   • External Validation: Tested on the AMI Meeting Corpus to assess cross-corpus generalization.

2. Generation Tasks:

   • Compared FT-Plain (fine-tuned direct generation), FT-Reason (fine-tuned with explicit role/intention reasoning), and Vanilla (no fine-tuning) models.
   • Evaluated via automatic metrics (speaker accuracy, semantic similarity) and human evaluation on Prolific (naturalness, coherence, speaker consistency).

Key Results

Prediction Performance

• Next-Speaker Prediction: Fine-tuning on TIDES yielded a 64.53% accuracy (Gemma FT-SU), a 13.8 percentage point improvement over the bigram baseline (50.75%) and outperforming strong proprietary zero-shot models (e.g., Opus 4.6 at 63.25%).
• Structural Sufficiency: Remarkably, using structure only (S+R, removing utterance text) achieved 65.74% accuracy, suggesting that structural features (speaker order, roles, stages) are sufficient for predicting turn-taking patterns, while text may introduce non-transferable lexical patterns.
• Intention Prediction: This task proved harder, with the best model (FT-SU) achieving 32.96% accuracy, only marginally beating the majority baseline (30.05%). Text was necessary for this task, as structure-only models dropped to 31.17%.
• Data Efficiency: Performance gains plateaued rapidly; most improvements occurred within the first 3–4 meetings of a specific team, indicating that models can quickly adapt to team-specific dynamics.
• External Validation: On the AMI Meeting Corpus, a balanced TIDES+AMI mix (combining 62k TIDES samples with 62k AMI samples for a total of 124k) achieved 45.79% accuracy. This result is within 2.06 percentage points of the state-of-the-art (Hilgert & Niehues, 2025) while using ~42% less training data than the in-domain SOTA baseline.

Generation and Human Evaluation

• Structural Coherence: Fine-tuned models significantly improved structural coherence, generating far fewer invalid speakers (152–178 at 10 turns) compared to Vanilla models (786 invalid speakers).
• Naturalness Mismatch: Despite superior structural prediction, human evaluators significantly preferred Vanilla models over fine-tuned models in terms of naturalness, coherence, and speaker consistency.
  • Vanilla won 21.1% of judgments vs. FT-Plain's 9.3% (p < .001).
  • This preference gap persisted even after post-processing fine-tuned outputs to correct capitalization and punctuation.
• Implication: The results suggest a potential mismatch: optimizing for conversational structure (predicting who speaks) does not necessarily yield content that humans perceive as natural or coherent.

Significance and Claims

The paper claims that TIDES provides a foundational resource for modeling longitudinal multi-party social dynamics, addressing a critical gap in existing datasets that rely on short-term or scripted interactions.

• Modeling Evolution: The dataset enables the study of how social roles, interaction patterns, and team development evolve over time, moving beyond static snapshots of group interaction.
• Structural vs. Content: A key contribution is the empirical finding that while structural understanding (roles, stages) significantly improves turn-taking prediction, it does not automatically translate to more natural utterance generation. This challenges the assumption that richer reasoning about group structure inherently leads to more natural contributions.
• Data Efficiency: The work demonstrates that a relatively small amount of well-structured, longitudinal meeting data is sufficient to capture team dynamics and transfer effectively to other corpora (AMI). Specifically, the general model was trained on 32,243 samples drawn from nine teams (Teams 1, 2, 3, 5, 8, 9, 10, 11, and 12), excluding the validation team (Team 4) and test teams (Teams 6 & 7). The cross-corpus transfer experiment achieved strong results using a balanced mix totaling 124k samples, outperforming larger-scale zero-shot approaches in specific structural tasks.

The authors conclude that while TIDES successfully supports the prediction of conversational structure, the disconnect between structural accuracy and human preference for naturalness highlights an open direction for future research: understanding how LLM agents can generate natural utterances that align with the structural realities of group dynamics.