🤖 AI

The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge

The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge introduces a comprehensive benchmark and dataset of 945 naturalistic, multilingual dyadic interactions to advance the modeling of interpersonal affect and social dynamics through three sub-challenges focused on directional influence, turn-taking, and rapport trajectory prediction.

Original authors: Panagiotis Tzirakis, Alice Baird, Jeffrey Brooks, Emilia Parada-Cabaleiro, Lukas Stappen, Sharath Rao, Theo Lebryk, Jakub Piotr Clapa, Jens Madsen

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Panagiotis Tzirakis, Alice Baird, Jeffrey Brooks, Emilia Parada-Cabaleiro, Lukas Stappen, Sharath Rao, Theo Lebryk, Jakub Piotr Clapa, Jens Madsen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine two people sitting across from each other, having a conversation. They aren't just taking turns speaking; they are dancing. One person's joke makes the other laugh; one person's hesitation makes the other pause; they build a sense of connection (or "rapport") as the minutes tick by.

For a long time, computers have been good at analyzing one person in a room—like a solo singer hitting a high note. But they have struggled to understand the duet: how Person A's mood changes Person B's, how they coordinate their timing, and how their relationship evolves.

This paper introduces a new "gym" for computer scientists called the ACII-DaiKon Challenge. It's a competition designed to teach computers how to understand these two-person dances.

Here is a breakdown of what they are doing, using simple analogies:

1. The Dataset: A Giant Library of Real Conversations

The organizers built a massive library called the Hume-DaiKon dataset.

The Collection: It contains 945 conversations (over 743 hours of video and audio) between pairs of people.
The Variety: These aren't scripted plays. They are real, natural chats between strangers who were paired up online. They speak five different languages (English, German, Spanish, Dutch, and Polish).
The Setup: Imagine a video call where two people are asked to chat about their weekends or vacations. The system records both of them separately (so the computer can see Person A's face while listening to Person B's voice) and captures everything from their tone of voice to their facial expressions.

2. The Three Challenges (The "Events")

The competition asks participants to build AI models that can solve three specific puzzles based on these conversations:

Puzzle A: The "Emotional Echo" (Influence)

The Goal: Predict how Person B is feeling right now, based on what Person A just said or did.
The Metaphor: Imagine Person A is a weather vane. If Person A looks angry, does Person B's mood shift to match? The AI has to guess if Person B is feeling "joy," "anxiety," or "boredom" based on the other person's behavior.
The Catch: It's not just guessing what Person B feels in a vacuum; it's guessing how Person A influenced that feeling.

Puzzle B: The "Floor Handoff" (Turn-Taking)

The Goal: Predict two things: Who will speak next, and when they will start speaking.
The Metaphor: Think of a hot potato game. The AI has to watch the players and guess: "Is the current speaker about to drop the potato (stop talking), or are they going to hold on tight? And if they drop it, how many seconds until the other person catches it?"
Why it's hard: Sometimes people talk over each other (overlap), sometimes they interrupt, and sometimes there is a long silence. The AI has to spot the subtle cues (like a deep breath or a look) that signal a change.

Puzzle C: The "Relationship Thermometer" (Rapport)

The Goal: Track how the "connection" between the two people changes from the start of the chat to the end.
The Metaphor: Imagine a thermometer that measures how well two people are getting along. Does the temperature go up (they are bonding) or down (they are awkward or arguing) as the conversation progresses?
The Challenge: The AI has to look at the whole conversation, not just one sentence, to see if the relationship is heating up or cooling down.

3. The Tools: What the Computers Are Using

To solve these puzzles, the researchers gave the participants a set of "eyes" and "ears":

The Ears (Audio): The computer listens to the voice using a tool called Whisper. It turns speech into a list of numbers that capture the tone, speed, and emotion of the voice.
The Eyes (Video): The computer watches the faces using a tool called FaceNet. It turns facial expressions into numbers that capture smiles, frowns, and head movements.
The Result: The computer gets a stream of numbers for both people, side-by-side, trying to find patterns.

4. What They Found So Far (The Baseline Results)

The organizers ran some simple "starter" AI models to see how hard the puzzles are. Here is what they discovered:

The Voice is King: For all three puzzles, the computer did much better when it just listened to the voices than when it just watched the faces.
- Analogy: It's like trying to understand a song by looking at the singer's lips vs. listening to the music. The music (voice) told the story much better than the visual (face) in these specific tests.
Mixing Them Didn't Help (Yet): When they tried to combine the eyes and ears, the simple models didn't get much better. In fact, for the "Rapport" puzzle, adding the video actually made the score slightly worse.
- Why? It's like trying to solve a puzzle while wearing foggy glasses. The simple models got confused by the extra visual data. The paper suggests we need smarter ways to combine the two senses in the future.
The Score: The best simple models got about 40% to 70% accuracy depending on the task. This means computers are starting to get the "gist" of the conversation, but they are still far from understanding the deep, complex dance of human interaction.

Summary

This paper is an invitation to the world's smartest computer scientists to stop looking at people in isolation and start looking at them as pairs.

They have provided the raw materials (the video library), the rules (the three puzzles), and a starting point (the simple models). The goal is to build AI that doesn't just hear words, but understands the rhythm, influence, and connection between two people talking to each other.

Technical Summary: The 2026 ACII Dyadic Conversations (DaiKon) Workshop & Challenge

Problem Statement

Current benchmarks in affective computing and conversational AI predominantly focus on speaker-centric prediction or static summaries, failing to capture the coupled, time-evolving nature of dyadic (two-person) interactions. Existing frameworks often underrepresent critical interpersonal dynamics, including directional influence (how one speaker affects another), conversational timing coordination (turn-taking), and the development of rapport over time. The 2026 ACII DaiKon Workshop & Challenge addresses this gap by establishing a unified benchmark for modeling these relational processes within a naturalistic, multimodal setting.

Methodology and Dataset

The Hume-DaiKon Corpus

The challenge is built upon the Hume-DaiKon dataset, a collection of 945 naturalistic dyadic conversations totaling 743.4 hours of audiovisual data.

Collection: Data was gathered via a dual-channel platform connecting participants from similar geographic regions. Conversations were open-ended, guided by prompts (e.g., "Where did you go on your last vacation?") to elicit expressive behaviors.
Multilingual Composition: The dataset includes sessions in five languages: English, German, Spanish, Dutch, and Polish.
Splits: The data is partitioned into 661 training, 142 validation, and 142 test sessions. The test set is held out and blind for official evaluation.
Modality & Features:
- Audio: Processed using a Whisper-small encoder to generate 768-dimensional frame-level embeddings at ~3 FPS.
- Video: Processed using FaceNet to generate 512-dimensional embeddings from face images at 3 FPS, retaining only frames with detection confidence $\ge$ 0.9.
- Format: Features are released in Parquet format to facilitate efficient loading without decoding raw media.

Label Generation Strategies

The challenge employs three distinct labeling approaches corresponding to its sub-challenges:

Influence Labels: Derived from an internal speech expression model that scores emotional intensity for both speakers per utterance. The target vocabulary includes 10 dimensions (anger, anxiety, uncertainty, confusion, doubt, boredom, surprise, curiosity, joy, amusement) selected to cover the valence–arousal space.
Turn-Taking Labels: Generated via per-speaker Voice Activity Detection (VAD). Events are annotated as switch (clean transfer), overlap, or interruption, capturing the next speaker, onset time, and transition type.
Rapport Pseudo-labels: Due to the high cost of human annotation for continuous rapport ratings, a Large Language Model (Qwen2.5-72B-Instruct) was used to generate pseudo-labels. The model scored rapport on a [0, 1] scale for 30-second windows based on transcripts, acoustic tags, and prior context. These scores were smoothed using a Gaussian kernel to create realistic trajectories.

Competition Tasks

The challenge is divided into three coordinated sub-challenges:

DaiKon Influence: Predicting the continuous intensity of 10 emotion dimensions for a target speaker based on the dyadic context.
- Metrics: Concordance Correlation Coefficient (CCC) and Pearson correlation.
DaiKon Turn-Taking: A dual-task predicting (a) the next speaker (classification) and (b) the time-to-next-speech onset (regression).
- Metrics: Macro-F1/Accuracy for classification; Mean Absolute Error (MAE) for regression.
DaiKon Rapport: Predicting the trajectory of rapport quality over the course of a conversation.
- Metrics: CCC and Pearson correlation.

Baseline Experiments and Results

The organizers released baseline systems using a shared two-layer MLP architecture with task-specific heads, trained on the standardized feature sets.

Key Findings:

Dominance of Audio: Across all three tasks, audio-only models significantly outperformed video-only models.
- Influence: Audio achieved 0.40 CCC vs. 0.19 CCC for video.
- Turn-Taking: Audio achieved 0.66 Macro-F1 vs. 0.51 for video.
- Rapport: Audio achieved 0.68 CCC vs. 0.26 CCC for video.
Multimodal Fusion Limitations: At the baseline level, simple concatenation of pooled audio and video features did not consistently improve performance over audio-only baselines. In the Rapport task, the multimodal baseline (0.59 CCC) actually underperformed the audio-only baseline (0.68 CCC), suggesting that naive fusion may introduce noise without sophisticated temporal modeling.
Performance Benchmarks:
- Influence: Best test results reached 0.40 CCC and 0.50 Pearson.
- Turn-Taking: Best test results reached 0.66 Macro-F1 and 1.50s MAE.
- Rapport: Best test results reached 0.68 CCC and 0.70 Pearson.

Significance and Claims

The paper positions the ACII-DaiKon Workshop & Challenge as a foundational step toward moving affective computing beyond isolated speaker analysis. Its primary contributions are:

A Unified Benchmark: It provides the first shared dataset and evaluation framework specifically designed to study directional influence, turn-taking coordination, and rapport development simultaneously.
Methodological Rigor: By utilizing fixed splits, standardized metrics, and released baselines, the challenge enables reproducible comparison across different architectural approaches.
Identification of Research Gaps: The baseline results explicitly highlight that while current methods capture coarse dyadic patterns, robust modeling of directional dependence and long-horizon interpersonal dynamics remains a significant challenge. The paper asserts that future progress requires architectures capable of representing temporal dependence, inter-speaker coupling, and more sophisticated multimodal fusion strategies.

The workshop aims to foster cross-disciplinary discussion among researchers in affective computing, machine learning, emotion science, and cultural psychology to advance the state of the art in socially aware conversational AI.