💻 computer science

PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction

This paper introduces PolySLGen, an online framework that generates contextually appropriate and temporally coherent multimodal speaking and listening reactions for target participants in polyadic group interactions by effectively aggregating motion and social cues from all group members.

Original authors: Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a lively dinner party with four friends. You aren't just talking to one person; you are part of a group. To fit in, you have to do three things at once:

Listen to what everyone is saying.
Watch who is looking at whom and how they are moving their hands.
React instantly—either by jumping in with a joke (speaking) or by nodding and smiling (listening).

Doing this naturally is incredibly hard for robots or AI. Most current AI is like a person at a party who can only talk to one person at a time, or worse, they only know how to speak and forget how to listen. They might stand there awkwardly staring at the wall when they aren't talking.

Enter PolySLGen. Think of it as the "Ultimate Party Guest" AI.

The Problem: The "Two-Person" Trap

Most AI researchers have been training robots to have conversations like a phone call between two people (dyadic). But real life is a group chat (polyadic).

The Old Way: If you asked an old AI to join a group, it would get confused. It might ignore the person sitting next to you because it's only looking at the person it's "talking" to. It also struggles to know when to speak and when to shut up and listen.
The Missing Piece: It doesn't just need to hear words; it needs to see body language. If your friend is looking at you while speaking, they want you to reply. If they are looking at someone else, you should stay quiet.

The Solution: PolySLGen (The "Social Brain")

The researchers built a system called PolySLGen (Polyadic Speaking-Listening Generation). Here is how it works, using a simple analogy:

1. The "Group Hug" Sensor (Pose Fusion)

Imagine you are in a room with four people. Instead of looking at them one by one, PolySLGen has a special "group hug" sensor. It takes the movements of everyone in the room and blends them into a single, compact "vibe" signal.

Why? If the AI tried to process every single person's hand movement separately, it would get overwhelmed (like trying to listen to four radio stations at once). By fusing them, it understands the group dynamic instantly.

2. The "Eye Contact" Detector (Social Cue Encoder)

Have you ever been in a group and felt like someone was ignoring you? That's because they aren't looking at you.
PolySLGen has a special module that tracks head orientation. It asks: "Who is looking at our target robot?"

If everyone is looking at the robot, the AI knows: "Okay, it's my turn to speak!"
If the robot is looking at someone else, the AI knows: "I should be listening and nodding."
This is crucial for knowing when to switch from "Listening Mode" to "Speaking Mode."

3. The "Dual-Mode" Actor (Speaking & Listening)

Old AI models usually have two separate brains: one for talking and one for moving. PolySLGen has one brain that handles both.

Speaking Mode: It generates words, a voice, and gestures that match the words.
Listening Mode: It doesn't just freeze! It generates natural reactions like nodding, tilting the head, or shifting weight, exactly like a human does when they are paying attention.
The Switch: It predicts a "Speaking Score." If the score is high, it talks. If it's low, it listens. This makes the transition smooth, like a real conversation, rather than a robotic "ON/OFF" switch.

How They Tested It

They tested this AI in a Dungeons & Dragons (D&D) game setting.

The Scenario: A group of players and a "Dungeon Master" (the target) are playing a game. They are shouting, laughing, and moving around a table.
The Test: They asked the AI to be the Dungeon Master.
The Result:
- Old AI (SOLAMI): Would often freeze when it wasn't talking, or make weird movements that didn't match the conversation. It looked like a robot trying to act human.
- PolySLGen: It looked natural. It knew when to interrupt, when to wait, and how to gesture while listening. In a "blind test" where humans watched videos, they preferred PolySLGen every time because it felt "real."

Why This Matters

This isn't just about making a cooler robot. It's about making AI that can actually hang out with us.

Education: A robot teacher that can manage a whole classroom, not just one student.
Therapy: A robot therapist that can sit with a group of people and understand the group's emotional flow.
Customer Service: A robot receptionist that can greet a family of four, not just the person at the front.

The Bottom Line

PolySLGen is like teaching an AI the art of "reading the room." It doesn't just listen to words; it watches the whole group, tracks eye contact, and knows exactly when to speak up and when to sit back and nod. It's a giant leap from "robot that talks" to "robot that hangs out."

1. Problem Statement

The paper addresses the challenge of generating natural, multimodal reactions (speech, body motion, and turn-taking behavior) for embodied AI agents in polyadic interactions (groups of three or more people).

Key Limitations of Existing Approaches:

Dyadic Bias: Most existing methods are designed for one-on-one (dyadic) interactions and fail to capture the complex, higher-order dependencies of group dynamics.
Modality Gaps: Many approaches are single-modal (e.g., text-to-motion only) or focus exclusively on speaking behaviors, neglecting listening reactions (non-verbal cues like nodding, head orientation, and silence).
Offline Constraints: Many state-of-the-art (SOTA) models rely on future context (non-causal), making them unsuitable for online, real-time applications where an agent must react to past observations only.
Lack of Turn-Taking: Existing systems often lack a unified mechanism to dynamically switch between speaking and listening states, leading to unnatural interruptions or awkward silences in group settings.

2. Methodology: PolySLGen Framework

PolySLGen is an online, causal framework that generates future reactions for a target participant based on past speech, motion, and social cues from all group members. The architecture (illustrated in Fig. 2 of the paper) consists of the following core components:

A. Input Encoding & Modality Integration

The system takes past observations ( $H$ frames) from $P$ participants, including:

Speech & Text: Audio is segmented, transcribed to text (using Whisper/stable-ts), and converted into speech style features (prosody, emotion) using StyleTTS 2.
Motion: 3D body and hand motions are processed.
Social Cues: Head orientations of non-target participants are analyzed to determine attention levels toward the target.

B. Core Modules

Pose Fusion Module:
- Problem: Feeding raw motion from multiple participants into an LLM consumes excessive context and ignores inter-personal coherence.
- Solution: A hierarchical transformer block that aggregates motions from all participants into a compact joint pose embedding. It uses self-attention for intra-participant dynamics and cross-attention to capture inter-participant dependencies, allowing the model to handle variable group sizes efficiently.
Social Cue Encoder:
- Function: Computes a "social cue score" based on the cosine similarity between a non-target participant's head orientation vector and the relative position vector to the target.
- Purpose: Encodes high-level group attention signals (who is looking at whom) to inform the target's reaction, crucial for realistic turn-taking.
LLM Backbone (Llama3-8B):
- A pre-trained LLM serves as the reasoning engine. It is adapted via LoRA (Low-Rank Adaptation) to understand multimodal inputs (text, style, motion, social cues).
- It generates a unified sequence of future text, speech style, and motion embeddings.
Speaking State Prediction:
- Instead of hard-coded transitions, the model predicts a speaking state score ( $r$ ). This scalar value indicates the confidence that the target should speak.
- This acts as a soft cue to guide turn-taking, allowing the system to dynamically alternate between generating speech/motion (speaking) and non-verbal listening cues.

C. Output Generation

The model produces:

Text: Natural language response.
Speech Style: Prosodic features for synthesis.
Motion: 3D body and hand trajectories.
State Score: A binary indicator (soft) for speaking vs. listening.

3. Key Contributions

First Online Polyadic Framework: The first work to propose an online multimodal reaction generation system that handles both speaking and listening behaviors in group (polyadic) settings.
Unified Turn-Taking Mechanism: Introduces a speaking state score prediction that allows the agent to dynamically switch between speaking and listening without enforcing rigid transitions, mimicking human conversational flow.
Novel Architecture for Group Dynamics:
- Pose Fusion: Aggregates multi-person motion into a compact representation, solving the context-length bottleneck of LLMs.
- Social Cue Encoder: Explicitly models visual attention (head orientation) to capture group-level social signals.
Comprehensive Evaluation: Demonstrates robustness to missing participants (occlusion/sensor failure) and outperforms baselines in motion quality, speech alignment, and social semantic coherence.

4. Experimental Results

The method was evaluated on the DnD Group Gesture Dataset (5 participants in a tabletop role-playing game).

Quantitative Performance (vs. SOTA Baselines like SOLAMI, LM-L2L, ConvoFusion):

Motion Quality: Achieved the lowest MPJPE (144.9 mm vs. ~180 mm for baselines) and best FID (12.18), indicating higher spatial accuracy and similarity to ground truth.
Speech & Alignment: Achieved the best BERTScore (0.508) and lowest BeatAlign Diff (0.007), showing superior synchronization between generated speech and body motion.
Speaking State: Significantly outperformed all baselines in Area under Precision-Recall (AP) for speaking state prediction (0.67 vs. ~0.50 for random/baselines).
Social Semantics: Lowest Mean Angular Error (MAE) for head orientation and lowest social cue score errors, proving the agent reacts appropriately to who is speaking or looking at it.

Qualitative & User Study:

A user study with 23 participants rated PolySLGen significantly higher than SOLAMI in Motion Continuity (3.8 vs. 2.4) and Overall Naturalness (3.4 vs. 2.5).
Visual comparisons showed PolySLGen effectively captures smooth transitions between speaking and listening, whereas baselines often produced disjointed or contextually irrelevant motions.

Robustness:

The model maintained performance even when 1–3 non-target participants were missing from the input (simulating sensor occlusion), outperforming baselines that rely on full group observation.

5. Significance and Impact

Real-World Applicability: By operating in an online, causal manner, PolySLGen is suitable for real-time deployment in robotics and virtual agents for education, therapy, and collaborative work.
Bridging the Gap: It moves beyond simple "text-to-motion" or "dyadic" models, addressing the complexity of multi-party social dynamics where attention and turn-taking are distributed.
Embodied AI Advancement: The ability to generate coherent listening reactions (not just speaking) is a critical step toward creating AI agents that can truly "engage" in a conversation rather than just waiting for a turn to speak.
Open Source: The authors have released the code and models, facilitating further research in multimodal social interaction.

In summary, PolySLGen represents a significant leap forward in embodied AI by successfully unifying speech, motion, and social awareness to generate natural, context-aware reactions in complex group environments.