💻 computer science

Talking Together: Synthesizing Co-Located 3D Conversations from Audio

This paper presents a novel dual-stream system that synthesizes realistic, spatially aware 3D facial animations for two co-located participants from mixed audio by modeling their dynamic spatial relationships and mutual gaze, supported by a large-scale dataset of over 2 million conversational pairs.

Original authors: Mengyi Shan, Shouchieh Chang, Ziqian Bai, Shichen Liu, Yinda Zhang, Luchuan Song, Rohit Pandey, Sean Fanello, Zeng Huang

Published 2026-03-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mengyi Shan, Shouchieh Chang, Ziqian Bai, Shichen Liu, Yinda Zhang, Luchuan Song, Rohit Pandey, Sean Fanello, Zeng Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to recreate a real-life conversation between two friends, but you only have a recording of their voices mixed together in one audio file. Most current technology is like a bad video call: it generates two separate "talking heads" floating in a void, looking straight ahead, completely ignoring each other. They might be talking, but they aren't interacting.

This paper introduces a new system called "Talking Together" that fixes this. It takes that single, mixed audio recording and generates a complete, realistic 3D scene where two people are standing face-to-face, looking at each other, nodding, and reacting naturally.

Here is how they did it, broken down into simple concepts:

1. The "Two-Stream" Kitchen

Think of the system as a kitchen with two chefs working side-by-side.

The Problem: Usually, if you give a chef a mixed pot of soup (the audio), they can't tell which ingredients belong to which dish.
The Solution: This system uses a Dual-Stream Architecture. Imagine two chefs (one for Person A, one for Person B) working in the same kitchen. They have a special "cross-talk" system. While Chef A is cooking, they can peek at what Chef B is doing to see if they need to pause, nod, or look surprised. This allows the system to figure out who is speaking and who is listening, even when both are talking at once.

2. The "Role-Playing" Badges

To help the chefs know who is doing what, the system gives them digital badges.

Speaker vs. Listener: The system analyzes the audio to guess who is talking and who is listening. It then gives the "Speaker" a badge that says, "You are talking, move your lips!" and the "Listener" a badge that says, "You are listening, nod and make eye contact!"
The Magic: Even if the audio is messy or both people talk at the same time, these badges help the system keep the two characters' animations distinct and realistic.

3. The "Eye-Contact" Coach

One of the hardest parts of a real conversation is looking at the other person. Old methods often made characters stare blankly at the camera.

The Fix: The researchers added a special "Eye Gaze Loss." Think of this as a strict coach standing over the animation, checking every frame. If the characters aren't looking at each other when they should be, the coach gives them a "penalty." This forces the AI to learn the subtle art of mutual eye contact and natural glances away, making the conversation feel alive.

4. The "Text-to-Scene" Director

In the past, you couldn't tell the computer where the people should stand. They just appeared in random spots.

The Innovation: This system lets you type a simple instruction, like "They are arguing across a table" or "They are whispering intimately."
How it works: The system uses a Large Language Model (like a smart assistant) to translate your text into 3D coordinates. It acts like a director telling the actors, "Okay, you stand here, and you stand there," before the animation even starts.

5. The "Data Diet"

AI models are like athletes; they need to eat a lot of data to get strong. The problem was that there wasn't enough high-quality video of two people talking in the same room.

The Feast: The team created a massive diet for their AI:
1. The "Wild" Buffet: They scraped over 2 million video clips of real people talking in the wild to learn how humans actually interact.
2. The "Sterile" Supplement: They also created a synthetic dataset by taking clean, high-quality videos of single people and mixing them together artificially. This taught the AI perfect lip-syncing (matching mouth movements to words) without the blur and noise of real-world videos.
The Result: By training on both, the AI learned to be both socially smart (from the wild videos) and technically precise (from the clean videos).

The Bottom Line

The result is a system that doesn't just make two people talk; it makes them converse. It captures the "dance" of a real conversation: the head tilts, the eye contact, the shared space, and the reaction to what the other person is saying. It moves us from a "video conference" feel to a "real-life meeting" feel, all generated from a single audio file.

Technical Summary: Talking Together: Synthesizing Co-Located 3D Conversations from Audio

Problem Definition

The paper addresses the challenge of generating complete, photorealistic 3D facial animations for two interacting, co-located participants from a single mixed audio stream. While existing audio-driven 3D animation research has advanced significantly, it predominantly focuses on single-speaker generation or produces "disembodied" talking heads that resemble video conference calls. These prior approaches fail to model the physical co-location of participants, ignoring crucial non-verbal cues derived from a shared 3D space, such as responsive head movements, mutual eye gaze, and dynamic spatial relationships (relative position and orientation). The authors identify a scarcity of large-scale, high-fidelity 3D training data capturing natural dyadic conversations as a primary barrier to progress in this domain.

Methodology

1. Data Curation Pipeline

To overcome data scarcity, the authors introduce a novel pipeline to curate two complementary datasets:

Dyadic Conversation Dataset: A massive corpus of over 2 million interacting pairs extracted from "in-the-wild" videos. This dataset captures natural conversational dynamics and spatial relationships but suffers from occlusions and lower resolution. The pipeline includes scenario filtering (removing video calls), quality control (filtering occluded/blurred faces), audio source separation (using Looking to Listen), and 3D face reconstruction.
Synthetic Dubbing Dataset: A high-fidelity dataset generated by re-assembling segments from high-quality, frontal-facing single-person videos. This creates "pseudo-conversations" with perfect ground-truth speaker masks and precise lip motion, essential for training a robust lip-sync expert.

2. Model Architecture

The core of the system is a dual-stream diffusion architecture built upon a shared U-Net backbone:

Dual-Stream Design: Two parallel streams process noisy inputs for each participant simultaneously, promoting a unified, speaker-agnostic representation of facial motion.
Cross-Attention Mechanisms: Dual-speaker cross-attention layers within the decoder enable bidirectional information exchange between the two streams. This allows the model to capture mutual influence, turn-taking dynamics, and reactive non-verbal behaviors.
Speaker Role Embeddings: Learnable embeddings ( $e_{speak}$ and $e_{listen}$ ) are interpolated based on speaker probability masks to dynamically encode the conversational state (speaking vs. listening) for each participant.
Dynamic Conditioning: The model is conditioned on a comprehensive vector containing Wav2Vec audio features, role embeddings, speaker probability masks, and crucially, the initial 3D translations of both heads.

3. Training Strategy

The authors employ a two-stage training strategy to balance interaction dynamics with articulation precision:

Stage 1 (Pre-training): The model is pre-trained on the massive, diverse Dyadic Conversation dataset to learn general audio-visual alignment and interactive behaviors (head rotation, translation, expression).
Stage 2 (Fine-tuning): The model is fine-tuned on the high-quality Synthetic Dubbing dataset. During this stage, the loss is restricted primarily to the lip and jaw articulation parameters of the speaking participant, correcting the lip-sync ambiguities inherent in the "in-the-wild" data.

4. Spatial Control and Gaze Modeling

Text-to-Spatial Control: To enable intuitive control over the relative positioning of participants, the system uses a Large Language Model (LLM) in a few-shot prompting manner. Users provide natural language prompts (e.g., "standing side-by-side"), and the LLM predicts the initial 3D translation coordinates, which condition the diffusion model.
Auxiliary Eye Gaze Loss: A targeted loss function is introduced to promote realistic mutual eye contact. This loss is applied selectively to samples with significant head movement, encouraging the model to learn natural gaze shifts and eye contact patterns.

Key Contributions

Large-Scale Dataset Curation: The creation of an automated pipeline yielding a 2-million-pair dyadic conversation dataset and a high-fidelity synthetic dubbing dataset for robust training.
Dual-Stream Diffusion Architecture: A novel model with a shared U-Net backbone, cross-attention, and FiLM conditioning that effectively disentangles mixed audio and models speaker-listener interactions.
Mixed-Data Training Strategy: A two-stage approach that leverages massive real-world data for interaction grounding and high-quality synthetic data for precise lip synchronization.
Controllable Spatial Dynamics: The first system to explicitly synthesize co-located 3D performances with controllable spatial layouts via text prompts and realistic mutual eye gaze via a specialized loss function.

Results

The method was evaluated against state-of-the-art baselines, including single-speaker models (CodeTalker, SelfTalk, FaceFormer) and existing dual-speaker interaction models (DualTalk, L2L).

Quantitative Performance: The proposed method achieved the best performance across nearly all metrics, including Fréchet Distance (FD) for realism, Paired FD (P-FD) for interaction coherence, and Mean Squared Error (MSE) for expression, rotation, and translation accuracy. Notably, it significantly outperformed baselines in eye gaze and listener movement metrics.
Human Evaluation: In a forced-choice user study involving 19 participants, the full model was selected as the best in all categories (Lip Quality, Speaker/Listener Movements, Interaction Quality, and Eye Gaze) with a preference rate of approximately 70-79%, significantly outperforming the best baseline (DualTalk).
Ablation Studies: Experiments confirmed that removing any component (e.g., the second training stage, cross-attention, or gaze loss) degraded performance, validating the necessity of the full pipeline.
Generalization: The model demonstrated superior generalization on an Out-of-Distribution (OOD) dataset containing unseen identities, outperforming specialized baselines.

Significance and Claims

The paper claims to be the first to explicitly model the dynamic 3D spatial relationship—including relative position, orientation, and mutual gaze—between two interacting participants from a single audio stream. By moving beyond the "video conference" paradigm of isolated talking heads, the system generates fluid, controllable, and spatially aware dyadic animations. The authors position this work as a foundational step toward truly immersive social interactions in virtual reality (VR) and telepresence, where the organic flow of face-to-face dialogue and non-verbal cues are critical for believability. The work emphasizes that realistic in-person dialogue requires not just accurate lip-sync, but the synthesis of a shared 3D space where participants react to one another in a physically meaningful way.