💻 computer science

Face-to-Face: A Video Dataset for Multi-Person Interaction Modeling

This paper introduces Face-to-Face with Jimmy Fallon (F2F-JF), a 70-hour dataset of dyadic talk-show interactions designed to model reactive human conversation, and demonstrates its utility by improving the performance of speech-driven digital avatars through cross-person visual conditioning.

Original authors: Ernie Chu, Vishal M. Patel

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ernie Chu, Vishal M. Patel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot how to have a natural conversation. If you only show the robot videos of people giving speeches to an empty room (like a news anchor or a lecturer), the robot learns to talk, but it never learns how to listen and react. It doesn't know that when someone else nods, you should nod back, or when they look confused, you should pause to explain.

This paper introduces a solution to that problem: a massive new video library called Face-to-Face with Jimmy Fallon (F2F-JF).

Here is the breakdown of what they did, using some everyday analogies:

1. The Problem: The "Monologue" Trap

Most video datasets used to train AI are like a library of solo karaoke tracks. They have great audio and video of one person singing, but no one else is in the room. If you try to teach a robot to be a conversationalist using only these, it will be a terrible listener. It won't understand the "call-and-response" rhythm of a real chat.

2. The Solution: The "Talk Show" Goldmine

The authors went to the internet and downloaded 400 hours of The Tonight Show Starring Jimmy Fallon. Why? Because a talk show is the perfect recipe for a conversation:

The Host (Jimmy): He is the same person in every clip. He is the constant anchor.
The Guest: They change every time. This gives the AI a huge variety of faces, voices, and reactions to learn from.
The Interaction: They are looking at each other, laughing, nodding, and reacting in real-time.

3. The "Magic Scissors" (The Pipeline)

You can't just dump 400 hours of TV into a computer; it's too messy. The team built a semi-automatic "robot butler" to clean it up. Think of it like a high-tech film editor:

Step 1: The Bodyguard (Tracking): The robot scans the video to find exactly where the two people are standing. It cuts out everything else (the audience, the band, the stage) so it only sees the two faces.
Step 2: The Ear (Diarization): The robot listens to the audio to figure out who is speaking. It knows, "Okay, that's Jimmy talking now," and "Now, that's the guest."
Step 3: The Face ID (Verification): A few humans double-check the work to make sure the robot didn't get confused. Once the robot learns what Jimmy looks like, it can find him in thousands of clips automatically.
The Result: They turned 400 hours of raw TV into 70 hours of perfect, synchronized clips where the host and guest are talking directly to each other.

4. The Test Drive: The "Digital Avatar"

To prove this dataset is useful, they built a new kind of digital avatar.

Old Way: You give the AI a voice, and it makes a face move to match the words. It's like a ventriloquist dummy.
New Way (With F2F-JF): You give the AI the guest's video first. The AI watches the guest nod, smile, or look surprised. Then, it generates the host's reaction.
- Analogy: Imagine a dance partner. If you just tell the partner "move your feet," they might move randomly. But if you let them watch you dance first, they can mirror your moves and dance in perfect sync.

5. What They Found

They tested this new "reactive" avatar against the old "solo" avatar.

The Good News: The new avatar was slightly better at matching the emotions and body language of the conversation. It felt more natural because it was actually "watching" the other person.
The Catch: The improvement was small. It's like upgrading from a basic robot to a slightly more human-like robot. It's a great start, but there is still a lot of work to do to make it perfect.

Why Does This Matter?

This paper isn't just about Jimmy Fallon; it's about blueprints.

The Dataset: It gives researchers a clean, organized playground to study how two people interact.
The Method: They showed how to turn messy TV shows into clean data for AI.
The Future: This is the first step toward creating digital avatars that don't just talk, but actually converse. Imagine a virtual assistant that doesn't just answer your questions, but understands your tone, pauses when you're thinking, and smiles when you're happy. This dataset is the training ground for that future.

In short: They built a giant library of real conversations and used it to teach an AI how to be a better listener, moving us one step closer to digital friends that feel truly human.

1. Problem Statement

Current progress in modeling human conversation is hindered by a lack of high-quality datasets that capture reactive, dyadic interactions.

Limitations of Existing Data: Popular audio-visual datasets (e.g., LRS2, VoxCeleb2) primarily feature isolated speakers delivering monologues or loosely co-located individuals who do not interact. They lack the "call-and-response" structure where Person B's behavior at time $t_1$ is tightly conditioned on Person A's actions during $[t_0, t_1]$ .
The Gap: There is a need for sustained, reciprocal turn-taking data with consistent identities, fine-grained timing, and dense interaction labels to train models capable of generating reactive digital avatars that respond naturally to visual and auditory cues from a partner.

2. Methodology

A. Dataset Construction Pipeline (F2F-JF)

The authors introduce Face-to-Face with Jimmy Fallon (F2F-JF), a 70-hour dataset containing 14,123 clips of two-person talk-show exchanges. The construction utilizes a semi-automatic pipeline with two human-in-the-loop checkpoints:

Initial Vetting: Raw broadcast footage (400 hours) is processed using multi-person tracking to isolate segments containing exactly two visible identities, reducing the dataset to 140 hours.
Host Localization (Audio): Speaker diarization identifies the recurring host (Jimmy Fallon) by training a lightweight binary classifier on speech embeddings. This filters for segments where the host is actively speaking.
Host Face Modeling: Face crops from identified host speech windows are manually vetted to create a high-precision reference set. ArcFace embeddings are used to establish a robust visual identity for the host.
Identity Tracking & Assignment: The system re-processes clips, classifying faces as "host" or "guest" based on the learned threshold. Non-host faces are clustered online to assign consistent guest IDs.
Canonical Crop Derivation: To prepare data for avatar generation, the pipeline extracts synchronized, tightly cropped video pairs:
- Guest Context: The guest's video during turn $[t_0, t_1]$ .
- Host Response: The host's video during the subsequent turn $[t_1, t_2]$ .
- Clips are filtered for detection coverage and aspect ratio consistency, with missing frames (e.g., camera cuts) rendered as blackouts to preserve temporal alignment.

B. Reactive Digital Avatar Generator

The authors propose a baseline model to demonstrate the dataset's utility, extending the MultiTalk architecture (built on the Wan 2.1 Diffusion Transformer backbone).

Architecture: The model uses a frozen Wan 2.1 VAE and DiT backbone.
Conditioning Mechanism:
- Audio: Injected via a SingleStream cross-attention module (standard).
- Visual Context (Novelty): A new video-conditioned modulation pathway is added. A frozen I3D encoder processes the guest's preceding video ( $[t_0, t_1]$ ). Its output is projected into shift, scale, and gate vectors ( $\mu, \sigma, \gamma$ ) that modulate the transformer blocks after the audio conditioning.
Training Strategy: A two-stage fine-tuning process:
1. Optimize audio paths only.
2. Freeze audio weights and train the video projector and modulation offsets (initialized to zero to preserve the pretrained model's behavior initially).

3. Key Contributions

F2F-JF Dataset: A large-scale, curated dataset of 70 hours of dyadic conversations with frame-level identity tracks, bounding boxes, and synchronized audio. It is the first resource specifically designed to study sequential, cross-person dependencies in video generation.
Scalable Pipeline: A reproducible, semi-automatic pipeline for converting unstructured talk-show footage into high-fidelity, turn-taking exemplars with minimal human annotation (only ~10% of clips require manual vetting for bootstrapping).
Reactive Baseline Model: A proof-of-concept diffusion model that conditions host synthesis on the guest's preceding visual context. This provides an end-to-end blueprint for studying how models can leverage partner cues (gaze, head motion) for reactive generation.
Canonical Crops & Metadata: The release includes pre-processed, aligned host/guest crops specifically tailored for training reactive avatars, along with metadata for missing frames.

4. Results and Evaluation

The model was evaluated using standard metrics (Sync-C/D for lip-sync, FVD for video quality, Emotion-FID for emotional alignment) on a held-out test set.

Video Guidance Ablation: The authors varied the Classifier-Free Guidance (CFG) scale for the video branch ( $\lambda_{video}$ $λ_{v i d eo}$ ).
- Lip-Sync (Sync-C/D): Performance was optimal when video CFG was near zero, suggesting the model already captures timing cues from audio and aggressive visual conditioning can slightly disrupt synchronization.
- Realism & Emotion (FVD/Emotion-FID): These metrics improved as video guidance increased. Higher CFG encouraged motion patterns that better matched the empirical distribution of the guest's behavior, leading to more natural head movements and emotional alignment.
Quantitative Gains: While the improvements were "small but consistent," they validated that cross-person visual conditioning provides measurable benefits over audio-only baselines, particularly in capturing reactive non-verbal cues.
Qualitative Observations: Visualizations showed that stronger video guidance ($CFG=5$) resulted in host responses that mirrored the guest's head motion and timing more effectively than the audio-only baseline ($CFG=0$).

5. Significance and Future Work

Benchmarking Reactive Behavior: F2F-JF fills a critical gap in computer vision by providing a controlled environment to study dyadic, sequential behavior, moving beyond isolated speaker synthesis.
Blueprint for Interaction: The paper offers a complete pipeline (data extraction, processing, and modeling) that researchers can adapt for other talk-show formats or languages.
Future Directions: The authors plan to expand the dataset to include varied hosts (e.g., Jimmy Kimmel), languages, and cultural settings. They also identify the need for new evaluation metrics that go beyond lip-sync to measure conversational semantics such as turn-taking latency, backchannel timing, and intent-level mirroring.

In summary, this work establishes the foundational data and methodology required to move from "talking head" generation to interactive, reactive digital avatars capable of understanding and responding to human partners in real-time.