Technical Summary: VideoFDB

Problem Statement

Natural human conversation is inherently full-duplex and audio-visual (AV2AV). Participants simultaneously speak, listen, and interpret nonverbal cues (e.g., nods, gaze shifts, gestures) without waiting for explicit turn boundaries. While recent advances in multimodal agents (e.g., Gemini Live, OpenAI Realtime) allow for streaming audio-visual input and speech output, existing evaluation benchmarks fail to assess these systems holistically.

Current benchmarks fall into three categories, each with significant limitations:

1. Full-duplex speech benchmarks: Evaluate turn-taking and interruption handling using audio alone, ignoring visual dynamics like gaze and gesture.
2. Audio-visual VLM benchmarks: Focus on video question answering (VQA), rewarding semantic correctness over continuous conversational dynamics.
3. Avatar benchmarks: Evaluate speaker and listener behaviors in isolation (split-role) rather than as continuous, joint interlocutors.

Consequently, there is no mechanism to evaluate whether an agent can function as an active conversational participant that continuously engages with human nonverbal dynamics in a full-duplex setting.

Methodology: VideoFDB Benchmark

The authors introduce VideoFDB, the first benchmark designed to evaluate full-duplex audio-visual-to-audio-visual (AV2AV) conversational agents.

Dataset Construction

• Source: 237 dyadic clips mined from real-world video calls, spanning 11 distinct nonverbal conversational dynamics.
• Annotation: A three-pass human annotation pipeline ensures high-quality curation. Clips are centered on specific events (e.g., a pause, a laugh, a gaze aversion) with precise timestamps, while preserving 1–3 turns of surrounding context.
• Taxonomy: The dataset categorizes dynamics into three groups based on evaluation requirements:
  • Perception-only: Dynamics where the agent must interpret user cues (e.g., Pause Handling, Gaze Avoidance with Pause, Adaptor Handling).
  • Shared (Perception + Generation): Dynamics requiring both interpretation and a corresponding response (e.g., Face Emotion Display, Nonverbal Backchanneling, Laughter).
  • Generation-only: Dynamics where the agent produces cues (e.g., Verbal Interruption, Turn-taking).

Evaluation Framework

VideoFDB employs a rubric-based "LM-as-judge" framework rather than exact-match metrics, acknowledging the non-deterministic nature of conversation.

• Perception Metrics:
  • Fluency: Coherence and naturalness of interaction.
  • Conversational Flow: Timing of responses relative to nonverbal cues (e.g., yielding the floor appropriately).
  • Semantic Grounding: Alignment of response content with perceived nonverbal cues.
• Generation Metrics:
  • Fluency: As above.
  • Dyadic Affect Match: Whether the agent's combined audio-visual affect corresponds to the user's state.
  • Nonverbal Cue Appropriateness: Whether produced cues are category-appropriate and well-timed.
• Timing Metrics: Introduces TOR-Alignment (Takeover-Rate Alignment), extending speech-only metrics to multimodal dynamics to measure if agents adhere to expected timing behaviors (e.g., staying silent vs. yielding) within specific windows.

Key Contributions

1. First Full-Duplex AV2AV Benchmark: VideoFDB establishes a foundation for systematically evaluating agents that must simultaneously perceive and generate verbal and nonverbal cues in natural dyadic conversation.
2. Comprehensive Taxonomy: It separates perception from generation behaviors across 11 nonverbal dynamics, moving beyond simple turn-taking to include gaze, affect, and body movement.
3. Failure Mode Analysis: The benchmark provides a structured method to identify specific systemic failures in current architectures, such as "captioning collapse" and "visual-stream ignorance."

Experimental Results

The authors evaluated a diverse set of models, including closed-source (Gemini 2.5/3.1, OpenAI Realtime), open-source (MiniCPM-o, MiniOmni2, VITA-1.5), and cascaded speech-to-avatar systems.

Key Findings

1. Performance Gap: No current model approaches human-level conversational naturalness. The largest deficits occur in fast social-coordination dynamics (e.g., Pause Handling, Nonverbal Backchanneling), with aggregate gaps between humans and models reaching up to 0.85 points on a 5-point scale.
2. Visual Input Limitations:
   • Captioning Collapse: Many models (e.g., MiniOmni2) treat visual inputs as captioning prompts, responding with visual descriptions ("I see you are...") rather than conversational engagement.
   • Visual-Stream Ignorance: Other models (e.g., gpt-realtime-mini) produce AV2A outputs that are paraphrases of their A2A counterparts, indicating the visual stream is ignored for context or timing.
   • Frame Rate Bottleneck: Increasing video sampling rates (e.g., from 1 FPS to 10 FPS) often degrades performance and speech coherence, suggesting a vision-speech fusion bottleneck where dense visual input overloads cross-modal attention.
3. Cascaded Architectures: Systems that chain a speech model with an avatar renderer (e.g., Gemini + Anam) preserve turn-taking discipline but fail to produce real-time nonverbal cues. They cannot insert cues during a user's turn due to architectural latency (2.8–3.5s behind ground truth) and the fact that motion is driven solely by generated speech.
4. A2A vs. AV2A: Counter-intuitively, audio-only (A2A) versions of these models often outperform their audio-visual (AV2A) counterparts on VideoFDB, as the addition of video introduces failure modes without providing conversational grounding benefits.

Significance and Claims

The paper positions VideoFDB as a critical step toward next-generation multimodal conversational agents. Its primary significance lies in:

• Establishing a Baseline: Providing the first standardized metric for full-duplex AV2AV interaction, moving beyond uni-modal or turn-based evaluations.
• Identifying Research Directions: The findings clarify that current systems rely on vision for explicit question answering but fail at the streaming joint audio-visual grounding required for natural conversation.
• Motivating End-to-End Solutions: The failure of cascaded speech-to-avatar pipelines highlights the need for end-to-end models capable of generating independent, real-time nonverbal motion alongside speech.

The authors conclude that while current systems are emerging, they remain well below human conversational naturalness. VideoFDB aims to accelerate the development of agents that can truly converse with humans by modeling the full spectrum of verbal and nonverbal dynamics.