Technical Summary: InteracVid

Problem Statement

While Large Language Models (LLMs) have established text as the default medium for human–AI interaction, text alone is insufficient for multimodal assistants, avatars, and embodied agents that require expressive, grounded responses. Recent audio-video generative models have improved synthesis fidelity and synchronization, yet their supervision remains largely descriptive: models are trained to render content specified by input captions rather than to produce responses caused by external user interactions.

Existing datasets for interactive audio-visual generation suffer from three primary limitations:

1. Lack of Causal Structure: They often pair video context with text but fail to identify the specific reactive video segments or provide the user-side query that triggered them.
2. Modality Constraints: Many datasets are restricted to talking-heads or human speaking behaviors, lacking diverse object interactions, screen-based actions, or embodied demonstrations.
3. Missing Supervision: Current resources do not provide sufficient data to learn query-triggered, context-grounded interactive responses where the model must generate an audio-visual reaction appropriate to a specific user stimulus.

Methodology

Dataset Curation: InteracVid

The authors introduce InteracVid, the first open-source, large-scale dataset designed to address the missing supervision for interactive audio-visual response generation. The dataset is constructed from live-chat videos, capturing the natural interaction loop where streamers react to viewer comments.

The curation pipeline employs a metadata-aware, two-branch approach to handle the sparsity and noise of raw livestreams:

1. Real-Query Branch (Natural Interactions):

   • For videos with time-stamped live-chat metadata, the system aligns viewer comments with subsequent streamer utterances.
   • A sentence-level semantic timeline is constructed from subtitles to determine discourse roles (proactive monologue vs. reactive response).
   • An LLM filters pairs to ensure a plausible query–response causal link, extracting the full interaction structure: preceding context ($C$), user query ($Q$), and the real audio-visual response ($Y$).

2. Reconstructed-Query Branch (Synthetic Triggers):

   • For videos lacking chat metadata (e.g., VODs), the system identifies reactive response moments in the transcript based on discourse role classification.
   • A vision-language model (VLM) observes the local video context and response transcript to reconstruct a plausible user query that could have triggered the response.
   • Crucially, the target response ($Y$) is always extracted from real footage; only the input query condition is synthetic. This ensures the dataset remains grounded in genuine human reactions.

Data Processing and Quality Control

• Scale: The pipeline yields over 454K context-query-response triplets from 59K+ livestream videos, spanning conversation-centered, object-centric, procedural, embodied, and screen-based scenarios.
• Multimodal Filtering: Rigorous quality control ensures semantic alignment between speech and visuals. Clips with abrupt scene transitions are removed. Automatic Speech Recognition (ASR) is re-run on extracted clips to refine transcripts and word-level timestamps.
• Auxiliary Captioning: An auxiliary audio-visual caption ($T$) is generated for each clip, combining speech and visual observations to support training and analysis.

Experimental Framework

The authors propose a two-stage generation pipeline to tackle the complexity of interactive generation:

1. Interaction Planner: A Vision-Language Model (VLM) takes the context ($C$) and query ($Q$) to predict an interaction-aware textual description ($T$).
2. Audio-Video Co-Generator: A diffusion-based model (MOVA) takes the description ($T$) and context ($C$) to synthesize the final multimodal response ($Y$).

Experiments involve fine-tuning both the planner and the generator on InteracVid, comparing them against zero-shot baselines and oracle-caption upper bounds.

Key Contributions

1. InteracVid Dataset: A large-scale, high-quality dataset of 454K+ interactive triplets ($C, Q, Y$) that explicitly models the causal link between external stimuli and audio-visual responses. It is the only dataset supporting both audio-visual input/output with preceding context across diverse video domains.
2. Metadata-Aware Construction Pipeline: A novel method for extracting interactive clips from long, noisy livestreams, utilizing a dual-branch strategy (real and reconstructed queries) to maximize data coverage while maintaining causal integrity.
3. Human Validation: A ten-rater study confirms that both real and reconstructed queries are causal, natural, and temporally complete. Reconstructed queries scored slightly higher in naturalness due to the removal of slang and elliptical phrasing common in raw live chat.
4. Empirical Validation of Interaction-Structured Data: Experiments demonstrate that fine-tuning on InteracVid significantly improves both the upstream interaction planner and the downstream audio-visual generator, proving that interaction-structured data is a critical foundation for multimodal agents.

Results

Semantic Performance

On a held-out benchmark of 100 genuine live-chat queries:

• Generator Fine-Tuning: Fine-tuning the audio-visual co-generator (MOVA) on InteracVid yielded substantial gains across all semantic dimensions (Relevance, Appropriateness, Expressiveness). For instance, using Qwen3.5-9B as the planner, the Overall Score (OVRL) increased by +1.41.
• Planner Fine-Tuning: Fine-tuning the interaction planner also provided consistent improvements, though smaller than generator fine-tuning (e.g., +0.44 OVRL for Qwen3.5-9B). This suggests that learning to plan an interactive response is a significant bottleneck.
• Oracle Comparison: Even with ground-truth captions (Oracle), the generator only saw marginal gains (+0.09 OVRL) when fine-tuned, indicating that the primary limitation lies in planning the correct response rather than rendering it. However, the Oracle still outperformed the best learned planner by 0.71 OVRL, highlighting the gap in current planning capabilities.

Human Evaluation

An independent human evaluation of 10 raters on generated videos reproduced the system rankings obtained by the automatic VLM judge (Pearson $r \approx 0.96$).

• Human raters confirmed that fine-tuning the generator was the dominant factor for improvement (+1.68 OVRL with InternVL planner).
• The results validated that the model responds to the query rather than just the scene, confirmed via counterfactual tests where swapping the query resulted in a maintained relevance score only when the model was query-aware.

Audio-Visual Quality

Fine-tuning on InteracVid improved low-level generation metrics:

• Video: Improvements in Subject Consistency (+1.3%), Background Consistency (+1.2%), and Motion Smoothness (+39.2% reduction in error).
• Audio: Gains in Content Enjoyment (+3.4%), Content Usefulness (+5.7%), and Production Quality (+5.6%).
• Synchronization: Lip-sync distance improved by 4.1%, and AV-Align increased by 16.3%.
• Contextual Audio: Experiments showed that conditioning on preceding audio (in addition to the first frame) significantly improved speaker similarity and acoustic state, recovering a substantial fraction of the gap to real recordings.

Significance and Claims

The paper claims that interaction-structured data is a critical missing foundation for the next generation of multimodal agents. While current models excel at descriptive generation, they struggle with interactive generation—producing responses that are causally triggered by external stimuli.

The authors posit that:

1. Data is the Bottleneck: The lack of datasets pairing external stimuli with grounded, reactive audio-visual responses has hindered progress in interactive agents. InteracVid fills this gap.
2. Planning is the Primary Challenge: The experiments suggest that learning to plan a plausible interactive response (the "thinker" stage) remains a more significant bottleneck than the rendering of that response (the "actor" stage).
3. Generalizability: The dataset's diverse coverage (from object manipulation to screen-based actions) enables the study of audio-visual interaction in general cases, moving beyond the limitations of talking-head datasets.

The work concludes that InteracVid facilitates research into multimodal assistants capable of grounded, expressive, and interaction-aware audio-visual responses, providing the necessary supervision to move from text-based chatbots to true embodied and avatar-based agents.