Technical Summary: Nemotron-Labs-Audio-Visual Flamingo (AV-Flamingo)

1. Problem Statement

Despite the dominance of video in global internet traffic and the human experience of the world as a continuous audio-visual sequence, current Large Language Models (LLMs) struggle to understand and reason over long-form, complex videos in a human-like manner. Existing Audio-Visual Large Language Models (AV-LLMs) primarily focus on short clips and often treat audio and visual modalities separately or rely on implicit cross-modal learning. Key challenges include:

• Data Scarcity: A lack of large-scale, high-quality, paired audio-visual data designed for joint reasoning, particularly for long-form content. Most available resources are unimodal or limited to short, recognition-oriented tasks.
• Architectural Limitations: Current models often exhibit a "visual bias," where deeper layers privilege vision and suppress latent audio representations, leading to poor joint perception.
• Temporal Reasoning: Existing approaches fail to effectively ground intermediate reasoning steps to specific timestamps in long audio-visual streams, resulting in performance degradation as video duration increases.
• Closed Ecosystems: The most capable models are often closed-source or open-weights-only, hindering reproducibility and open progress.

2. Methodology

The authors present Nemotron-Labs-Audio-Visual Flamingo (AV-Flamingo), a fully open state-of-the-art AV-LLM designed for joint understanding and reasoning over long and complex real-world videos. The approach is built on three core technical pillars:

A. Architecture

AV-Flamingo is based on the OmniVinci architecture, utilizing a Qwen2.5-7B base LLM. The system integrates five key components:

1. SigLip Vision Encoder: Extracts dense spatial features from images and video frames, processed through a "Spatial-Scale-then-Compress" Dynamic S2 module to handle high-resolution inputs without excessive token consumption.
2. AF-Whisper Audio Encoder: A Whisper-based encoder using a sliding-window mechanism to process arbitrary-length audio (segmented into 30-second chunks) without truncation.
3. Cross-Modal Alignment: Visual and audio token sequences are temporally interleaved and aligned using Constrained Rotary Time Embeddings (CRTE). This ensures the LLM can distinguish the ordering and relative timing of multimodal events.
4. LLM Backbone: Processes the interleaved token sequence autoregressively.
5. Streaming TTS: A decoder-only transformer module enabling real-time voice-to-voice interaction.

B. Data Curation: Audio-Visual-Skills (AV-Skills)

To address the data bottleneck, the authors introduce AV-Skills, a large-scale dataset comprising approximately 7 million training instances (captions and QA pairs) across short (up to 60s) and long (60s–15 min) videos.

• Sources: Derived from public datasets (e.g., YouTube-8M, LSMDC, MovieClips) and open-internet videos.
• Skills Focus: Unlike prior datasets focused on recognition, AV-Skills targets complex reasoning skills including:
  • Short-form: Relation reasoning, emotion change, temporal reasoning, spatial perception, causal reasoning, hallucination detection, and counting (audio/video).
  • Long-form: Needle-in-the-haystack reasoning, temporal referring/ordering, sub-scene understanding, holistic reasoning, topic-level reasoning, and detailed captioning.

C. Training Curriculum

The model undergoes a three-stage curriculum to progressively scale capabilities:

1. Pre-training (Short-Context): Initializes from OmniVinci and performs full fine-tuning on a mixture of unimodal data and AV-Skills-Short (max 5 min duration, 16K context). This preserves foundational skills while introducing cross-modal reasoning.
2. Mid-training (Long-Context): Expands to AV-Skills-Long data (max 15 min duration, 32K context). This stage focuses on long-form captioning, precise temporal grounding, and complex QA, retaining short-context skills via down-sampled data. The resulting model is AVF-Instruct.
3. Post-training (Reasoning): Trains on AV-Think, a dataset of question-answer-reasoning triplets. This stage employs Supervised Fine-Tuning (SFT) followed by Group Relative Policy Optimization (GRPO) reinforcement learning. The goal is to produce structured, step-by-step rationales anchored to specific timestamps. The final model is AVF-Think.

D. Temporal Audio-Visual Interleaved Chain-of-Thought (TAVIT)

A novel reasoning framework where intermediate reasoning steps are explicitly grounded to timestamps in both audio and visual streams (e.g., <t>...</t> tags). This improves temporal alignment, interpretability, and the model's ability to navigate long, complex streams.

3. Key Contributions

1. AV-Flamingo Model: A fully open frontier AV-LLM capable of joint audio-visual understanding and reasoning over long, complex real-world videos.
2. Scalable Recipe: A comprehensive framework for next-generation AV-LLMs, including internet-scale data curation (AV-Skills), targeted capability expansion, and temporally grounded reasoning (TAVIT).
3. Open Source Release: The authors open-source the model weights, training code, inference code, and associated techniques to support future research.
4. Performance: AV-Flamingo outperforms similarly sized open models by clear margins and remains highly competitive with, or surpasses, much larger open-weight and closed models (e.g., Gemini 2.5 Pro, GPT-4o) on long and complex tasks.

4. Experimental Results

The model was evaluated on 15+ benchmarks across audio, vision, omni-modal, and speech understanding.

• Omni-Modal Understanding: AVF-Think achieved 51.6% on WorldSense and 73.9% on DailyOmni, outperforming Qwen2.5-Omni and OmniVinci. On MMOU (a benchmark for long video reasoning), AVF-Think reached 60.2%, significantly outperforming the open-source baseline Minicpm-o 4.5 (46.8%) and approaching the proprietary Gemini-2.5 Pro (64.2%).
• Video Understanding: On Video-MME, AVF-Instruct scored 70.7% (without subtitles) and 71.2% (with subtitles), surpassing NVILA and OmniVinci.
• Audio Understanding: AVF-Instruct achieved the best overall average score (73.49) on MMAU, outperforming Audio Flamingo 3 and OmniVinci. It also set new state-of-the-art results on several ASR benchmarks, including 1.64 WER on LibriSpeech test-clean.
• Robustness: The model demonstrated strong generalization to unseen tasks and robustness on long-form videos where performance typically degrades in other models.

5. Significance and Claims

The paper claims that AV-Flamingo represents a significant step toward scaling open audio-visual intelligence beyond academic benchmarks. By leveraging internet-scale data and targeted post-training for reasoning, it addresses the critical gap in long-form, temporally grounded audio-visual understanding.

The authors emphasize that the release of the model, data, and code lowers the barrier for researchers without access to proprietary resources, fostering a more inclusive research ecosystem. They acknowledge limitations, such as potential source bias in the open-internet data and challenges with extremely dense or sparse evidence in very long videos, but position AV-Flamingo as a robust foundation for future open audio-visual systems. The work highlights that explicit temporal grounding and cross-modal reasoning are essential for moving beyond simple recognition to true understanding of complex real-world video content.