💻 computer science

AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

This paper introduces AVCap, a comprehensive framework comprising the high-quality AVCap-100K dataset, a Detail-Aware GRPO-optimized model, and a specialized benchmark with atomic-level metrics, to overcome existing limitations in detailed audio-video joint captioning.

Original authors: Mingyang Wu, Kaituo Feng, Bohao Li, Kaixiong Gong, Zihao Yin, Xiangyu Yue

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingyang Wu, Kaituo Feng, Bohao Li, Kaixiong Gong, Zihao Yin, Xiangyu Yue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to watch a movie and tell you exactly what happens. In the world of artificial intelligence, this is called "video captioning." For a long time, these robots were like people who only had their eyes open; they could describe the colors and movements on the screen but were completely deaf to the soundtrack. They missed the scary creak of a floorboard, the specific accent of a character, or the way the music swelled to match a dramatic moment. Recently, scientists have built "multimodal" models that can both see and hear, but teaching them to describe a video with perfect detail is like trying to write a recipe for a complex dish while blindfolded and wearing earplugs. The robots often guess wrong, mixing up what they see with what they think they hear, or they skip the tiny, important details that make a story real. This paper tackles that problem by asking: How do we teach a robot to be a super-observant, detail-obsessed narrator who never misses a beat, a sound, or a glance?

The researchers behind this study, AVCap, decided that the old way of teaching these robots wasn't working because the "homework" they were given was too vague and the "grading" was too easy. To fix this, they built three new tools: a massive, ultra-detailed library of video descriptions, a new way to grade the robots' work that punishes tiny mistakes, and a special test to see if the robots are actually paying attention to the fine print.

First, they realized that existing video libraries were like a collection of blurry snapshots; they had the main events but lacked the texture. So, the team created AVCap-100K, a dataset containing 100,000 video clips paired with incredibly rich descriptions. Imagine taking a 60-second video and breaking it down not just into "a dog runs," but into "a golden retriever with a muddy paw pads sprints across the grass, its collar jingling, while a distant lawnmower hums." They achieved this by using a clever pipeline that first listens to the audio and watches the video separately to get the facts straight, and then combines them into one perfect story. This ensures the robot doesn't hallucinate (make things up) by guessing that a sound belongs to a visual event that isn't actually there.

Next, they tackled the problem of how to teach the robot to be this detailed. Previous methods were like a teacher who only gave a grade based on whether the story "sounded good" overall. If the robot missed a specific sound effect or got a color wrong, the teacher might not notice. The authors introduced a new training method called Detail-Aware GRPO (Da-GRPO). Think of this as a strict, hyper-focused editor. Instead of just giving a score, this editor acts like a detective. It takes the robot's description and asks a series of specific questions based on the real video, such as "What color was the shirt?" or "What sound did the door make?" If the robot's answer doesn't match the facts, it gets a penalty. This forces the robot to learn that getting the tiny, atomic details right is just as important as telling the main story.

Finally, to prove their new robot was actually better, they built AVCap-Bench and a new scoring system called AVCap-Score. Instead of just checking if the robot used the right words, this system checks if the robot actually knows the facts. It's like a pop quiz where the robot has to answer 20 specific questions about the video it just described. If it can't answer them correctly, it loses points, no matter how well-written the story was.

The results were impressive. Their new model, AVCap, trained with this strict "detective" method, became a champion at describing videos. On standard tests, it outperformed many other open-source models and even matched or beat some of the most expensive, commercial "black box" models used by big tech companies. For instance, on a specific test called AVCap-Bench, their 30-billion-parameter model scored 56.94, while the best commercial model they compared it to, Gemini-2.5-Pro, scored lower. They also showed that their model could describe videos with fewer "hallucinations" (mistakes where it invents details) and fewer missed events.

In short, this paper suggests that if you want a robot to truly understand a video, you can't just let it guess; you have to give it a massive library of detailed examples and then grade it like a strict teacher who checks every single fact. By doing this, they created a system that doesn't just summarize a video but captures the full, rich experience of seeing and hearing it, paving the way for robots that can tell stories with the precision of a human observer.

Technical Summary: AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

Problem Statement

The field of multimodal video understanding and generation faces three critical structural deficiencies that hinder the creation of detailed, fine-grained audio-visual captions:

Data Scarcity: There is a lack of high-quality public datasets annotated with fine-grained, temporally aligned audio-visual details. Existing resources are often vision-centric or lack the necessary semantic granularity.
Coarse Reward Signals: Reinforcement Learning (RL) methods in this domain rely on coarse, event-level reward signals (e.g., narrative checklists). These fail to optimize for atomic-level factual precision, often overlooking specific auditory or visual details and their synchronization.
Evaluation Gaps: Existing benchmarks and metrics focus on coarse event-level alignment rather than evaluating the informational density and atomic accuracy of joint audio-visual semantics.

Consequently, current methods struggle to produce captions that are both factually precise and semantically comprehensive, leading to hallucinations and omissions in fine-grained details.

Methodology

The authors propose a holistic framework comprising a new dataset, a novel training algorithm, and a specialized evaluation benchmark.

1. AVCap-100K Dataset

To address data scarcity, the authors curated AVCap-100K, a dataset of 100,000 video-caption pairs. The annotation pipeline employs a rigorous, multi-stage strategy to ensure atomic-level density and temporal alignment:

Dynamic Segmentation: Videos are segmented into 60-second units, with trailing clips merged to preserve semantic integrity.
Unimodal Disentanglement: To prevent cross-modal hallucinations, the pipeline extracts priors independently:
- Visual Branch: Processes mute video streams to generate descriptions free of auditory bias.
- Hierarchical Audio Branch: Uses source separation (Demucs) to isolate vocals and background music (BGM). Vocals undergo ASR and speaker diarization; BGM is segmented into 15-second windows for temporal dynamics.
Joint Reasoning: A large multimodal model (Qwen3-Omni-Thinking) synthesizes these unimodal priors into a joint caption, explicitly aligning visual events with auditory cues.
Quality Filtering: A rating-based filter evaluates captions across visual accuracy, audio fidelity, and alignment precision, retaining only high-confidence samples.

2. Detail-Aware GRPO (Da-GRPO)

The authors introduce AVCap, a model optimized via Detail-Aware Group Relative Policy Optimization (Da-GRPO). This approach reformulates the RL reward mechanism to enforce precise controllability:

Raise-Answer-Check Paradigm: Instead of holistic scoring, a Judge Model generates a set of $N$ atomic-level probes (questions) derived from the ground truth, categorized into Visual, Audio, and Joint interactions.
Dense Reward Aggregation: For a generated caption, the Judge Model answers these probes using the caption as the sole context. The predicted answers are semantically verified against the ground-truth answers.
Reward Signal: The final reward is the normalized average of semantic similarity scores across all probes. This creates a dense, detail-aware signal that strictly measures whether the caption preserves fine-grained visual and auditory facts, guiding the policy to minimize hallucinations.

3. AVCap-Bench & AVCap-Score

To evaluate performance at the atomic level, the authors established AVCap-Bench (1,000 held-out examples) and the AVCap-Score metric.

QA-Based Verification: Similar to the training reward, the metric uses a Judge Model to generate 20 atomic questions from the ground truth.
Scoring: The generated caption is used to answer these questions. The answers are compared against the ground truth to compute a semantic similarity score (0–5).
Granularity: The score is decomposed into Visual-Score, Audio-Score, and Joint-Score, providing a precise standard for evaluating joint semantic alignment beyond surface-level lexical overlap.

Key Contributions

AVCap-100K: A high-quality dataset of 100K video-caption pairs annotated with superior audio-visual density, resolving the scarcity of fine-grained training data.
AVCap Model: A model trained with Da-GRPO that achieves state-of-the-art performance among open-source models and competes with proprietary systems.
AVCap-Bench & AVCap-Score: The first specialized benchmark and metric designed to rigorously quantify fine-grained informational content at the atomic level.

Experimental Results

Extensive evaluations demonstrate the efficacy of the proposed framework:

Performance on General Benchmarks: On Video-SALMONN-2, AVCap-30B achieves a total score of 32.7 (reducing hallucination rates to 10.3%) and outperforms existing open-source baselines. On UGC-VideoCap, it reaches an average score of 85.1.
Comparison with Proprietary Models: AVCap-30B matches or surpasses proprietary models like Gemini-2.5-Pro on several evaluations, including WorldSense (34.3 vs. 33.8) and AVCap-Score (56.94 vs. 50.62). While Gemini-2.5-Pro retains a lead on DailyOmni (60.2 vs. 52.1), AVCap-30B remains competitive across the broader suite of benchmarks.
Fine-Grained Evaluation: On the AVCap-Score benchmark, AVCap-30B achieves a total score of 56.94, significantly outperforming open-source baselines (e.g., AVoCaDO at 46.05) and proprietary models (Gemini-2.5-Pro at 50.62).
Ablation Studies: Results confirm that the atomic-level reward signals of Da-GRPO consistently outperform coarse-grained alternatives (N-gram or holistic rewards) and that the dataset (AVCap-100K) provides substantial gains over standard SFT.

Significance and Claims

The paper claims that AVCap redefines the standards for detailed audio-visual captioning by addressing the "structural deficiencies" of the field. Its significance lies in:

Data-Centric Improvement: Demonstrating that a high-quality, fine-grained dataset (AVCap-100K) is critical for unlocking the potential of multimodal models, allowing a 7B model to surpass previous SOTA baselines.
Methodological Shift: Proving that atomic-level reward signals (via Da-GRPO) are superior to coarse event-level rewards for grounding models in factual reality, effectively reducing hallucinations and improving temporal synchronization.
Evaluation Rigor: Establishing a new evaluation paradigm (AVCap-Score) that moves beyond lexical overlap to measure the actual informational equivalence between video and text, providing a more reliable metric for fine-grained understanding.

The authors position their work as a comprehensive solution that enables models to deliver captions that are not only fluent but also semantically detailed and factually aligned with the complex interplay of vision and sound.