💻 computer science

A Benchmarking Methodology to Assess Open-Source Video Large Language Models in Automatic Captioning of News Videos

This paper introduces a comprehensive benchmarking methodology featuring novel fidelity metrics (TFS and EFS) to evaluate eight open-source Video Large Language Models on news video captioning, revealing that Gemma 3 outperforms other models while highlighting the limitations of standard lexical and semantic metrics in this domain.

Original authors: David Miranda Paredes, Jose M. Saavedra, Marcelo Pizarro

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Miranda Paredes, Jose M. Saavedra, Marcelo Pizarro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, chaotic library. This library doesn't have books; it has millions of video clips of news broadcasts. Every day, thousands of new clips arrive. Right now, a team of human librarians has to watch every single clip, write a short summary, and tag it with keywords so people can find it later. It's slow, expensive, and exhausting.

Enter Video Large Language Models (VidLLMs). Think of these as super-smart AI robots that can watch a video and instantly write a summary for you. The big question is: Which robot is the best librarian?

This paper is a "taste test" where the authors put eight of the smartest open-source AI robots to work on news videos to see which one writes the best summaries. Here is the breakdown of their experiment in simple terms:

1. The Test Track: Two Different Libraries

To make sure the robots were being tested fairly, the authors used two very different "libraries" (datasets):

The Chilean Library (ChTV): About 1,300 clips from a local TV station. These were short, fast-paced clips, and the "answers" (the human-written summaries) were often just short lists of keywords in Spanish.
The BBC Library: Nearly 10,000 clips from the BBC. These were in English, slightly longer, and had more detailed, story-like summaries.

2. The Old Rulers vs. The New Rulers

Usually, when we test AI, we use standard rulers (metrics) like ROUGE or METEOR.

The Problem: Imagine you ask a robot to describe a picture of a cat.
- Human Answer: "A fluffy orange cat sleeping on a sofa."
- Robot Answer: "There is a cat, it is orange, and it is asleep."
- The Old Ruler: "You didn't use the exact same words! You get a low score!"
- The Reality: Both answers are perfect. The old rulers are too obsessed with matching exact words, which doesn't work well for news where the AI might tell the story differently than a human editor.

The authors realized these old rulers were broken for this job. So, they invented two new rulers specifically for news:

The "Theme Check" (Thematic Fidelity Score): Does the summary get the big picture right? If the video is about a political election, does the summary say "Politics," or does it accidentally say "Sports"?
The "Name Check" (Entity Fidelity Score): Did the AI catch the important names? If the news is about "President Biden," does the summary say "President Biden," or just "a man in a suit"?

3. The Race Results

The authors ran the eight robots through the gauntlet. Here is who won:

The Champion: Gemma 3. This robot was the most consistent all-rounder. It understood the themes best, caught the most names, and wrote summaries that sounded the most like a human editor. It was the "smartest" librarian.
The Runner-Up: Qwen-VL. This one was a strong second place, doing very well on almost everything.
The "Word Match" Specialist: LLaVA-OneVision. This robot was great at matching the exact words and sentence structures (like a parrot repeating a sentence perfectly), but it wasn't as good at understanding the deeper meaning or catching specific names compared to Gemma 3.

4. The "Gotchas" (What Went Wrong)

The study found some funny and important flaws in how we usually test AI:

The "Static Frame" Trap: News videos often show a reporter standing still in front of a green screen. Some AIs got confused and thought the video was just about the background, not the news story.
The "Keyword" Problem: On the Chilean dataset, the human answers were just lists of keywords (e.g., "Crime, Rain, Police"). The AIs tried to write full sentences. Because the human answers were so short and different from the AI's sentences, the AI got punished by the old rulers, even though it was doing a great job.
The "Name" Gap: In the Chilean dataset, the human answers rarely included full names (just "a police officer"). The AI tried to guess the name (e.g., "Officer Smith"), but since the human answer didn't have the name, the AI got a zero score for "Name Check," even though it was being creative.

5. The Big Takeaway

The main lesson from this paper is: Don't just ask "Did the AI use the same words?"

For news videos, we need to ask:

Did it get the topic right? (Theme Check)
Did it get the names right? (Name Check)

When they used these new questions, Gemma 3 clearly stood out as the best tool for automatically summarizing news videos. It's the robot that doesn't just repeat words, but actually understands the story.

In short: We finally have a way to measure if an AI is actually good at reading the news, rather than just good at guessing the right words. And right now, Gemma 3 is the top student in the class.

1. Problem Statement

The automatic generation of textual descriptions (captions) for news videos is critical for indexing, retrieval, and accessibility but remains largely a manual, labor-intensive process. While Video Large Language Models (VidLLMs) have shown promise in general video understanding, there is a significant lack of comprehensive evaluation specifically within the news domain.

Challenges: News videos often occur in uncontrolled, noisy environments with complex narratives.
Evaluation Gap: Existing standard metrics (e.g., ROUGE, METEOR, BERTScore) are ill-suited for this domain. They suffer from:
- Surface-form dependence: Penalizing valid semantic rephrasing common in news reporting.
- Static-frame insensitivity: Failing to capture dynamic video content.
- Function-word inflation: Over-rewarding common words (articles, prepositions) rather than factual accuracy.
- Lack of thematic/factual grounding: Inability to verify if the caption correctly identifies the news topic or specific named entities (people, organizations).

2. Methodology

The authors conducted a comparative study of eight state-of-the-art open-source VidLLMs using a novel, multi-dimensional evaluation framework.

2.1 Models Evaluated

The study benchmarked the following models (released 2024–2025):

Gemma 3 (Google DeepMind)
Vript (Shanghai Jiao Tong Univ.)
InternVL 3.5 (OpenGVLab)
Video-LLaMA 2 & 3 (Alibaba DAMO Academy)
LLaVA-NeXT-Video & LLaVA-OneVision (ByteDance/NTU)
Qwen-VL (Alibaba Cloud)

2.2 Datasets

Two complementary benchmark datasets were constructed:

ChTV (Chilean TV News): ~1,345 clips from a local Chilean station. Features diverse topics, variable lengths (10s–5min), and annotations in Spanish (translated to English for evaluation). Annotations are often keyword-based or short editorial tags.
BBC News: ~9,838 clips from the BBC. Features English editorial descriptions, shorter durations (mostly <180s), and richer named entity content.

2.3 Evaluation Metrics

The authors employed a hybrid metric suite:

Traditional Lexical/Semantic Metrics: METEOR, ROUGE-L, BERTScore, CLIPScore, Text Similarity (Sentence-BERT), and Mean Reciprocal Rank (MRR).
Novel Fidelity Metrics:
- Thematic Fidelity Score (TFS): Measures how well the generated caption preserves the high-level thematic structure of the ground truth. It uses zero-shot classification (DeBERTa-v3) to assign one of 15 news themes (e.g., Politics, Crime, Economy) and calculates the micro-averaged F1-score.
- Entity Fidelity Score (EFS): Measures the accuracy of named entity (Person, Location, Organization) coverage. It uses spaCy for extraction and RapidFuzz for fuzzy matching (threshold $\theta=85$ ) to calculate the harmonic mean of Precision and Recall.

3. Key Contributions

Benchmark Datasets: Introduction of two large-scale, real-world news video datasets (ChTV and BBC) with editorial annotations, filling a gap in domain-specific video captioning benchmarks.
Critical Analysis of Standard Metrics: Demonstration that traditional metrics (ROUGE, METEOR, BERTScore) have limited discriminative power in news captioning due to the nature of editorial text vs. generative output.
Novel Evaluation Framework: Proposal of TFS and EFS to directly assess thematic preservation and factual entity grounding, addressing the specific needs of news indexing.
Comprehensive Benchmarking: A rigorous comparison of eight leading open-source models across two distinct linguistic and editorial contexts.

4. Results and Findings

4.1 Model Performance

Top Performer: Gemma 3 consistently achieved the highest overall performance across both datasets and most metrics. It excelled in Text Similarity, CLIPScore, MRR, and Entity Fidelity (on BBC).
Runner-up: Qwen-VL was a consistent second-place performer, particularly strong in TFS on the ChTV dataset.
Specialized Strengths: LLaVA-OneVision achieved the highest ROUGE-L and BERTScore F1 scores, indicating strong token-level alignment, though it lagged in thematic and entity grounding compared to Gemma 3.

4.2 Metric Analysis

Standard Metrics: Showed low variance and ceiling effects (e.g., BERTScore > 0.77 for all models), failing to distinguish between high and low-quality models effectively.
Thematic Fidelity (TFS): All models achieved high scores (>0.83), suggesting that zero-shot classification can broadly identify topics. However, the narrow spread (0.026–0.050) limits its ability to differentiate model quality alone.
Entity Fidelity (EFS):
- BBC Dataset: Showed clear differentiation. Gemma 3 dominated (EFS = 0.314), significantly outperforming others (next best ~0.198). This highlights Gemma 3's superior ability to extract and ground specific named entities.
- ChTV Dataset: Scores were near-zero for all models. This was attributed to the "keyword-style" ground truth (translated tags) which lacked explicit named entities, rendering the metric uninformative for this specific dataset.

4.3 Qualitative Insights

Case Studies: Gemma 3 successfully identified complex narratives (e.g., distinguishing a migrant crisis from a tourist boat) and specific named entities (e.g., "Lieutenant Daniel Medina"), whereas weaker models (e.g., LLaVA-NeXT-Video) often hallucinated generic visual descriptions or missed critical context.
Shuffled-Pairs Validation: Confirmed that high-performing models (Gemma 3) generate descriptions specifically relevant to the video, while lower-performing models produce generic text that matches references by chance.

5. Significance and Conclusion

This work establishes a new standard for evaluating VidLLMs in the news domain. It proves that Gemma 3 is currently the most balanced and capable model for automatic news captioning, particularly regarding factual grounding and thematic accuracy.

Key Implications:

Metric Evolution: The field must move beyond surface-level n-gram overlap toward fidelity metrics (TFS/EFS) that measure factual and thematic correctness.
Data Quality: The effectiveness of entity-based metrics depends heavily on the richness of the reference data (annotations must contain named entities).
Future Directions: The authors suggest future work should focus on creating richer, purpose-built ground-truth annotations, exploring ensemble methods (combining Gemma 3's grounding with LLaVA's visual alignment), and deploying these models in real-world editorial workflows.

The study concludes that while current models are capable of generating coherent text, Gemma 3 stands out for its ability to align with the specific factual and thematic requirements of professional news production.