💻 computer science

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

This paper introduces CLIP-CC-Bench, a novel evaluation framework comprising 90-second movie clips with expert-written paragraph descriptions and a robust LLM-based ensemble methodology to assess and rank the long-form video description capabilities of 17 state-of-the-art video-language models, addressing the gap left by existing short-clip and QA-only benchmarks.

Original authors: Mukhtiar Ali, Harsh Dubey, Sugam Mishra, Chulwoo Pack

Published 2026-08-06

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mukhtiar Ali, Harsh Dubey, Sugam Mishra, Chulwoo Pack

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to watch movies and tell you what happens. For a long time, scientists have been great at testing these robots with short, 5-second clips, asking them to write a single sentence like "A dog runs." But what happens when you ask the robot to watch a whole scene, maybe a minute and a half long, and write a full paragraph describing the story, the characters' feelings, and the tiny details? This is the tricky frontier of "Video-Language Models." Think of these models as super-smart students who have read the internet and watched millions of videos. The big question isn't just if they can see a dog; it's if they can understand a complex story, keep the timeline straight, and describe it without making things up. Until now, there hasn't been a fair way to grade their homework on these long stories. Old grading tools were like checking for spelling mistakes; they counted how many words matched but missed the point of the story entirely.

Enter CLIP-CC-Bench, a new, rigorous report card designed by researchers at South Dakota State University to test how well these AI students can write long-form movie descriptions. Instead of just looking for matching words, the researchers built a "panel of expert judges" made of five different advanced AI systems. These judges don't just count words; they read the story to understand the meaning, checking both the big picture (the plot) and the tiny details (who was wearing what). They tested 17 of the smartest video-AI models out there. The results were eye-opening: while some models are getting really good at the general story, almost all of them struggle to get the small details right, and none of them are perfect yet. The study proves that simply making models bigger doesn't automatically make them better storytellers, and it provides a new, transparent way to see exactly where they are failing so we can fix them.

The Problem: The "Spelling Bee" vs. The "Book Report"

For years, scientists tested video-AI models using a method that was a bit like a spelling bee. They would show the AI a short clip and ask it to describe it. Then, they would compare the AI's answer to a human's answer using old-school math tools like BLEU or ROUGE. These tools are great for checking if the AI used the same words as the human, but they are terrible at understanding meaning. If a human wrote, "The man is sad," and the AI wrote, "The guy is crying," the old tools might say, "Bad job, you didn't use the word 'man'!" even though the meaning is perfect.

Furthermore, most tests only used short clips and asked for one-sentence answers. It's like testing a student's ability to write a novel by only asking them to write a tweet. Real life—and real movies—are long, complex, and full of details that happen over time. The researchers realized that to truly know if an AI understands video, we need to ask it to write a full paragraph about a 90-second movie scene and grade it like a real book report.

The Solution: A New Movie Test

To fix this, the team created CLIP-CC-Bench. Imagine they took 5 hours of movies and chopped them into 200 distinct clips, each about 90 seconds long. These weren't just random clips; they were carefully chosen to be visually rich and complex, featuring things like character interactions, camera movements, and changing scenes.

Here is the clever part: To make sure the AI was actually watching the video and not just guessing based on famous names it memorized from the internet, the researchers banned all proper nouns. You won't see names like "Harry Potter," "New York," or "Toyota" in the answers. Instead, the human experts wrote descriptions like "a man in a red jacket" or "a luxury car." This forces the AI to describe what it actually sees, not what it remembers.

The Judges: A Panel of Five

How do you grade a paragraph? You can't just ask one AI to grade another; that's like asking a student to grade their own homework. So, the researchers built a "jury" of five different, state-of-the-art AI embedding models. Think of these as five different English teachers with different styles.

The Coarse Judge: Looks at the whole paragraph to see if the general story matches. Did the AI get the main plot right?
The Fine Judge: Looks at the sentence level. Did the AI mention the specific actions, the colors, and the order of events correctly?

The system combines these two views. If an AI writes a vague paragraph that gets the plot right but misses all the details, the "Fine Judge" will give it a low score. If it lists random details that don't make sense, the "Coarse Judge" will ding it. The final score is a harmonic mean, which means the AI has to be good at both to get a high grade.

The Findings: Who Passed the Test?

The researchers ran 17 different video-language models through this gauntlet. Here is what they found:

The Top Performer: VideoLLaMA3 took the top spot, earning a perfect Borda rank (a consensus score based on how every judge ranked it) from all five judges. However, its actual average score was 0.67 (on a scale where 1.0 would be perfect), showing that even the winner has significant room for improvement.
The Runner-Up: mPLUG-Owl3 came in a close second.
The Gap: There is a clear gap between the top models and the rest. The best model, VideoLLaMA3, only achieved an average score of 0.67. This tells us that even the best AI today still has a long way to go.
The Big Problem: The study found a consistent "Coarse-Fine Gap." The models were much better at getting the general story (Coarse) than at getting the specific details (Fine). For example, a model might say, "Two men fight in the snow," which is a good summary. But it might miss that one man was holding a gun, or that the snow was falling heavily. The "Fine" scores were often much lower, sometimes as low as 0.47, showing that the models are still struggling with the nitty-gritty details.
Architecture Matters: The study found that models built on "Transformer" architectures (a specific type of AI design) generally did better than those built for specific, narrow tasks. This suggests that being a general-purpose "smart" model is more important for storytelling than being a specialized "video" model.

Why This Matters

This paper doesn't just say "AI is getting better." It gives us a precise map of where they are failing. It proves that we can't just rely on old methods that count words, and we can't just make models bigger and hope for the best. By using this new, multi-judge system, the researchers showed that current models are like students who can summarize a movie but forget the details of the characters' outfits or the specific order of events.

The team released all their data, code, and the results of all 17 models to the public. This means other scientists can use this same test to build better models, ensuring that the next generation of video-AI doesn't just "guess" the story, but truly understands it. The path forward is clear: we need models that can bridge the gap between the big picture and the tiny details, and CLIP-CC-Bench is the ruler we'll use to measure that progress.

Technical Summary: CLIP-CC-Bench

Problem Statement

Current benchmarks for Video-Language Models (VLMs) predominantly focus on short video clips and single-sentence captions, or utilize question-answering formats that test discrete reasoning rather than generative capability. Existing evaluation metrics, such as BLEU, ROUGE, and METEOR, rely on surface-level n-gram overlap, failing to capture the semantic coherence and discourse structure essential for long-form, paragraph-level descriptions. While embedding-based methods and "LLM-as-judge" frameworks have emerged, the former often suffer from limited context windows unsuitable for paragraphs, and the latter exhibit significant consistency issues and "black-box" opacity, particularly when evaluating complex, multi-event narratives. Consequently, there is a lack of a rigorous, reproducible framework to assess whether VLMs can generate accurate, holistic descriptions of minute-scale video content.

Methodology

The authors introduce CLIP-CC-Bench, an evaluation suite designed to benchmark long-form video description capabilities through a multi-granular, ensemble-based approach.

Dataset Construction

The benchmark consists of 200 curated movie clips, totaling approximately 5 hours of content. Each clip is standardized to 90 seconds and sourced from over 140 distinct films and series spanning 1959–2024.

Bias Mitigation: A critical design principle is the systematic elimination of proper nouns (e.g., character names, specific locations, brands) from ground-truth descriptions. Annotators were trained to use descriptive alternatives (e.g., "a man in a black jacket" instead of a character name) to ensure models are evaluated on visual understanding rather than memorized associations.
Annotation: Reference descriptions were generated by four graduate students who narrated the clips in real-time, avoiding proper nouns. These narrations were transcribed via ASR and cleaned by a grammar-only GPT-4o prompt to fix disfluencies without altering content or introducing names.
Diversity: The dataset covers diverse narrative types, including Dialogue/Drama (52%), Violence/Combat (13%), Action/Chase (11.5%), and Public/Social (11.5%), with description lengths ranging from ~100 to over 1,000 words.

Evaluation Framework

The evaluation employs an ensemble of five state-of-the-art LLM-based embedding models (selected from the MTEB leaderboard: GTE-Qwen2-7B, KaLM-Embedding-Gemma3-12B, Llama-Embed-Nemotron-8B, NV-Embed-v2, and Qwen3-Embedding-8B) to mitigate single-model bias. The framework utilizes two complementary methodologies:

Coarse-Grained Semantic Matching: Computes the cosine similarity between the entire generated paragraph and the reference paragraph to assess holistic semantic alignment.
Fine-Grained Semantic Matching: Embeds individual sentences to construct an $n \times m$ cosine matrix. It calculates precision (average best match for each generated sentence) and recall (average best match for each reference sentence), combining them into a sentence-level F1 score.

For each judge, a Harmonic Mean (HM-CF) of the coarse and fine scores is calculated to penalize models that excel in only one granularity. Final rankings are determined via Borda count aggregation across the five judges, assigning points based on rank to ensure robustness against individual judge scale biases.

Key Contributions

Curated Bias-Minimizing Dataset: A dataset of 200 clips with expert-written, proper-noun-free descriptions that isolates fundamental visual understanding from world knowledge retrieval.
Ensemble-Based Evaluation Protocol: A structured multi-granular measurement protocol using five embedding judges. This approach separates narrative-level alignment from detail coverage and uses Borda aggregation to cancel scale biases, providing a transparent and reproducible alternative to black-box LLM judges.
Public Release: The authors release the dataset, evaluation scripts, model outputs, and aggregation tools to support reproducibility.

Results

The framework was used to evaluate 17 state-of-the-art VLMs, including VideoLLaMA3, mPLUG-Owl3, LLaVA-OneVision, and various Qwen and InternVL variants.

Top Performers: VideoLLaMA3 achieved a perfect Borda score of 80/80, securing the top rank across all five embedding judges with a mean HM-CF score of 0.67. mPLUG-Owl3 followed closely (Borda: 75), and LLaVA-OneVision ranked third.
Architecture Insights: Transformer-based models dominated the top tier. Specialized temporal models (e.g., TimeChat, TS-LLaVA) and efficient architectures generally underperformed, suggesting that temporal modeling alone is insufficient for long-form description without strong semantic understanding.
Granularity Gap: A consistent gap was observed between coarse-grained and fine-grained scores across all models. While models generally captured the overall semantic meaning (coarse), they struggled with precise detail matching (fine). For instance, on the KaLM judge, VideoLLaMA3 scored 0.82 on coarse matching but 0.76 on fine-grained matching; this gap was even more pronounced with the NV-Embed judge (0.68 vs. 0.47).
Reliability: The protocol demonstrated high internal reliability, with inter-judge rank correlations ranging from Spearman $\rho=0.96$ to $0.99$. Bootstrap analysis confirmed the stability of the rankings, with the top-ranked model preserved in 100% of resampled replicates.

Significance

CLIP-CC-Bench addresses a critical gap in video-language evaluation by providing a practical framework for assessing long-form, paragraph-level descriptions. The paper demonstrates that current VLMs, while capable of capturing high-level narrative structures, still face significant challenges in generating detailed, factually consistent descriptions of complex visual sequences. The ensemble-based, embedding-driven protocol offers a statistically stable and reproducible alternative to existing n-gram or black-box LLM evaluation methods, establishing a new baseline for measuring progress in long-form video understanding. The results indicate substantial headroom for improvement, with even the top-performing model achieving a mean score of only 0.67, highlighting the complexity of the task.