UniVBench: Towards Unified Evaluation for Video Foundation Models

The paper introduces UniVBench, a comprehensive benchmark featuring 200 high-quality, human-created multi-shot videos and a unified agentic evaluation system (UniV-Eval) to holistically assess video foundation models across understanding, generation, editing, and reconstruction tasks, addressing the limitations of existing fragmented and task-specific evaluations.

The Gist

Imagine you are a film director trying to hire a new assistant. You have three very different jobs for them:

1. The Critic: They need to watch a movie and write a perfect review describing every detail.
2. The Creator: They need to take a written description and magically conjure a movie out of thin air.
3. The Editor: They need to take an existing movie and change specific things (like the weather or the actor's clothes) without breaking the scene.

Until now, we've been hiring assistants for just one of these jobs at a time. We have "Critic" tests, "Creator" tests, and "Editor" tests. But the new generation of AI models wants to be a "Super Assistant" that can do all three at once. The problem? We didn't have a single test to see if this Super Assistant was actually good at everything, or if it was just a "one-trick pony."

Enter UniVBench.

Think of UniVBench as the ultimate "All-Star Sports Day" for video AI. Instead of testing the athlete's running, swimming, and jumping separately, this benchmark throws them into a complex, multi-stage obstacle course that requires all those skills simultaneously.

Here is how it works, broken down into simple concepts:

1. The Obstacle Course (The Dataset)

Most previous tests used short, simple clips or movies ripped from the internet (which can be legally messy). UniVBench is different.

• The Content: The team created 200 brand-new, high-quality videos from scratch. These aren't just 5-second loops; they are like short movie scenes with multiple "shots" (camera angles) and a real story.
• The Instructions: For every video, they wrote a detailed "script" (what the video looks like), a "director's note" (how to edit it), and even "reference photos" (what the characters should look like).
• The New Challenge: They added a new task called Video Reconstruction. Imagine showing the AI a video, asking it to describe it in words, and then asking it to re-make the video based only on those words. If the AI forgets a detail in the description, the new video will be wrong. This tests if the AI truly "understands" what it sees.

2. The Referee (UniV-Eval)

In the past, grading these videos was messy.

• The Old Way: It was like a teacher giving a student a single grade of "75/100" for an essay. You know they passed, but you don't know why they failed. Did they have bad grammar? Was the story boring? Was the spelling wrong?
• The UniVBench Way: They built a Smart Referee System (called UniV-Eval). Instead of one number, this referee breaks the performance down into 21 specific categories, like:
  • Did the lighting look right?
  • Did the camera move smoothly?
  • Did the characters keep their clothes on correctly?
  • Was the background consistent?

This system acts like a film critic with a magnifying glass. It doesn't just say "Good job." It says, "The lighting was perfect, but the character's hand disappeared for a second, and the background changed color when it shouldn't have."

3. The Results (What We Learned)

When they ran this "All-Star" test on current AI models, the results were eye-opening:

• Specialists vs. Generalists: Some AIs are amazing at making videos but terrible at understanding them (like a painter who can't explain art). Others are great at watching videos but can't create them.
• The "Action" Problem: Almost every model struggled with movement. If you ask an AI to make a video of a dog running, it often makes the dog slide or freeze. It's like a puppeteer who can draw a perfect dog but can't make the puppet's legs move naturally.
• The Gap: No single model currently exists that is perfect at understanding, creating, and editing all at once. UniVBench proves that while we are getting closer, the "Super Assistant" isn't quite ready for prime time yet.

Why Does This Matter?

Think of video AI as a new language. Before, we were teaching students to read, write, and speak separately. UniVBench is the first fluency test that checks if they can hold a real conversation, tell a story, and react to questions all at the same time.

By providing a fair, detailed, and comprehensive way to measure these models, UniVBench gives developers a clear map of where they are failing. It's the difference between saying, "Your car is fast," and saying, "Your car is fast, but the brakes squeak, the AC is broken, and the GPS is wrong."

In short: UniVBench is the first "Report Card" that actually tells us if our video AI is a true genius or just a one-trick pony, helping us build the next generation of machines that can truly see, understand, and create the world around us.

Technical Summary

1. Problem Statement

Video foundation models (VFMs) aim to unify video understanding, generation, editing, and instruction following within a single architecture. However, current evaluation benchmarks suffer from three critical limitations that prevent the assessment of these unified capabilities:

• Fragmented Scope: Existing benchmarks are task-specific (e.g., separate datasets for captioning, text-to-video, or editing) and cannot evaluate a model's ability to perform multiple tasks simultaneously or transition between them.
• Limited Complexity: Most benchmarks rely on short, single-shot clips or simple web-scraped videos, failing to capture the complexity of multi-shot narratives, cinematic coherence, and long-form temporal reasoning.
• Inconsistent Metrics: Evaluation metrics are disjointed. Understanding tasks use n-gram overlap (BLEU/CIDEr), generation tasks use distributional metrics (FVD), and editing tasks rely on ad-hoc combinations. There is no unified framework to compare performance across different modalities (text, image, video) or to diagnose whether failures stem from perception (understanding) or generation components.
• Data Contamination: Many existing datasets use copyrighted web videos, posing legal risks and data contamination issues for models trained on internet-scale data.

2. Methodology

The authors propose UniVBench, a unified benchmark, and UniV-Eval, an agentic evaluation system.

A. UniVBench Dataset Construction

• Scale & Composition: The dataset contains 200 high-quality, human-created videos (100 single-shot, 100 multi-shot) with an average of 3.72 shots per video.
• Cinematic Dimensions: Videos are annotated across 8 fundamental dimensions and 21 fine-grained sub-dimensions, including:
  • Style, Subject, Action, Background.
  • Camera: Shot size, motion, perspective, angle, height, focus, techniques.
  • Lighting & Color: Direction, brightness, effect, hue, saturation, contrast.
  • Spatial Relationships: Inter-frame layout, subject-to-subject, and subject-to-camera relations.
• Annotations: Each video includes detailed ground-truth captions, multi-format editing instructions, and reference images. All content is copyright-free to ensure fair evaluation.
• New Task (Video Reconstruction): The benchmark introduces Video2Video (V2V) Reconstruction, where a model must first understand a video to generate a detailed caption and then reconstruct the video solely from that caption. This directly tests the coupling between perception and generation.

B. UniV-Eval System

Instead of relying on fixed scalar scores, UniV-Eval is an agentic evaluation system that standardizes prompting and scoring across six unified tasks:

1. V2T: Video Captioning
2. T2V: Text-to-Video Generation
3. R2V: Reference-Image-to-Video Generation
4. TV2V: Text-Instruction Video Editing
5. RV2V: Reference-Image Video Editing
6. V2V: Video Reconstruction

Workflow:

1. Decomposition & Planning: Long videos are segmented into shots. The agent aligns inputs (source video, reference images, instructions) with specific shots.
2. Shot-Level Evaluation: A "Shot Evaluation Agent" compares the model output against the input tuple using a Weakness Checklist across the 21 sub-dimensions.
3. Scoring: A "Score Agent" aggregates these fine-grained diagnostics into quantitative scores for the 8 dimensions, providing traceable feedback (e.g., "Action Consistency: Low") rather than a single number.

3. Key Contributions

1. First Unified Multi-Task Benchmark: UniVBench is the first dataset designed to evaluate the full spectrum of video foundation model capabilities (understanding, generation, editing, reconstruction) on multi-shot, copyright-free content.
2. Unified Agentic Evaluation System (UniV-Eval): A novel framework that replaces fragmented metrics with a standardized, multi-dimensional, shot-level scoring system. It enables direct cross-task comparison and fine-grained error attribution (perception vs. generation).
3. Principled Framework for Capability Attribution: By introducing the V2V reconstruction task and structured checklists, the framework allows researchers to pinpoint exactly where a unified model fails (e.g., does it fail because it misunderstood the scene or because it couldn't render the motion correctly?).

4. Experimental Results

The authors evaluated various commercial (e.g., Gemini 2.5 Pro, Seedance) and open-source models (e.g., CogVideoX, Wan2.1, Omni-Video) on UniVBench.

• Task Specialization vs. Unification: Results reveal a significant gap between specialized and unified models.
  • Understanding: Specialized models like Gemini 2.5 Pro (54.1% avg) vastly outperform unified models like Showo-2 (16.3%) in video captioning.
  • Generation: Specialized generators like Seedance-1.0-Pro (77.9% avg) lead in text-to-video tasks.
  • Unified Performance: No single model currently excels across all tasks. Unified models often struggle with the Action dimension (temporal dynamics) but perform better on static attributes like Color and Lighting.
• Reconstruction Gap: The V2V reconstruction task highlights the "perception-generation coupling" problem. Models that generate good captions often fail to reconstruct the video faithfully, indicating a disconnect between their understanding and generation modules.
• Human Alignment: UniV-Eval was validated against human experts, achieving ~85% agreement, demonstrating that the agent-based scoring faithfully reflects human judgment.

5. Significance

• Benchmarking Standard: UniVBench establishes a necessary baseline for the next generation of video foundation models, moving beyond single-task metrics to holistic evaluation.
• Diagnostic Tool: The fine-grained, shot-level feedback provided by UniV-Eval offers actionable insights for model training, helping developers identify specific weaknesses (e.g., poor camera motion control or subject identity drift) rather than just observing a low aggregate score.
• Future Direction: The benchmark highlights that while current models are improving in static attributes, temporal reasoning and multi-shot narrative coherence remain the primary bottlenecks for achieving truly general-purpose, instruction-following video intelligence.

In summary, UniVBench provides the first comprehensive, legally safe, and methodologically rigorous framework to measure progress toward unified video intelligence, shifting the focus from isolated task performance to integrated multimodal capability.