💻 computer science

QEVA: A Reference-Free Evaluation Metric for Narrative Video Summarization with Multimodal Question Answering

This paper introduces QEVA, a reference-free evaluation metric for narrative video summarization that assesses coverage, factuality, and chronology via multimodal question answering, alongside the MLVU(VS)-Eval benchmark, demonstrating superior correlation with human judgments compared to existing methods.

Original authors: Woojun Jung, Junyeong Kim

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Woojun Jung, Junyeong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a 20-minute movie about a group of friends getting their car stuck in the snow, and you ask a robot to write a short story summary of what happened.

The Problem: The "Reference" Bottleneck
Until now, checking if that robot's summary was good was like grading a student's essay by comparing it to a "perfect" essay written by a human teacher.

The Old Way: You needed a human to write a perfect summary first. Then, a computer would count how many words the robot and the human shared (like counting matching puzzle pieces).
The Flaw: This is expensive, slow, and often misses the point. If the robot tells the story in a different order or uses different words but gets the meaning right, the old computer might say, "Bad job!" just because the words didn't match perfectly. Plus, hiring humans to write "perfect" summaries for every video is a huge cost.

The Solution: QEVA (The "Pop Quiz" Method)
The authors of this paper, Woojun Jung and Junyeong Kim, invented a new way to grade summaries called QEVA. Instead of comparing the summary to a human-written one, QEVA treats the summary like a substitute teacher.

The core idea is simple: "If your summary is good, I should be able to read it and answer questions about the video without ever seeing the video."

Here is how QEVA works, broken down into three steps using a "Pop Quiz" analogy:

1. The Setup (Generating the Quiz)

QEVA looks at the original video and the robot's summary. It acts like a strict teacher who creates a quiz based only on the video.

Coverage Quiz: "Did the summary mention the main event?" (e.g., Did the car get stuck in snow or mud?)
Factuality Quiz: "Is the detail true?" (e.g., Were there two wolves or three?)
Chronology Quiz: "Did the summary get the order right?" (e.g., Did they push the car before or after they got out of it?)

2. The Test (Taking the Quiz)

Now, QEVA gives this quiz to the robot's summary (not a human). It asks the summary to answer the questions.

If the summary says, "The car got stuck in mud," but the video showed snow, the summary fails the Factuality question.
If the summary says, "They pushed the car, then it got stuck," but the video showed the opposite, it fails the Chronology question.

3. The Grade

The system calculates a score based on how many questions the summary answered correctly.

High Score: The summary captured the story, the facts, and the timeline perfectly.
Low Score: The summary missed key parts, made things up, or got the timeline mixed up.

Why is this a big deal?

The paper introduces a new dataset called MLVU(VS)-Eval (a collection of 800 summaries from 200 videos) to test this idea. They found that QEVA is much better at predicting what a human would think than the old methods.

Old Methods: Like checking if two essays use the same vocabulary.
QEVA: Like checking if the essay actually tells the truth and makes sense.

The Catch (Limitations)

The authors are honest about the downsides:

It's expensive: QEVA uses very powerful AI models to generate the questions and grade the answers, which costs money and computing power (like hiring a super-smart tutor for every single video).
It can make mistakes: Sometimes the AI generating the quiz might get confused or "hallucinate" (make up a question that doesn't fit), though they have a filtering system to catch most of these errors.
Bias: It might slightly prefer summaries that sound like they were written by other AI models.

In a nutshell: QEVA is a new, reference-free tool that grades video summaries by seeing if they can pass a pop quiz about the video. It's faster, cheaper (in terms of human labor), and more accurate than the old ways of just counting matching words.

1. Problem Statement

Video-to-text summarization has advanced significantly with the rise of Video-Large Multimodal Models (Video-LMMs), yet the field lacks robust, scalable evaluation methods. Current evaluation approaches suffer from two main limitations:

Dependency on Reference Summaries: Traditional metrics (e.g., ROUGE, BLEU, METEOR) and recent LLM-based judges rely on human-written reference summaries. Acquiring these references is costly, time-consuming, and often lacks the granularity to capture nuanced semantic details.
Modality Gap: Existing "reference-free" methods often attempt to compare text summaries directly to video embeddings (e.g., CLIPScore) or rely on simple text-text comparisons. These fail to capture the spatio-temporal nature of videos, leading to poor sensitivity regarding chronological fidelity (event order) and factual accuracy.

There is an urgent need for a reference-free evaluation paradigm that can directly assess a generated summary against the source video without human references.

2. Methodology: QEVA

The authors propose QEVA (Question-answering based Evaluation metric for Video summArization), a reference-free framework that evaluates summaries by testing their ability to serve as a substitute for the original video. The core principle is that a high-quality summary should allow a model to correctly answer questions derived from the source video.

A. Three Evaluation Dimensions

QEVA decomposes video summary quality into three distinct, complementary criteria:

Coverage: Does the summary comprehensively capture essential content, key events, and themes?
Factuality: Are the details in the summary accurate and free of hallucinations relative to the video?
Chronology: Does the summary preserve the correct temporal order of events (including handling flashbacks/flash-forwards)?

B. The Multimodal QA Pipeline

QEVA employs a collaborative pipeline using Video-LMMs and LLMs to generate and evaluate Question-Answer (QA) pairs:

Question Generation:
- Coverage: A Video-LMM analyzes the source video to generate high-level, inference-driven questions (avoiding trivial facts) to test if the summary covers the main narrative.
- Factuality: An LLM extracts entities, actions, and attributes from the candidate summary and generates targeted factual questions.
- Chronology: A Video-LMM extracts a sequence of key events from the video. The system then generates questions testing the order of these events (e.g., "Did event A happen before B?").
Question Filtering (Two-Stage):
- Trivial Filtering: Questions are discarded if an alternative model can answer them correctly without any context (ensuring the question requires the specific video/summary content).
- Low-Quality Filtering: Questions are discarded if the model cannot answer them even with the correct context (ensuring the question is unambiguous).
Scoring:
- Coverage & Chronology Scores: Calculated as the proportion of questions correctly answered using the candidate summary as the context.
- Factuality Score: Calculated as the proportion of questions confirmed as accurate using the source video as the context.
- Final Score: The arithmetic mean of the three component scores (0–1 scale).

3. Key Contributions

QEVA Metric: The first fully reference-free evaluation metric specifically designed for narrative video summarization, utilizing multimodal QA to assess Coverage, Factuality, and Chronology.
MLVU(VS)-Eval Benchmark: A new annotated dataset derived from the MLVU dataset, containing:
- 200 videos with human-written reference summaries.
- 800 candidate summaries generated by state-of-the-art Video-LMMs (GPT-4o, QwenVL, InternVL, Video-LLaVA).
- High-quality human annotations for the three QEVA dimensions (Inter-annotator agreement $\alpha = 0.68$ ).
Systematic Framework: A structured approach that moves beyond surface-level text similarity to deep semantic and temporal understanding of video content.

4. Experimental Results

The authors evaluated QEVA against existing metrics (Rule-based, Similarity-based, and LLM-based) on the MLVU(VS)-Eval benchmark.

Correlation with Human Judgment: QEVA achieved the highest correlation with human judgments across all metrics (Kendall's $\tau_b$ $τ_{b}$ , $\tau_c$ $τ_{c}$ , and Spearman's $\rho$ $ρ$ ).
- QEVA: $\tau_b \approx 0.6465$ , $\rho \approx 0.7326$ .
- Best Baseline (Video-LMM Judge): $\tau_b \approx 0.5376$ , $\rho \approx 0.6810$ .
- Traditional Metrics (e.g., ROUGE-L): Significantly lower correlations ( $\tau_b < 0.11$ ).
Robustness: QEVA maintained strong positive correlations across summaries generated by diverse Video-LMMs (Qwen, InternVL, GPT-4o), whereas some baselines showed negative or unstable correlations for specific models.
Ablation Studies:
- QEVA performed well across all three individual dimensions (Coverage, Factuality, Chronology), confirming the necessity of evaluating them separately.
- The metric remained effective even when using open-source models (e.g., Qwen2.5-VL, LLaMA-3) instead of expensive API-based models, demonstrating cost-effectiveness.

5. Significance and Impact

Scalability: By eliminating the need for human-written reference summaries, QEVA enables the evaluation of video summarization systems at a scale previously impossible due to annotation costs.
Semantic Depth: Unlike n-gram metrics, QEVA explicitly measures chronological fidelity and factual accuracy, which are critical for narrative videos but often ignored by current benchmarks.
Future Research: The introduction of MLVU(VS)-Eval provides a standardized, transparent framework for the community to benchmark future Video-LMMs, accelerating the development of reliable automated content generation for news, education, and media platforms.

Limitations Noted by Authors

Hallucination: The QA generation process relies on LLMs, which may occasionally hallucinate questions or answers not present in the video.
Cost & Speed: The reliance on large models for inference can be computationally expensive and slow for real-time applications.
Bias: There is a potential for subtle bias toward summaries generated by models similar to the evaluator LLM.