VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding

This paper introduces VirtueBench, a new benchmark designed to evaluate the trustworthiness of Vision-Language Models in long video understanding by distinguishing between answerable and unanswerable cases to prevent misleading accuracy scores caused by guessing under uncertainty.

The Gist

Imagine you are taking a very difficult history test, but there's a catch: you are only allowed to look at 5 random pages of the textbook instead of reading the whole chapter.

If the answer to the question is on page 100, but you only saw pages 1, 5, 10, 15, and 20, you have no idea what the answer is.

• The Honest Student: Says, "I didn't read those pages, so I don't know the answer."
• The Guesser: Takes a wild guess. Maybe they guess right by luck, maybe they guess wrong.

The Problem:
Current AI tests (benchmarks) are like a strict teacher who punishes the Honest Student for saying "I don't know" and rewards the Guesser if they happen to get the lucky guess right. This tricks us into thinking the AI is smart when it's actually just gambling.

The Solution: VirtueBench
The authors of this paper created a new test called VirtueBench. Think of it as a "Character Test" for AI. Instead of just asking "Did you get the right answer?", it asks: "Did you know when you shouldn't have answered?"

Here is how they built it, using some fun analogies:

1. The "Zoom Lens" Experiment

Imagine you have a long movie.

• Old Way: You show the AI a few blurry snapshots (frames) from the movie and ask a question. If the AI guesses, it gets points.
• VirtueBench Way: They take the same movie and show the AI different "zoom levels."
  • Level 1: 64 snapshots (very blurry, missing the key scene).
  • Level 2: 128 snapshots.
  • Level 3: 1,024 snapshots (almost the whole movie).

They then create a "Answer Key" for every single level.

• If the key scene is missing in the 64-snapshot version, the "correct" answer is "I don't know."
• If the key scene is there in the 1,024-snapshot version, the "correct" answer is the actual fact.

2. The "Honesty vs. Bravery" Score

In this new test, the AI gets points for two things:

1. Accuracy: Getting the answer right when it can see the evidence.
2. Virtue (Refusal): Saying "I don't know" when the evidence is missing.

If an AI guesses the answer when it can't see the evidence, it gets a zero, even if it guesses the right word by accident. This stops the AI from "gambling" its way to a high score.

3. What They Found (The Plot Twist)

The researchers tested 25 different AI models (like Qwen, LLaVA, GPT-4o, etc.) and found some surprising things:

• The "Silent" AI: Many popular models are terrible at admitting they don't know. They are like students who will write something on the test even if they haven't studied, hoping to get lucky. Some of these models had a "refusal rate" of nearly 0% (they never said "I don't know").
• The "Honest" AI: A few models (like the newer Qwen and Gemini versions) were much better. They would look at the blurry snapshots, realize the info was missing, and politely say, "I can't answer this."
• The "Prompt" Trap: When the researchers removed the instruction telling the AI to "be honest," the good models suddenly got worse. It turns out, these AIs are naturally wired to be "people-pleasers" who want to give an answer, even a bad one, unless you explicitly tell them, "It's okay to say you don't know."

Why This Matters

Imagine you are a doctor using an AI to diagnose a patient.

• Old Benchmark: The AI guesses a disease. If it's right, you think the AI is a genius. If it's wrong, you think it's just a fluke.
• VirtueBench: The AI sees the symptoms are unclear. Instead of guessing a disease and risking a wrong treatment, it says, "I need more tests."

The Bottom Line:
This paper argues that we need to stop praising AI for guessing and start praising it for knowing its limits. VirtueBench is the tool we need to build AI that is not just "smart," but also trustworthy. It's about moving from AI that "acts like it knows everything" to AI that "knows when it doesn't know."

Technical Summary

Here is a detailed technical summary of the paper "VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding."

1. Problem Statement

Current Vision-Language Models (VLMs) face a critical evaluation flaw in long video understanding. Due to computational constraints, models typically process only a limited subset of frames (e.g., 64–512 frames) from a video that may contain hundreds of thousands of frames.

• The Core Issue: When key frames containing the necessary evidence are missing from the input, standard benchmarks treat a model's honest refusal to answer as an incorrect response. Conversely, models that "guess" or hallucinate an answer based on prior knowledge (rather than visual evidence) are marked correct if the guess happens to be right.
• Consequence: This creates a misleading evaluation metric that rewards speculation and penalizes honesty, encouraging models to guess rather than acknowledge uncertainty. This gap leads to untrustworthy behaviors in real-world applications where information may be incomplete.

2. Methodology: VirtueBench

To address this, the authors introduce VirtueBench, a benchmark explicitly designed to evaluate model trustworthiness under uncertainty.

A. Data Construction

• Source: Derived from existing long video benchmarks (e.g., MLVU, LVBench, Video-MME, LongVideoBench).
• Frame Sampling Strategy: For each video, multiple temporal clips are generated at different frame densities: 64, 128, 256, 512, and 1024 frames.
• Ground Truth Annotation:
  • Instead of a single ground truth per video, VirtueBench provides specific ground truths for each frame sampling level.
  • If a specific clip lacks the key frames required to answer a question, the ground truth is labeled: "The video does not provide enough information."
  • This allows for a binary distinction between "answerable" and "unanswerable" cases based strictly on the provided input.
• Dataset Statistics:
  • 1,328 high-quality instances covering 901 original videos.
  • Balanced distribution between Perception (59.7%) and Reasoning (40.3%) tasks.
  • Questions are converted from Multiple Choice (MCQ) to Open-Ended (OE) to prevent guessing via option elimination.
  • Unanswerability Distribution: The dataset includes cases that are unanswerable at all frame levels, unanswerable only at low frame levels, and answerable at all levels.

B. Evaluation Protocol

• Input: Models are fed specific frame subsets (64–1024 frames) with a prompt explicitly instructing them: "If the frames do not provide enough information, DO NOT guess or infer any answer that is not visually supported."
• Scoring Logic:
  • Answerable Questions: Evaluated on semantic correctness against the reference answer.
  • Unanswerable Questions: A model is considered correct only if it refuses to answer (e.g., stating insufficient information). Providing a guess is marked as incorrect.
• Metrics:
  • Overall Accuracy: Weighted performance across all questions.
  • Refusal Accuracy: The ability to correctly identify and decline unanswerable questions.
  • Non-Refusal Accuracy: Performance on questions where the model attempts an answer.

3. Key Contributions

1. Identification of Evaluation Bias: The paper highlights that current long-video benchmarks inadvertently reward guessing and penalize honesty due to limited frame inputs, leading to inflated accuracy scores that do not reflect true understanding.
2. VirtueBench Benchmark: The first benchmark designed to measure trustworthiness under uncertainty in long video understanding, distinguishing between "can answer" and "cannot answer" based on input sufficiency.
3. Comprehensive Empirical Analysis: Evaluation of 25 VLMs (both open-source and commercial) reveals significant disparities in refusal behaviors and highlights the lack of inherent honesty in current models.

4. Experimental Results

The evaluation of 25 models (including Qwen-VL, LLaVA-Video, InternVideo, GPT-4o/5, and Gemini-2.5-Flash) yielded several critical findings:

• Overall Performance:

  • Gemini-2.5-Flash achieved the highest overall accuracy.
  • Qwen3-VL-32B showed the strongest performance among open-source models, narrowing the gap with commercial models.
  • Counter-intuitive Trend: Unlike previous benchmarks where accuracy improves with more frames, VirtueBench shows a decline in accuracy as frame count increases. This is because higher frame counts reveal more "unanswerable" instances (where the model previously guessed correctly on sparse data but now correctly identifies the missing info or fails to find the specific evidence).

• Refusal Behavior Analysis:

  • Model Family Differences: There is a stark divide in refusal capabilities.
    • High Refusal: Qwen2.5/3-VL, Mimo-VL, and commercial models (GPT-4o, Gemini) show refusal accuracy ranging from 50% to over 80% (e.g., Qwen2.5-72B reached ~84% on 128 frames).
    • Low Refusal: Models like LLaVA-Video, InternVideo, and VideoChat-Flash exhibit near 0% refusal accuracy, almost always guessing even when information is missing.
  • Scale Effect: Larger models generally demonstrate better refusal capabilities (e.g., Qwen3-32B > Qwen3-8B), likely due to better pattern recognition for data insufficiency.
  • Reasoning & RL: Models with Reinforcement Learning (RL) or "Thinking" capabilities (e.g., Qwen3-Thinking, Mimo-RL) show improved refusal accuracy, suggesting these training methods help models evaluate the completeness of visual evidence.

• Prompt Sensitivity (Ablation Study):

  • Removing the explicit instruction to "not guess" caused a drastic drop in refusal accuracy (often by 20–30 percentage points).
  • This indicates that current VLMs are not inherently honest; they default to guessing unless explicitly constrained by the prompt.

5. Significance and Conclusion

• Paradigm Shift: VirtueBench shifts the focus from "how many questions can a model answer?" to "how reliably can a model know what it doesn't know?"
• Trustworthiness Gap: The results expose a fundamental gap in current VLMs: they are optimized to maximize answer generation rather than to acknowledge uncertainty.
• Future Direction: The paper argues for the development of benchmarks and leaderboards that prioritize reliability and honesty. It suggests that future training procedures must explicitly reward models for correctly identifying unanswerable scenarios to prevent hallucination and build truly trustworthy multimodal systems.

In summary, VirtueBench provides a rigorous framework to detect and penalize the "guessing" behavior prevalent in current VLMs, offering a more realistic assessment of their capabilities in real-world scenarios where information is often incomplete.