INFACT: A Diagnostic Benchmark for Induced Faithfulness and Factuality Hallucinations in Video-LLMs

The paper introduces \textsc{INFACT}, a comprehensive diagnostic benchmark with 9,800 QA instances and fine-grained taxonomies that evaluates Video-LLMs on faithfulness and factuality under various induced degradation modes, revealing that high base accuracy does not guarantee robustness against hallucinations and that many models struggle significantly with temporal sensitivity.

The Gist

Imagine you have a very smart, very confident robot friend who loves watching videos. You ask it, "What happened in this video?" and it answers with a story. Sometimes, the story is perfect. But often, the robot makes things up. It might say a character was wearing a red hat when they were actually wearing a blue one (it's lying about what it saw), or it might claim that a car can fly because it "knows" cars are cool, even though the video clearly shows the car crashing into a wall (it's lying about how the world works).

This paper introduces a new tool called INFACT to test exactly how good (or bad) these video-watching robots are at telling the truth.

Here is the breakdown of the paper using some everyday analogies:

1. The Two Types of "Lying"

The researchers realized robots make two different kinds of mistakes, and they need to be tested differently:

• Faithfulness (The "Did you watch the video?" Test):

  • The Analogy: Imagine you show a robot a video of a cat chasing a mouse. If the robot says, "The cat was sleeping," it failed the Faithfulness test. It didn't pay attention to the actual evidence in front of its eyes.
  • The Goal: Does the robot stick to the facts shown in the video?

• Factuality (The "Do you know how the world works?" Test):

  • The Analogy: Imagine the video shows a person trying to walk on water. If the robot says, "Yes, that's a normal way to travel," it failed the Factuality test. Even if the video looks like someone walking on water (maybe it's a special effect), the robot should know that humans can't walk on water based on real-world physics.
  • The Goal: Does the robot know the rules of reality, even when the video tries to trick it?

2. The "Stress Test" Gym

Most previous tests were like a calm walk in the park. The robots watched a clear, high-quality video and answered questions. But in the real world, videos are messy! They get blurry, have bad subtitles, or get edited.

INFACT is like a stress-test gym for these robots. It puts them through four specific drills:

• Drill 1: The Clean Room (Base Mode): The robot watches a perfect video. This is just to see how smart it is normally.
• Drill 2: The Foggy Window (Visual Degradation): The researchers add static, blur, or make the video look like it was recorded on a cheap phone.
  • The Test: Can the robot still see the cat chasing the mouse, or does it get confused by the "fog" and start guessing?
• Drill 3: The Gaslighting (Evidence Corruption): This is the tricky one. The researchers add fake subtitles or misleading text on top of the video.
  • The Test: The video shows a door opening, but the fake subtitle says "The door is closing." Does the robot trust its eyes, or does it get tricked by the text? (Spoiler: Many robots get tricked easily).
• Drill 4: The Time-Traveler (Temporal Intervention): The researchers take a video of someone making a sandwich (bread, then peanut butter, then jelly) and shuffle the frames so it looks like they put the jelly on first, then the peanut butter, then the bread.
  • The Test: Does the robot realize the order is wrong? Or does it just say, "Yep, that looks like a sandwich," ignoring the chaos?

3. The Results: "Smart" Doesn't Mean "Reliable"

The researchers tested 14 different video robots (including big names like GPT-5 and Gemini). Here is what they found:

• The "Overconfident" Problem: Many robots are great at the "Clean Room" test. They get high scores. But as soon as you add fog or fake text, their performance crashes. It's like a student who memorizes the answers to a practice test but fails the real exam when the questions are slightly different.
• The "Time-Blind" Problem: When the researchers shuffled the video frames, many robots didn't even notice! They kept giving the same answer as if nothing changed. This is called Temporal Inertia. It's like watching a movie played backward and saying, "That's a perfectly normal story," because the robot isn't actually tracking the sequence of events, just the general "vibe."
• The "Fact" Gap: The robots were much worse at the "Factuality" test (knowing real-world rules) than the "Faithfulness" test (watching the video). They often hallucinate wild physics or wrong historical facts with 100% confidence.

4. Why This Matters

Think of these Video-LLMs as the future of medical diagnosis, security monitoring, or educational tools.

• If a medical robot watches a surgery video and hallucinates that a tool was used when it wasn't, that's dangerous.
• If a security robot watches a video of a person walking on water (a special effect) and reports it as a real event, that's a false alarm.

INFACT is the new "lie detector" that doesn't just ask, "Are you smart?" but rather, "Are you reliable when things get messy?"

The Bottom Line

The paper concludes that just because a robot gets an "A" on a clean test doesn't mean it can be trusted in the real world. We need to build robots that don't just memorize patterns but actually understand what they are seeing and know how the world works, even when the video is blurry, confusing, or playing tricks on them.

Technical Summary

1. Problem Statement

Despite rapid advancements in Video Large Language Models (Video-LLMs), they remain unreliable due to hallucinations. These hallucinations manifest in two distinct ways:

• Faithfulness Hallucinations: Outputs that contradict the visual evidence provided in the video (e.g., misidentifying an object or action).
• Factuality Hallucinations: Outputs that contradict verifiable world knowledge (e.g., incorrect historical facts, procedural errors, or violations of physical laws).

Limitations of Existing Benchmarks:
Current benchmarks primarily focus on faithfulness in "clean" settings (unperturbed videos). They often fail to:

1. Adequately evaluate factuality hallucinations.
2. Test model reliability under induced perturbations (e.g., visual degradation, misleading text, or temporal disruption).
3. Distinguish between models that rely on genuine video understanding versus those exploiting language priors or static cues (shortcuts).

2. Methodology: The INFACT Benchmark

The authors introduce INFACT, a diagnostic benchmark comprising 9,800 QA instances spanning both real and synthetic videos.

A. Fine-Grained Taxonomy

INFACT categorizes hallucinations based on the evidentiary basis required for verification:

1. Faithfulness (Video-Grounded):
   • Static Entities & Attributes: Object recognition, counting, attribute recognition, scene text.
   • Dynamic Actions & Motions: Action recognition, repetitive counting, motion attributes.
   • Spatio-Temporal Relations: Spatial relations, temporal relations, state transitions, temporal localization.
2. Factuality (World-Knowledge-Grounded):
   • Domain Knowledge (Know-What): Cultural events, history, geography, entertainment.
   • Procedural Knowledge (Know-How): Validity of step sequences in electronics, mechanics, domestic tasks, and clinical procedures.
   • Physical Knowledge (Know-Why): Adherence to Newtonian mechanics, fluid dynamics, material properties, and spatio-temporal continuity.

B. Four Evaluation Modes

To probe reliability beyond clean settings, INFACT employs four modes:

1. Mode I: Base (Clean): Standard evaluation on unaltered videos to establish baseline accuracy.
2. Mode II: Visual Degradation: Applies label-preserving perturbations (Gaussian noise, motion blur, video compression) to test robustness against perceptual uncertainty.
3. Mode III: Evidence Corruption: Introduces label-preserving but misleading auxiliary cues (caption injection with conflicting text, subtitle corruption, adversarial noise) to test if models prioritize visual evidence over misleading text.
4. Mode IV: Temporal Intervention: Applies label-changing interventions (frame shuffling, reversal) to order-sensitive items to test if models rely on temporal logic or static priors.

C. Evaluation Metrics

• Resist Rate (RR): Measures stability in Modes II and III. It calculates the percentage of items where the model maintains its correct Base prediction despite the perturbation.
• Temporal Sensitivity Score (TSS): Measures sensitivity in Mode IV. It calculates the rate at which a model changes its prediction after temporal disruption. A high TSS indicates the model correctly recognizes that the temporal order is broken; a low TSS (temporal inertia) indicates the model ignores the disruption and relies on static priors.

3. Key Contributions

1. Comprehensive Benchmark: INFACT is the first benchmark to jointly evaluate faithfulness and factuality with fine-grained taxonomies across real and synthetic videos.
2. Induced Reliability Protocol: Introduces a four-mode evaluation framework (Base, Degradation, Corruption, Intervention) with paired metrics (RR and TSS) to quantify model stability and temporal grounding.
3. Systematic Diagnosis: Evaluates 14 representative Video-LLMs (including proprietary and open-source models), revealing that high Base accuracy does not guarantee reliability under induced conditions.

4. Experimental Results & Analysis

The authors evaluated 14 models (e.g., GPT-5.1, Gemini-3-flash, Qwen3-VL, InternVL, VideoLLaMA2) on INFACT.

• Base Accuracy vs. Reliability: While there is a strong correlation between Base accuracy and induced-mode reliability, rankings diverge. High accuracy in clean settings does not ensure stability under perturbations.
• Stability (RR):
  • Evidence Corruption degrades performance more significantly than Visual Degradation.
  • Caption Injection (misleading text) is particularly damaging, suggesting models often over-rely on textual cues rather than visual evidence.
  • Models like Gemini-3-flash and GPT-5.1 show the highest resistance rates.
• Temporal Sensitivity (TSS):
  • Many open-source baselines exhibit near-zero TSS on factuality questions, indicating "temporal inertia"—they fail to detect when the chronological order is shuffled or reversed.
  • Gemini-3-flash achieved the highest TSS (0.719), while several open-source models scored 0.000, highlighting a gap in temporal grounding.
• Fine-Grained Performance:
  • Factuality remains a significant challenge, particularly in Procedural Judgement and Physical Reasoning.
  • Faithfulness errors are concentrated in motion-centric and structure-centric dimensions (e.g., temporal localization).
• Frame Sampling: Increasing the number of sampled frames improves Base accuracy but yields diminishing returns for reliability metrics (RR and TSS), suggesting that architectural improvements are needed rather than just more data.

5. Significance

• Beyond Clean Benchmarks: INFACT shifts the evaluation paradigm from "can the model answer?" to "is the model's answer robust and grounded?"
• Diagnostic Power: The metrics (RR and TSS) provide actionable insights into specific failure modes, such as over-reliance on text priors or inability to process temporal sequences.
• Guidance for Future Models: The results suggest that current Video-LLMs need better mechanisms for:
  1. Prioritizing visual evidence over conflicting text.
  2. Explicitly modeling temporal dynamics to avoid temporal inertia.
  3. Integrating world knowledge without hallucinating procedural or physical details.

In conclusion, INFACT provides a rigorous, multi-dimensional framework for diagnosing the reliability of Video-LLMs, revealing that current state-of-the-art models still struggle with induced hallucinations and temporal grounding, despite high performance on standard clean benchmarks.