💻 computer science

DualFact+: A Multimodal Fact Verification Framework for Procedural Video Understanding

DualFact+ introduces a novel dual-layer, multimodal framework that separates procedural video facts into conceptual and contextual components to provide a more interpretable and human-aligned evaluation of video captioning, revealing that state-of-the-art models often suffer from systematic factual omissions and inconsistencies that standard metrics fail to detect.

Original authors: Cennet Oguz, Yasser Hamidullah, Josef van Genabith, Simon Ostermann

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cennet Oguz, Yasser Hamidullah, Josef van Genabith, Simon Ostermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a cooking show or a DIY furniture tutorial. A smart computer is trying to write a recipe or a set of instructions based on what it sees in the video. Sometimes, the computer writes a sentence that sounds perfect and flows beautifully, but it's actually lying about what happened. Maybe it says, "The chef chopped the onions," when the video clearly shows them chopping carrots. Or maybe it forgets to mention that the chef used a knife instead of a spoon.

This paper introduces a new way to check if these computer-generated instructions are actually true. They call their system DualFact.

Here is how it works, broken down into simple ideas:

1. The Two Layers of Truth

The authors realized that checking a video description isn't just about reading the words; it's about understanding two different layers of reality:

The "Conceptual" Layer (The Plan): This is the abstract idea of what should happen. For example, "Cut the vegetable." It doesn't care which vegetable or what tool is used yet; it just knows an action is happening. Think of this like a rough sketch of a painting.
The "Contextual" Layer (The Reality): This is the specific, grounded detail of what actually happened in the video. Did they cut a tomato or an onion? Did they use a knife or a cleaver? This is the finished, detailed painting.

The Problem: Most current computer programs are great at the "Conceptual" layer (they sound smart) but often fail at the "Contextual" layer (they get the specific details wrong). They might say "cut the vegetable" (true) but miss that they used a dull knife, or they might invent a tool that wasn't there at all.

2. The "DualFact" Detective

To fix this, the researchers built a detective system called DualFact. It acts like a strict editor who checks the computer's work in two ways:

The Text Check: It compares the computer's sentence against the "gold standard" text (the perfect human-written instructions).
The Video Check: It compares the computer's sentence against the actual video footage.

The system breaks every sentence down into tiny "facts" (like: Action = Cut, Object = Tomato, Tool = Knife). Then, it asks: "Is this fact supported by the video?"

3. Catching the Three Types of Mistakes

The paper explains that computers make three specific types of lies, and DualFact is designed to catch them:

Hallucinations (The "Magic" Mistake): The computer invents something that isn't there.
- Example: The video shows a bowl, but the computer says, "The chef used a blender." The blender never existed in the video.
Omissions (The "Missing" Mistake): The computer forgets to mention something important that was there.
- Example: The video shows the chef adding salt, but the computer just says, "The chef mixed the soup," forgetting the salt entirely.
Salience Errors (The "Distraction" Mistake): This is the trickiest one. The computer mentions something that is in the video, but it's not important to the task.
- Example: The chef is cutting a tomato, but there is a lemon sitting on the counter in the background. The computer says, "The chef cut the lemon." The lemon was there (so it's not a hallucination), but it wasn't the main event. The computer got distracted by the background noise.

4. What They Found

The researchers tested this system on two types of videos: cooking (YouCook3) and furniture making (CraftBench). They found some surprising things:

Fluency $\neq$ Truth: The best-sounding, most fluent sentences were often the ones with the most factual errors. The computers were "smooth talkers" but bad at details.
Old Metrics Lie: Standard ways of measuring success (like counting how many words match) were terrible at catching these errors. They gave high scores to sentences that were factually wrong.
Video is the Truth-Teller: When the system checked against the video instead of just the text, it found that many "lies" were actually just "distractions" (Salience errors) or "omissions." The video grounded the truth in a way text alone couldn't.

5. The Takeaway

The paper concludes that to truly understand procedural videos (like cooking or building), we need a system that doesn't just check if the words sound nice, but verifies if the specific roles (Action, Tool, Object) match the visual evidence.

DualFact is like a rigorous fact-checker that separates the "big idea" from the "specific details," ensuring that when a computer describes a video, it tells the whole truth, not just a pretty version of it.

1. Problem Statement

Procedural video captioning (e.g., cooking, furniture assembly) requires models to generate descriptions that are not only fluent but also factually correct and visually grounded. Existing evaluation metrics fail to address the specific challenges of this domain:

Lexical Metrics (BLEU, ROUGE): Rely on surface-level overlap and cannot detect semantic role errors (e.g., swapping an ingredient for a tool) or missing arguments.
Vision-Language Metrics (CLIPScore, EMScore): Measure global semantic similarity but lack structured reasoning over predicate-argument relations. They often miss fine-grained inconsistencies.
Current Fact-Based Metrics: Typically represent facts as flat, untyped statements. They struggle to distinguish between conceptual errors (wrong abstract role, e.g., "add salt" when it should be "sugar") and contextual errors (wrong visual realization, e.g., "add salt to the pan" when the video shows a bowl).
Implicit Arguments: Procedural captions often omit visually present arguments (e.g., "stir it" without specifying "soup" or "spoon"), leading to ambiguity that standard metrics ignore.

2. Methodology: The DualFact/MultiFactScore Framework

The authors propose a dual-layer fact verification framework that separates Conceptual Facts from Contextual Facts to enable fine-grained evaluation.

A. Dual-Layer Fact Representation

Conceptual Facts ( $F_{con}$ ): Capture abstract task semantics independent of surface realization. They identify the intended action and core participant roles (e.g., ACTION=STIR, INGREDIENT=SOP, TOOL=SPOON).
Contextual Facts ( $F_{ctx}$ ): Capture the grounded predicate-argument relations visible in the video (e.g., STIR(soup, bowl, spoon)).
- Significance: This separation allows the system to detect if a model understands the task (conceptual) but fails to describe the visual execution (contextual), or vice versa.

B. Dataset Construction (YouCook3-Fact & CraftBench-Fact)

The authors created two new benchmarks with three key preprocessing steps:

Clause Decomposition: Splitting multi-action sentences into atomic clauses aligned with specific video segments.
Implicit Argument Augmentation (VIA): Annotating missing but visually present arguments (e.g., expanding "stir it" to "stir the soup with a spoon in the pot") to ensure role consistency.
Structured Fact Annotation: Extracting both conceptual and contextual facts, along with negative fact generation (creating plausible but false variants for contrastive learning).

C. Verification Pipeline

The framework uses Natural Language Inference (NLI) models to verify facts in two modes:

DualFact-T (Textual): Verifies generated facts against the gold caption.
DualFact-V (Multimodal): Verifies generated facts directly against the video clip.
Scoring: The MultiFactScore is calculated as the proportion of facts judged as "SUPPORTED" by the verifier.
Error Decomposition: The system categorizes errors into:
- Hallucination: Content not supported by the video.
- Salience Error: Content is visually present but task-irrelevant (e.g., mentioning a lemon when cutting a cucumber).
- Omission: Missing task-critical entities or roles.

3. Key Contributions

Dual-Layer Fact Representation: A novel framework separating abstract role semantics from grounded visual realizations, enabling the detection of role-level inconsistencies and omissions.
New Benchmarks: Introduction of YouCook3-Fact (cooking) and CraftBench-Fact (furniture crafting), featuring clause-level segmentation, VIA, and structured fact annotations.
Empirical Insights: Demonstration that state-of-the-art Multimodal LLMs (e.g., Qwen2.5-VL) produce fluent captions that are systematically factually incomplete, with high rates of omissions and role-level errors that standard metrics miss.
Evaluation Protocol: A method showing that caption-only evaluation significantly overestimates hallucinations compared to video-grounded verification, which reveals "salience errors" (visually present but irrelevant details).

4. Experimental Results

Experiments were conducted on YouCook3-Fact and CraftBench-Fact using Qwen2.5-VL and standard metrics.

Fact Extraction: The LLM-based extractor achieved high accuracy (F1 > 96% for conceptual roles, >95% for contextual relations), ensuring evaluation noise is minimal.
Verifier Performance:
- The verifier performs near-perfectly on gold facts (Textual: ~99%, Multimodal: ~88-93%).
- Model Consistency Bias: When verifying predicted captions, accuracy drops significantly (e.g., Textual verification of predicted captions dropped to ~20-40%), indicating models fail to encode ground-truth entities.
Error Analysis:
- Caption-Only vs. Video-Grounded: Caption-only evaluation labeled ~50-60% of errors as "Hallucinations." Video-grounded verification reclassified many of these as Salience Errors (e.g., 36% of object errors were salience, not hallucination).
- Omissions: The primary failure mode was omission of task-relevant entities (e.g., missing tools or ingredients), which standard metrics failed to penalize heavily.
Correlation with Human Judgment:
- The proposed Caption-based Conceptual Fact Score showed the strongest correlation with human judgments (Spearman $\rho$ = 0.429), outperforming BLEU, ROUGE, and EMScore.
- Video-based scores showed lower correlation due to "ceiling effects" (high absolute correctness but low variance).

5. Significance and Conclusion

Beyond Fluency: The paper argues that fluency does not equate to factual correctness in procedural tasks. A caption can be grammatically perfect but miss critical steps or misassign roles.
Interpretability: The framework provides a diagnostic tool to distinguish between what the model missed (omission), what it invented (hallucination), and what it got right but irrelevant (salience).
Future Direction: The work highlights the need for future models to focus on role-consistent and visually grounded generation, moving beyond surface-level semantic similarity. It establishes a foundation for more robust instructional video understanding systems.

Limitations: The current scope is limited to cooking and crafting; fact extraction relies on the accuracy of the underlying LLM; and the framework does not yet model attribute-oriented facts (e.g., size, color) or distinguish between severity levels of hallucinations.