💬 NLP

Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity

This paper challenges the assumption that visual evidence always improves multimodal fact-checking by proposing AMuFC, an adaptive framework that uses a collaborative Analyzer to determine visual necessity and a Verifier to assess claim truthfulness, thereby achieving superior performance on standard and a newly released realistic dataset.

Original authors: Jaeyoon Jung, Yejun Yoon, Kunwoo Park

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaeyoon Jung, Yejun Yoon, Kunwoo Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Is this news story true or fake?

For a long time, detectives (and computer programs) believed that to solve the mystery, you needed everything: the written story and the accompanying photo. They thought, "A picture is worth a thousand words, so more pictures mean a better solution!"

But this paper, titled "Is a Picture Worth a Thousand Words?", argues that sometimes, that old saying is actually a trap.

Here is the simple breakdown of what the researchers discovered and how they fixed it.

1. The Problem: The "Cluttered Desk" Effect

The researchers found that blindly adding a photo to a fact-checking task often makes the computer dumber, not smarter.

The Analogy: Imagine you are trying to read a recipe to see if it's correct. If someone hands you the recipe and a random photo of a cat, the photo doesn't help you. In fact, it might distract you.
The Reality: Sometimes, a news claim is about a number or a quote. A photo of a person's face or a generic landscape adds no value. Worse, if the computer tries to "read" that irrelevant photo, it might get confused by the noise and make a mistake.

The team tested this by forcing computers to look at photos even when they didn't need them. The result? The computers got less accurate.

2. The Solution: The "Two-Person Detective Team"

Instead of one robot trying to do everything at once, the researchers built a system called AMUFC (Adaptive Multimodal Fact-Checking). Think of it as a detective agency with two specialists working together:

🕵️‍♂️ Agent 1: The Analyzer (The Gatekeeper)

Before the main detective looks at the evidence, the Analyzer steps in.

Role: This agent looks at the claim and the photo and asks: "Do we actually need this picture to solve this case?"
Action: If the picture is just a distraction (like the cat photo), the Analyzer says, "Ignore this, it's useless." If the picture shows something crucial (like a document or a specific event), it says, "This is vital! Look at this!"
The Metaphor: The Analyzer is like a bouncer at a club. It checks the ID of the photo. If the photo doesn't belong, it gets turned away before it can cause trouble inside.

🧠 Agent 2: The Verifier (The Judge)

This is the main detective who decides if the claim is True, False, or "Not Enough Info."

Role: The Verifier takes the written story, the photo (if the Gatekeeper let it in), and the Gatekeeper's opinion.
Action: The Verifier thinks, "The Gatekeeper said this photo is important, so I will focus on it," or "The Gatekeeper said the photo is junk, so I will ignore it and focus on the text."
The Metaphor: The Verifier is the Judge who listens to the Gatekeeper's advice before making a final ruling.

3. The Result: Smarter Decisions

When they tested this new "Two-Person Team" against old methods:

Old Way: The computer looked at everything (text + photo) and often got confused.
New Way (AMUFC): The computer learned to be selective. It only used photos when they actually helped.

The Outcome: The new system was significantly more accurate. It proved that sometimes, less is more. By filtering out the "visual noise," the computer could focus on the real facts.

4. A New Tool for the Future

The researchers also built a new dataset called WebFC.

The Analogy: Most old tests used "textbook" examples where the answers were already known. This new dataset is like a live newsroom. It uses real, recent news stories and forces the computer to go out and find the evidence on the internet, just like a real journalist would.
Why it matters: It ensures the system works in the real world, not just in a controlled lab.

Summary

This paper teaches us that in the world of AI and fact-checking, blindly trusting a picture can be a mistake.

By teaching AI to act like a smart editor—asking "Do I really need this image?" before using it—we can build systems that are less distracted and much better at spotting the truth. It's not about having more information; it's about having the right information.

1. Problem Statement

Automated fact-checking has evolved from text-only verification to multimodal fact-checking, which integrates text and images. The prevailing assumption in current research is that incorporating visual evidence universally improves verification accuracy (i.e., "a picture is worth a thousand words").

However, the authors challenge this assumption, positing that:

Indiscriminate use of visual evidence can degrade performance. Irrelevant, redundant, or noisy images can confuse Vision-Language Models (VLMs), leading to lower accuracy compared to text-only baselines.
Visual necessity varies by claim. Some claims are verifiable solely through text, while others strictly require visual evidence.
Current limitations: Existing Retrieval-Augmented Generation (RAG) frameworks for fact-checking typically retrieve and fuse both modalities without assessing whether the visual modality is actually necessary for a specific claim.

2. Methodology: AMUFC Framework

The authors propose AMUFC (Adaptive Multimodal Fact-Checking with Visual Evidence Necessity), a framework designed to adaptively utilize visual evidence through the collaboration of two distinct VLM agents.

Core Components

Retriever: Retrieves textual ( $E_t$ ) and visual ( $E_v$ ) evidence from a knowledge source $K$ .
Analyzer Agent:
- Role: Determines whether the retrieved visual evidence is necessary for verifying the specific claim.
- Input: The claim, retrieved text evidence, and retrieved image evidence.
- Output: A natural-language assessment explaining why the image is necessary (provides unique info) or unnecessary (redundant/irrelevant).
- Training: Fine-tuned using natural language reasoning labels distilled from GPT-4o.
Verifier Agent:
- Role: Predicts the final verdict (Supported, Refuted, or Not Enough Information).
- Input: The claim, all retrieved evidence, and the natural-language assessment generated by the Analyzer.
- Mechanism: Instead of blindly fusing image and text, the Verifier uses the Analyzer's assessment to reason about the relevance of the visual evidence during its internal inference process.

Workflow

Retrieve evidence ( $E_t, E_v$ ).
Analyzer evaluates if $E_v$ is necessary given $E_t$ and the claim, generating a text explanation.
Verifier receives the claim, evidence, and the Analyzer's explanation to make the final prediction.

3. Key Contributions

Challenging the "Visual Necessity" Assumption: The paper provides empirical evidence that adding visual evidence to text-only baselines often reduces accuracy across four different VLMs (Qwen2-VL, Llama-3.2-V, GPT-4o, Gemini-2.5-Pro) when the visual evidence is not strictly required.
Adaptive Multi-Agent Framework (AMUFC): A novel architecture that separates the assessment of evidence necessity from the verification of the claim, allowing the system to dynamically weigh visual inputs.
WebFC Dataset: Introduction of a new, temporally realistic dataset containing 621 claims from PolitiFact (Jan 2024 – Sep 2025). Unlike previous datasets, WebFC restricts evidence retrieval to sources published before the fact-checking article, simulating real-world constraints where future information is unavailable.
Comprehensive Evaluation: Extensive experiments on MOCHEG, FIN-FACT (financial domain), and WebFC, demonstrating generalizability.

4. Experimental Results

A. Visual Evidence Impact Analysis (MOCHEG)

Text-Only vs. Multimodal: Adding "Gold" (perfect) images to text-only inputs decreased accuracy for all tested models.
- Example: Llama-3.2-V dropped from 0.496 (Text) to 0.462 (Text + Gold Image).
Retrieved Images: Adding retrieved images caused even larger performance drops due to noise and irrelevance.
Oracle Setting: When an oracle selected the optimal modality (Text only or Text+Image) for each instance, performance significantly improved (e.g., GPT-4o reached 0.680), proving that adaptive selection is the key to success.

B. AMUFC Performance

MOCHEG Test Set:
- AMUFC achieved 0.612 Accuracy and 0.600 F1 (Gold setting), outperforming all baselines (e.g., MOCHEG baseline: 0.520 ACC; MetaSumPerceiver: 0.556 ACC).
- In the Retrieved setting (realistic scenario), AMUFC maintained a significant lead (0.546 ACC) over the next best method (0.486 ACC).
Ablation Studies:
- Removing the Analyzer (Verifier-only) caused a performance drop.
- Using a simple binary label from the Analyzer (Label-only) was less effective than providing the Analyzer's natural-language reasoning.
- Separating the Analyzer and Verifier roles outperformed a "Unified Verifier" that attempted to do both tasks in one pass.
Cross-Domain Generalization:
- On FIN-FACT (financial claims) and WebFC (newer claims), AMUFC consistently outperformed the Verifier-only baseline, proving the framework adapts well to unseen domains and temporal shifts.

C. Error Analysis

AMUFC reduced the tendency to over-predict "Refuted" labels compared to baselines.
It showed a more conservative approach, increasing "Not Enough Information" (NEI) predictions, which is often preferable in real-world fact-checking to avoid premature judgments.

5. Significance and Conclusion

This paper fundamentally shifts the paradigm of multimodal fact-checking from "always use images" to "use images only when necessary."

Theoretical Insight: It demonstrates that visual evidence is not inherently superior; its utility is context-dependent. Irrelevant images act as noise that distracts VLMs.
Practical Application: The AMUFC framework offers a scalable, agent-based solution that mimics human fact-checkers who first assess the relevance of evidence before drawing conclusions.
Future Directions: The authors suggest extending this framework to support dynamic tool use and exploring multilingual benchmarks to broaden its applicability.

In summary, the paper proves that quality of evidence integration matters more than quantity, and adaptive, agent-driven reasoning is essential for robust multimodal fact-checking.