💬 NLP

The Automatic Verification of Image-Text Claims (AVerImaTeC) Shared Task

This paper presents the AVerImaTeC shared task, which focused on advancing image-text claim verification systems, detailing the evaluation methodology, reporting that all six testing-phase submissions outperformed the baseline with the winning team HUMANE achieving a score of 0.5455, and discussing key insights from the results.

Original authors: Rui Cao, Zhenyun Deng, Yulong Chen, Michael Schlichtkrull, Andreas Vlachos

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rui Cao, Zhenyun Deng, Yulong Chen, Michael Schlichtkrull, Andreas Vlachos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the clues you are given are a mix of a suspicious photo and a shaky rumor.

This paper is about a "detective contest" (called the AVerImaTeC Shared Task) where computer programs (AI) were challenged to figure out if these photo-and-text rumors were True, False, or Unproven.

Here is the story of the contest, broken down simply:

1. The Problem: The "Fake News" Trap

In the real world, bad actors often spread lies by taking a real photo and writing a fake caption under it.

The Old Way: Previous AI tests mostly looked at text-only rumors (like "The President said X").
The New Reality: Most lies today are multimodal—they use pictures and words together.
The Challenge: If an AI just reads the text, it might miss that the photo is from 2016, not today. If it just looks at the photo, it might not know the context. The AI needs to be a super-detective that checks both.

2. The Game Setup: The "Evidence Library"

To make the contest fair, the organizers (the University of Cambridge team) built a massive Evidence Library (called a Knowledge Store).

The Trap: In real life, finding proof on the internet is like looking for a needle in a haystack. You have to pay for search engines, and sometimes you find junk.
The Solution: The organizers did the hard work for the contestants. They pre-scraped the internet and built a library of good evidence, bad evidence, and irrelevant noise for every single claim.
The Rule: The AI had to pick the right "needle" (evidence) from the library to prove its verdict. If the AI guessed the answer correctly but picked the wrong evidence, it didn't get full points. It had to prove why it was right.

3. The Contestants: The "Detective Teams"

Fourteen teams entered the development phase, and six made it to the final round. They had to build systems that could:

Ask Questions: "When was this photo taken?" "Who are these people?"
Find Proof: Dig through the library to find articles or other photos that answer those questions.
Make a Verdict: Decide if the claim is Supported, Refuted, or Not Enough Evidence.
Write a Report: Explain the logic in plain English.

The Winner: The team named HUMANE took first place.

Their Secret Sauce: They realized the "Evidence Library" had some empty shelves (broken links). They built a better "vacuum cleaner" (a more advanced web scraper) to suck up more useful text. They also used a very smart AI brain (Gemini) to connect the dots between the photo and the text.

4. The Scoreboard: How Did They Do?

The judges didn't just count right/wrong answers. They used a special score called the AVERIMATEC Score.

The Analogy: Imagine a student taking a test. If they get the right answer but show no work, they get a zero. They only get points if they get the right answer and show the correct math steps (evidence).
The Result: The winning team scored about 0.55, while the baseline (a simple starter bot) only scored 0.11. This means the new AI systems are getting much better at finding proof, but there is still a long way to go to reach human-level perfection.

5. The Big Lessons (What We Learned)

The paper concludes with three "Aha!" moments for future researchers:

Lesson 1: Better Tools Matter. The winning team didn't just use a better brain; they used a better "magnifying glass." They found that standard web scrapers often miss important details, so using advanced tools to clean up the data is crucial.
Lesson 2: Don't Rely on One Search Engine. When checking if a photo is fake, using just one "Reverse Image Search" tool is like asking one person for directions. Using multiple tools (like Google Lens + another engine) ensures you don't miss the truth.
Lesson 3: The "Black Box" Problem. The best teams used "closed-source" models (AI brains you can't see inside, like a secret recipe). While these are powerful, they are expensive and hard to study. The challenge for the future is to make "open-source" models (public recipes) just as smart without the high cost.

The Bottom Line

This paper is a report card for AI on its ability to fight image-based misinformation.

Good News: AI is getting much better at checking photos and text together.
Bad News: It still struggles with tricky cases (like conflicting evidence) and relies heavily on expensive, secret AI models.
Future Goal: We need to build AI that is not only smart but also transparent, cheap to run, and capable of spotting lies that mix pictures and words perfectly.

In short: The AI detectives are getting sharper, but they still need better magnifying glasses and a more open mind.

1. Problem Definition

The paper addresses the challenge of Automated Fact-Checking (AFC) for multimodal claims (image-text pairs). While existing AFC benchmarks focus primarily on text, online misinformation is increasingly multimodal, with images often used to enhance perceived credibility.

Key Challenges Identified:

Data Scarcity & Quality: Existing datasets are often synthetic (manipulated image-text pairs) or lack sufficient evidence annotations, failing to reflect real-world verification processes.
Temporal Leakage: Models may inadvertently use future information during training due to pre-training on data containing the claims.
Evidence Retrieval: Verifying image-text claims requires retrieving specific evidence (both textual and visual) that is often missing from the claim itself.
Evaluation Difficulty: Standard metrics (e.g., accuracy) do not account for the quality of the retrieved evidence, which is crucial for interpretability.

2. Methodology & Task Design

2.1 The AVerImaTeC Dataset

The foundation of the task is the AVerImaTeC dataset, comprising 1,297 real-world image-text claims extracted from fact-checking articles.

Structure: Each claim includes metadata (date, source), an image, and a ground-truth (GT) verification process decomposed into a sequence of Question-Answer (QA) pairs.
Evidence: GT evidence is explicitly annotated, including URLs, textual snippets, and associated images.
Temporal Constraint: All evidence in the dataset is constrained to be published before the claim date to prevent temporal leakage.
Distribution: The dataset is dominated by Event/Property and Media Analysis claims. It heavily utilizes strategies like Reverse Image Search (RIS) and Image Analysis.

2.2 Knowledge Store

To reduce the financial barrier of using commercial search APIs, the organizers released a curated Knowledge Store:

Content: Contains claim-relevant evidence (textual and visual), adversarial (irrelevant) evidence, and GT evidence.
Construction:
- Textual: Generated via search queries (using Gemini-2.5-Flash) and Google Search API, filtered by date.
- Visual: Generated via Reverse Image Search (Google Cloud Vision) and Google Image Search.
Purpose: Simulates an open-web retrieval environment while ensuring reproducibility.

2.3 Baseline System

A pipeline-based baseline was provided, utilizing Gemini-2.0-Flash (an LLM/MLLM) across four stages:

Question Generator: Creates evidence-seeking questions (few-shot prompted).
Answer Generator: Answers questions using retrieved evidence or VQA capabilities.
Verifier: Aggregates evidence to predict a verdict.
Justification Generator: Produces a textual explanation.

2.4 Evaluation Protocol

The primary metric is the AVerImaTeC Score, a conditional verdict accuracy:

A verdict is only counted as correct if the evidence score exceeds a predefined threshold (0.3).
Evidence Scoring: Uses an LLM-as-a-Judge framework (Gemma-3-27B) to evaluate:
- Textual: Semantic similarity to GT text.
- Visual: Visual similarity (VQA task) between predicted and GT images (0-10 scale; <8 is invalid).
Auxiliary Metrics: Question score, justification score, and evidence recall.

3. Key Contributions

Real-World Multimodal Benchmark: Introduction of the AVerImaTeC dataset, which moves beyond synthetic data to real-world claims with explicit, decomposed evidence annotations.
Curated Knowledge Store: A resource that eliminates the need for participants to pay for search APIs, democratizing access to high-quality retrieval data.
Conditional Evaluation Framework: A novel scoring mechanism that penalizes systems for retrieving poor-quality evidence, even if the final verdict is correct, thereby enforcing rigorous reasoning.
Comprehensive Analysis: Detailed insights into the performance of 14 teams, highlighting the shift toward multimodal retrieval and the challenges of conflicting evidence.

4. Results & System Analysis

4.1 Performance Overview

Participation: 14 teams in development, 6 in testing.
Winning Team: HUMANE achieved an AVerImaTeC score of 0.5455, significantly outperforming the baseline (0.1136).
General Trend: All testing-phase systems surpassed the baseline, indicating rapid progress in multimodal fact-checking.

4.2 Key Technical Insights from Participating Systems

Knowledge Source Enhancement: Top teams (e.g., HUMANE) improved the provided Knowledge Store by using advanced scraping tools (Playwright) to extract content from blocked or poorly scraped URLs, increasing evidence volume by ~24%.
Retrieval Strategies:
- Two-Stage Retrieval: Successful teams combined sparse retrieval (BM25) with dense retrieval (neural embeddings like mxbai-embed-large-v1).
- Multimodal Retrievers: Teams like ADA-AGGR and Xxp utilized specialized multimodal retrievers (ColPali, SigLIP) to handle image-text alignment, proving crucial for Media Analysis claims.
- Multiple RIS Engines: Using multiple Reverse Image Search engines (e.g., Google Lens + Google Cloud Vision) mitigated the "empty result" problem.
Question Answering (QA):
- Iterative vs. One-shot: While some teams (HUMANE, REVEAL) used iterative QA, others (ADA-AGGR) found that removing the QA module saved computation time with minimal performance loss, suggesting retrieval alone might suffice in some cases.
- Joint Generation: Some teams generated questions and answers simultaneously, raising questions about the reliance on parametric knowledge vs. retrieved evidence.
Verdict Prediction: All top systems fed the entire set of retrieved evidence jointly into the verdict model, rather than assessing items individually, emphasizing the need for holistic reasoning.

4.3 Performance by Claim Type

Strong Performance: Systems excelled on Event/Property and Media Analysis claims (the majority of the dataset).
Weak Performance: Systems struggled with Causal claims and, critically, with Not Enough Evidence (NEE) and Conflicting Evidence (CE/C) verdicts. Scores for these categories were near zero, highlighting the difficulty models face in reasoning under uncertainty or conflict.

4.4 Human Evaluation of Evidence

A human evaluation of 100 claims revealed:

Alignment Gap: Automatic evaluation scores showed limited correlation with human judgments (Spearman $\rho \approx 0.31$ ).
Multimodal Discrepancy: The automatic evaluator struggled when systems provided text-only evidence against multimodal ground truth. Annotators often deemed text sufficient, while the automated metric penalized the lack of image evidence.
Reliability: Automatic evaluation was more reliable for high-scoring predictions but less so for text-only evidence in multimodal contexts.

5. Significance & Future Directions

Shift to Real-World Verification: The task establishes a new standard for evaluating AFC systems on real-world, multimodal misinformation, moving away from synthetic benchmarks.
Importance of Evidence Quality: The conditional scoring metric proves that retrieving the right evidence is as important as the final verdict.
Open Questions:
1. Is generating information-seeking queries necessary, or is direct retrieval sufficient?
2. Is iterative QA beneficial, or does it introduce unnecessary complexity?
3. How can open-weight models be fine-tuned to close the performance gap with closed-source models (e.g., Gemini-2.5-Pro)?
Ethical Considerations: The paper acknowledges that biases in the source fact-checking articles are inherited by the dataset and models, and that current evaluation methods need improvement to better align with human judgment on multimodal evidence.

In conclusion, the AVerImaTeC shared task demonstrates significant progress in multimodal fact-checking but highlights that robust evidence retrieval, handling of conflicting information, and better evaluation metrics for multimodal data remain critical areas for future research.