Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking

Technical Summary: Novel Claim or Déjà Vu? Rethinking "Contamination-Free" Dynamic Evaluation for Multimodal Automated Fact-Checking

Problem Statement
Multimodal Automated Fact-Checking (MAFC) systems verify claims by retrieving and reasoning over external evidence. However, the evaluation of these systems faces a critical "contamination" problem. Traditional static benchmarks (e.g., AVeriTeC) contain claims from past events that may already exist in the pretraining data of Large Language Models (LLMs). Consequently, models can verify these claims using parametric (internal) knowledge rather than performing the core MAFC task of retrieving external evidence, leading to inflated performance estimates.

While "dynamic" benchmarks have emerged to address this by collecting claims published after an LLM's knowledge cut-off date, this paper questions the assumption that such benchmarks are inherently contamination-free. The authors posit that "novel" post-cut-off claims may still be verifiable using public knowledge available before the cut-off (e.g., historical facts, established policies, or synthesized pre-existing information), thereby allowing models to bypass external retrieval and still achieve high scores.

Methodology
The study employs an empirical approach to quantify contamination and its impact on MAFC evaluation across four Research Questions (RQs):

1. Contamination Detection Pipeline: The authors adapt the evidence sufficiency evaluation pipeline from AVeriTeC. For a given claim, a target LLM generates a fact-checking article based solely on its internal knowledge. An "oracle" fact-checking article (human-written) is retrieved from the original source. The system extracts evidence items from both articles and computes a contamination score based on the similarity between the LLM-generated evidence and the oracle evidence.

   • Metrics: Three similarity metrics are used: METEOR (lexical), Gemma-Emb-0.3B (semantic), and Qwen3-Emb-0.6B (semantic).
   • Thresholds: A claim is flagged as "potentially contaminated" if its similarity score exceeds a predefined threshold ($\tau$).

2. Benchmarks:

   • Static: AVeriTeC (claims up to Oct 2020).
   • Dynamic: ClaimReview2025Q4, a newly constructed benchmark containing 901 claims published in Q4 2025 (after the August 2025 cut-off of GPT-5.2).

3. Models Evaluated: Six state-of-the-art (SOTA) LLMs (GPT-5.2, GPT-4o-Mini, Gemini-3.0-Pro, Gemini-3.0-Flash, DeepSeek-V3.2, Qwen3.5-122B-A10B) are evaluated using the DEFAME framework, which employs a ReAct-style agentic paradigm for evidence retrieval.

4. Experimental Design:

   • RQ1 (Extent): Measure contamination scores on both static and dynamic benchmarks.
   • RQ2 (Cause): Conduct qualitative case studies on high-scoring claims in the dynamic benchmark to identify why they are contaminated.
   • RQ3 (Impact): Compare MAFC performance (Accuracy and Macro-F1) on contaminated vs. uncontaminated subsets of the dynamic benchmark. Analyze search trajectories to understand behavioral changes.
   • RQ4 (Controlled Evaluation): Re-evaluate all models on a unified "contamination-controlled" set (claims uncontaminated for all models under all metrics).

Key Findings
The study yields 16 findings, summarized by three main results:

1. Dynamic Benchmarks Reduce but Do Not Eliminate Contamination:

   • While dynamic benchmarks show lower contamination scores than static ones, they are not immune.
   • 17.09% to 29.30% of claims in the dynamic ClaimReview2025Q4 benchmark remain potentially contaminated across all models and metrics.
   • Semantic similarity metrics identify a significantly larger contaminated subset than lexical matching, suggesting contamination often occurs via semantic reformulation rather than verbatim reuse.

2. Sources of Contamination in Dynamic Benchmarks:

   • Contamination arises because many "new" claims rely on pre-cut-off public knowledge.
   • Direct Reference: Claims may discuss historical events, laws, or policies (e.g., EMTALA, 1986) that are well-documented in training data.
   • Synthesis: Claims may be novel in formulation but verifiable by synthesizing multiple pre-existing facts (e.g., calculating Barron Trump's age in 2028 based on his 2006 birth year and the 1787 Constitution).

3. Significant Inflation of Performance and Distorted Rankings:

   • Contamination artificially inflates MAFC performance. Moving from contaminated to uncontaminated subsets results in statistically significant drops in Macro-F1 (up to 11.34 percentage points) and Accuracy.
   • Behavioral Analysis: Contaminated claims allow models to retrieve relevant evidence more precisely with fewer "Exploration" steps, effectively shortcutting the evidence retrieval process.
   • Ranking Distortion: The ranking of models changes based on contamination. For instance, Gemini-3.0-Flash ranks highest on the contaminated subset, while GPT-5.2 ranks highest on the uncontaminated subset.

Re-evaluation of SOTA Models
Under a strictly contamination-controlled setting (using only claims uncontaminated for all models), the performance of all evaluated LLMs drops significantly.

• DeepSeek-V3.2 achieves the highest Macro-F1 (55.93%).
• All models score below 56% Macro-F1, indicating substantial room for improvement in current MAFC solutions.
• The evaluation set exhibits label imbalance (dominated by "Refuted" claims), necessitating the use of Macro-F1 as the primary metric.

Significance and Contributions
The paper claims to provide a critical re-examination of the "contamination-free" assumption in dynamic evaluation for MAFC. Its primary contributions are:

• Empirical Evidence: Demonstrating that dynamic benchmarks still suffer from non-trivial contamination risks, challenging the notion that time-based filtering is sufficient.
• Methodological Tool: Providing a contamination detection pipeline that can be used to filter benchmarks and ensure trustworthy evaluation.
• Guidelines for Trustworthy Evaluation: Offering practical guidelines for future benchmark construction, emphasizing the need to control for claim novelty beyond simple timestamps and to account for the synthesis of pre-existing knowledge.
• Baseline for Future Work: Establishing a contamination-controlled baseline that reveals the true, currently limited capabilities of SOTA LLMs in multimodal fact-checking, urging the community to focus on genuine generalization to unseen claims.