Multilingual, Multimodal Pipeline for Creating Authentic and Structured Fact-Checked Claim Dataset

1. Problem Statement

The rapid proliferation of misinformation across online platforms has created an urgent need for robust, explainable, and multilingual fact-checking resources. However, current automated fact-checking research faces several critical limitations:

• Data Scarcity & Bias: Existing benchmarks are predominantly English-centric, text-only, and static, failing to capture the multimodal nature (images, videos) of modern disinformation.
• Lack of Structural Depth: Current datasets often lack fine-grained annotations linking claims to specific evidence types and verdicts. They treat retrieved documents as interchangeable text rather than functionally distinct sources (e.g., distinguishing between expert testimony, official records, and eyewitness accounts).
• Methodological Gaps: Automated systems often rely on secondary sources (like Wikipedia) rather than primary evidence (official records, court documents) and fail to replicate the "source criticism" and triangulation processes used by professional human fact-checkers.
• Explainability: Existing systems rarely provide local, checkable pointers (e.g., text spans, timestamps) or structured justifications that link evidence to verdicts.

2. Methodology

The authors propose a fully scriptable, reproducible, multi-stage pipeline to construct large-scale, multilingual (French and German), and multimodal fact-checking datasets. The pipeline integrates real-world data from fact-checking organizations and leverages Large Language Models (LLMs) for processing.

A. Data Collection Pipeline (5 Stages)

1. Aggregation: The pipeline aggregates ClaimReview feeds (via Google API and publisher sites) to collect claim statements, metadata, and URLs to full debunking articles.
2. Scraping: It scrapes full debunking articles from publisher sites, extracting text, embedded images, captions, and videos. Organization-specific adapters are used to handle varying content structures.
3. Metadata Enrichment: A few-shot prompted GPT-4o model categorizes claim sources based on concepts in the text, images, and statistics.
4. Verdict Normalization: To handle heterogeneous rating schemes (binary, numeric, free-text) across different organizations, the pipeline maps them deterministically to a unified label set (True, False, Mixture/Partially True, Other) based on the ClaimsKG schema.
5. Evidence Extraction & Categorization: This is the core innovation. Instead of simple text matching, the pipeline uses a fine-grained, journalism-based evidence schema to categorize evidence into six epistemic roles:
   • Expert Testimony (professional analysis/judgment).
   • Quantitative Data & Statistics (measurable figures).
   • Official Records & Legal/Policy Documents (government/institutional reports).
   • Media Record (journalism/social media circulation).
   • Multimedia Evidence (photos, videos, charts used as proof).
   • Eyewitness & Firsthand Accounts (direct personal observation).

B. LLM-Based Processing

• Evidence Extraction: State-of-the-art LLMs (Gemini-2.5-Pro, Llama3.3-70B, Qwen2.5-72B) are prompted to extract evidence from debunking articles and assign them to the six categories. The prompts enforce grounding in the source text only, using stable locators (timestamps, figure IDs) where available.
• Justification Generation: LLMs and Multimodal LLMs (MLLMs) generate structured justifications that explicitly reason over the categorized evidence to support the verdict.
  • Text-only mode: Uses extracted text evidence.
  • Multimodal mode: Incorporates visual evidence (images, video frames, captions) and enforces grounded references to visual cues.

3. Key Contributions

1. A Reproducible Pipeline: A fully scriptable end-to-end system for creating multilingual, multimodal fact-checking datasets enriched with structured metadata and evidence annotations.
2. Two New Large-Scale Datasets:
   • French Dataset: 5,170 claims (2014–2025) from multiple organizations.
   • German Dataset: 3,555 claims (2017–2023).
   • Both include aligned visual media (images/videos) and fine-grained evidence categorization.
3. Journalistic Evidence Schema: A novel categorization framework that models the epistemic diversity of real-world fact-checking, distinguishing evidence by its role (e.g., expert vs. eyewitness) rather than just modality.
4. Cross-Organizational Analysis: The datasets enable the analysis of verification strategies across different media systems and organizations, revealing how evidence usage varies by publisher and verdict type.
5. Comprehensive Benchmarking: An evaluation of LLMs and MLLMs for evidence extraction and justification generation using both automated (G-Eval) and human assessment.

4. Results & Evaluation

The system was evaluated using G-Eval (LLM-as-a-judge with GPT-4o) and Human Annotation across three criteria: Correctness, Coherence, and Completeness.

• Model Performance:
  • Gemini-2.5-Pro consistently outperformed other models (Qwen2.5, Llama3.3) in both evidence extraction and justification generation across all metrics and languages.
  • Multimodal Integration: When visual evidence (images/video frames) was included, Gemini-2.5-Pro showed superior integration, yielding higher coherence and correctness scores compared to text-only baselines. Qwen2.5-VL showed weaker alignment between textual and visual signals.
• Evidence Usage Patterns:
  • Publisher Specialization: Science-focused outlets relied heavily on expert testimony and quantitative data, while general news outlets used more multimedia evidence.
  • Verdict Correlation: "Partially True" claims relied most heavily on documentary evidence (expert/official/quantitative), whereas "False" claims often involved higher usage of multimedia and eyewitness accounts (particularly in the French dataset).
• Human Validation: Human annotators confirmed the reliability of the LLM-as-a-judge approach, showing a strong correlation between human scores and G-Eval scores.

5. Significance

• Bridging the Gap: The work bridges the divide between automated fact-checking systems and professional journalistic verification practices by encoding the "how" of verification (source criticism, triangulation) into the dataset structure.
• Explainability: By forcing models to categorize evidence and generate justifications based on specific epistemic roles, the pipeline promotes transparent and interpretable AI decision-making.
• Multilingual & Multimodal: It addresses the critical gap in non-English (French/German) and multimodal resources, moving beyond text-centric benchmarks to reflect the reality of modern misinformation.
• Future Research: The released datasets and pipeline provide a foundation for developing more robust, evidence-grounded fact-checking models and for conducting cross-cultural studies on misinformation verification strategies.

The authors emphasize that LLMs in this framework are not acting as fact-checkers themselves but as extraction and reasoning tools that operationalize the complex workflows of human journalists, thereby creating a more trustworthy and interpretable verification system.