Technical Summary: CAAFC – Chronological Actionable Automated Fact-Checker

Problem Statement

The rapid proliferation of online content, including AI-generated text prone to hallucinations, has rendered manual fact-checking infeasible. Existing Automated Fact-Checking (AFC) systems suffer from critical misalignments with professional fact-checking practices. Specifically, current systems often:

1. Rely on Secondary Sources: They frequently retrieve evidence from media outlets rather than primary sources (e.g., official statements, raw data), introducing interpretative bias.
2. Lack Temporal Grounding: They fail to account for the chronological nature of facts, often using outdated evidence or missing temporal context, leading to "evidence chronological mismatches."
3. Unify Tasks Poorly: They treat non-factual hallucination detection and automated fact-checking as separate problems rather than a unified task.
4. Generate Low-Quality Justifications: Their explanations often lack explicit structure, traceability to sources, and actionable corrections (identifying errors, providing fixes, and citing sources).
5. Over-rely on Large Models: There is an assumption that only massive Large Language Models (LLMs) can perform these tasks effectively, ignoring the potential of decomposing tasks for smaller, efficient models.

Methodology: The CAAFC Framework

The authors introduce CAAFC (Chronological Actionable Automated Fact-Checker), a modular framework designed to bridge the gap between professional methodologies and automated systems. CAAFC processes claims, conversations, and dialogues using quantized LLMs (primarily Gemma3-27B) and is structured around six key modules:

1. Extractor-Segmentor: Unlike classical pipelines that filter for "check-worthy" claims, this module extracts all possible claims from input text and decomposes complex claims or conversations into atomic sub-claims. This granularity allows for precise verification and transparency.
2. Primary Chronological Evidence Retriever: This module directs retrieval toward primary sources (e.g., official government records, original statements) rather than secondary media. Crucially, it incorporates chronology, fetching evidence relevant to the time the claim was made as well as updated knowledge up to the present. It utilizes Google AI Mode (Gemini-Pro-3) to summarize search results from these primary sources.
3. Fact-Checker: This module evaluates each atomic claim against the retrieved evidence, assigning a veracity label (True, False, or Unverifiable) and generating a preliminary justification. A final verdict for the overall claim is aggregated using a logic where a single false atomic claim renders the whole claim false.
4. Actionable Justifier: A novel module that synthesizes the claim, evidence, and atomic veracity labels into a structured, actionable explanation. This explanation explicitly identifies factual errors, provides corrections, and cites supporting sources.
5. Actionability Evaluator: This module assesses the quality of the generated justification using the FinGrAct framework. It scores the justification based on three criteria: error detection, error correction, and the relevance/functionality of cited links. A score of 4/7 or higher is required for acceptability.
6. Justification Revisory: If the Actionability Evaluator finds the justification lacking, this module refines the output by incorporating feedback to add missing corrections or highlight overlooked errors, ensuring the final output meets the actionability threshold.

Key Contributions

The paper outlines five primary contributions addressing specific research questions:

• C1 (Chronology & Knowledge Updates): CAAFC introduces a method to detect outdated or missing knowledge in databases by retrieving timestamped, ordered evidence. It corrects knowledge bases based on this new chronological evidence, ensuring explanations reflect the correct sequence of events.
• C2 (Primary Source Alignment): The framework explicitly mimics professional fact-checkers by prioritizing original/primary sources over secondary media, thereby reducing interpretative bias.
• C3 (Task Unification): It reformulates non-factual hallucination detection and automated fact-checking into a single, unified task for misinformation detection and correction.
• C4 (Efficiency via Decomposition): The authors demonstrate that decomposing the complex task of fact-checking into smaller sub-tasks allows smaller, quantized LLMs (specifically Gemma3-27B) to outperform significantly larger frontier models (e.g., GPT-5.2, LLAMA3.3-70B, GPT-OSS-120B) in terms of accuracy and macro F1-score, while requiring substantially fewer computational resources (17 GB GPU memory vs. significantly higher for larger models).
• C5 (Structured Actionability): The framework introduces new architectural components (Actionable Justifier, Evaluator, Revisory) to generate high-quality, structured justifications that enhance transparency and user trust, moving beyond simple true/false labels.

Experimental Results

The framework was evaluated on four benchmark datasets: AVeriTeC, CoverBench, FACTors, and DiaHalu (dialogue-based hallucination detection).

• Evidence Quality: The authors identified a significant "evidence chronological mismatch" in existing datasets, where dataset evidence was outdated or lacked context compared to Google-retrieved primary sources. Human annotators confirmed that Google-retrieved evidence was superior in these cases.
• Performance on Fact-Checking:
  • On AVeriTeC, CAAFC (Gemma3-27B) achieved 0.894 accuracy (after data cleaning), outperforming InFact (0.720) and fine-tuned LLaMA 3.1-70B (0.752).
  • On CoverBench, it achieved 0.829 accuracy, surpassing the SOTA Claude 3.5 Sonnet (0.689).
  • On FACTors, it reached 0.919 accuracy, exceeding the fine-tuned RoBERTa baseline (0.849).
  • Notably, the smaller Gemma3-27B model matched or exceeded the performance of models with 2.6x to 4x more parameters.
• Performance on Hallucination Detection: On the DiaHalu dataset, CAAFC achieved a 0.684 macro F1-score, outperforming GPT-5.2 (0.682) and significantly outperforming GPT-4 (0.473) and Temporal Graph Networks (0.495).
• Actionability: The generated justifications received high actionability scores (mostly 6-7 out of 7) according to the FinGrAct framework, indicating they successfully identified errors, provided corrections, and cited functional sources.

Significance and Claims

The paper claims that CAAFC represents a shift in how automated fact-checking should be approached. Its significance lies in:

1. Proving that smaller models can outperform larger ones when the task is properly decomposed and supported by high-quality, chronological evidence.
2. Bridging the gap between professional and automated fact-checking by adhering to professional methodologies (primary sources, temporal grounding, and traceable justifications).
3. Enhancing reliability and transparency through actionable, structured explanations that allow users to verify claims independently.
4. Demonstrating the critical importance of evidence chronology, showing that many "errors" in existing benchmarks are actually artifacts of outdated dataset evidence.

The authors acknowledge limitations, including the current restriction to English-language content and the potential for search engine bias in retrieved evidence. They also note that while the framework reduces GPU memory requirements, the multi-step inference process introduces latency, which they plan to address in future work.