Technical Summary: Context-Driven Paradigm for Verifiable Claim Detection (ContextClaim)

Problem Statement

Automated Fact-Checking (AFC) pipelines typically begin with a filtering stage to determine which claims warrant the expensive downstream processes of evidence retrieval and verification. A central task in this stage is Verifiable Claim Detection (VCD), which assesses whether a statement is, in principle, checkable against external evidence.

Existing approaches to VCD (and the related task of "check-worthiness") rely exclusively on the claim text itself, utilizing linguistic heuristics, surface-level features, or Large Language Models (LLMs) trained on the claim alone. The authors argue this is restrictive because determining verifiability often depends on identifying the specific entities and events mentioned and verifying whether external information about them actually exists. For instance, a claim mentioning fictional characters is unverifiable as a real-world fact, whereas a claim about a public figure is potentially verifiable; this distinction often requires external knowledge not present in the claim text. Current methods fail to leverage this external knowledge at the detection stage, creating a gap between the definition of verifiability (checkability against evidence) and the input available to the detector.

Methodology: ContextClaim

The paper proposes ContextClaim, a paradigm that augments the verifiable claim detection stage by integrating retrieved background context. The pipeline consists of four sequential components:

1. Entity Extraction: The system identifies named entities in the input claim using a BERT-based Named Entity Recognition (NER) model. To address domain-specific gaps (specifically for COVID-19 related claims in the CT22 dataset), the entity type set is extended to include a DISEASE category via lexicon matching, alongside standard types (Person, Location, Organization, Miscellaneous).
2. Context Retrieval: For each extracted entity, the system queries Wikipedia via its API to retrieve candidate article extracts. These candidates are reranked using a hybrid score combining:
   • Semantic similarity between the extract and the original claim.
   • Alignment between the Wikipedia article title and the entity mention.
     A type-based filtering step retains high-confidence extracts for core entities (PER, LOC, ORG) and applies a relevance threshold for miscellaneous or disease entities.
3. Context Summarization: The raw retrieved extracts often contain redundant or irrelevant information. An LLM (either GPT-4o or Mistral-7B) generates a concise, factual summary (~150 words) conditioned on both the original claim and the retrieved knowledge base. This step aims to distill claim-relevant information while suppressing noise and hallucinations.
4. Verifiable Claim Detection: The final classifier receives both the original claim and the generated context summary. The system is evaluated using both encoder-only models (BERT, RoBERTa) and decoder-only models (Llama3, Mistral, GPT-4o) across three learning settings: fine-tuning, zero-shot, and few-shot.

Key Contributions

• Paradigm Shift: The authors introduce ContextClaim, moving retrieval upstream from the verification stage to the detection stage. This allows retrieved context to directly inform the judgment of whether a claim is verifiable, rather than just verifying its truthfulness.
• Systematic Evaluation: The study provides a comprehensive evaluation across two distinct datasets (CT22 for COVID-19 tweets and PoliClaim for political debates), multiple model architectures (encoders and decoders), and three learning paradigms.
• Analysis of Conditions: The paper identifies specific conditions under which context augmentation improves performance, characterizing the roles of domain characteristics, model architecture, and learning settings.
• Cross-Task Utility: The authors demonstrate that context retrieved for detection can be reused for downstream claim verification tasks (specifically on the FEVER dataset), suggesting a reusable retrieval strategy across the AFC pipeline.
• Open Resources: Code, prompts, and configurations are released to support reproducibility.

Results

Experiments were conducted on the CT22 and PoliClaim datasets. The findings indicate that enriching claims with retrieved context generally improves verifiable claim detection, but the magnitude and consistency of gains vary significantly:

• Overall Effectiveness: Context augmentation improved performance in the majority of configurations (13/20 for Accuracy and 14/20 for F1). The best fine-tuning result was achieved by RoBERTa with GPT-4o summaries on PoliClaim (89.71% F1), outperforming previous state-of-the-art systems like AFaCTA.
• Model Architecture: The stability of context integration proved more critical than model scale. Encoder-only models (BERT, RoBERTa) generally integrated context robustly. Decoder-only models showed mixed results; while Llama3 and GPT-4o benefited, Mistral exhibited instability, where context sometimes degraded performance (e.g., a significant F1 drop in fine-tuned Mistral on PoliClaim).
• Learning Settings:
  • Fine-tuning: Provided the most stable environment for context integration, yielding consistent gains across datasets.
  • Zero-shot: Showed benefits, particularly on PoliClaim, where context acted as a "decision anchor" for models lacking demonstration examples.
  • Few-shot: Results were mixed. In some cases, the presence of few-shot examples created latent decision cues that made additional context a competing signal rather than a complementary one, leading to negligible or negative gains.
• Summarization vs. Raw Retrieval: Ablation studies revealed that raw retrieved extracts are unreliable, causing large performance drops in some configurations. LLM-based summarization was crucial for stabilizing these results, often recovering performance lost by raw retrieval and providing additional gains in difficult settings (e.g., few-shot).
• Error Analysis: Context augmentation introduced specific error trade-offs. Decoder-only models tended to develop a false-positive bias (predicting claims as verifiable when context was present but not decisive), while encoder-only models reduced false positives but occasionally introduced false negatives when retrieval failed. Human evaluation confirmed that while summaries were highly relevant, they often lacked "signal clarity" regarding the specific verifiability of the claim.
• Downstream Verification: Applying ContextClaim's retrieved context to the FEVER verification task improved F1 scores by over 1 point, driven primarily by increased recall, confirming the utility of early-stage retrieval for downstream tasks.

Significance and Claims

The paper claims that ContextClaim offers a more objective foundation for claim detection by grounding the "verifiability" judgment in actual external evidence availability, rather than relying solely on subjective linguistic cues.

The authors emphasize that the effectiveness of context augmentation is not uniform; it depends heavily on how stably a model can integrate additional information. They argue that retrieval strategies must be aligned with the downstream model's capacity for context integration. Furthermore, the work highlights that while retrieval improves detection, the quality of the summarization (specifically its ability to provide clear signals for decision-making) is a critical bottleneck.

The paper concludes that while ContextClaim improves detection in most configurations, it is not a universal solution. Its success is contingent on the interplay between the domain (entity coverage), the model architecture (integration stability), and the learning setting (fine-tuning vs. in-context learning). The authors modestly note that their evaluation is limited to two specific domains and that future work is needed to address coverage gaps for emerging events and to improve the "signal clarity" of generated summaries.