Entailed Opinion Matters: Improving the Fact-Checking Performance of Language Models by Relying on their Entailment Ability

1. Problem Statement

Automated fact-checking is a critical task for combating misinformation, yet existing systems struggle with real-world deployment due to low accuracy. The primary challenges identified are:

• Contradictory Evidence: Claims are often supported by multiple pieces of evidence that contradict one another. Standard models struggle to parse these conflicts, leading to poor decision-making.
• Limitations of Current Approaches:
  • Encoder-Only Models (ELMs): While effective, they often rely on attention mechanisms that may discard useful information when focusing only on top-k evidence or fail to handle long, noisy contexts.
  • Generative Language Models (GLMs) via Prompting: While large GLMs (e.g., >1 trillion parameters) can achieve human-level performance via prompting, they are computationally expensive, often behind paywalls, and prone to hallucinations and bias when used on smaller models (e.g., 7B parameters).
• The Gap: There is a lack of a simple, effective training paradigm that leverages the strong reasoning capabilities of GLMs to preprocess evidence for more efficient, accurate, and deployable models.

2. Methodology

The authors propose a new learning paradigm called Entailed Opinion Matters, which utilizes the entailment ability of Generative Language Models (GLMs) to generate structured justifications that train Encoder-Only Language Models (ELMs). The process follows a three-stage pipeline (illustrated in Figure 1):

1. Entailment (Evidence Classification):
   • A GLM is prompted to analyze a claim and its associated evidence sentences.
   • The GLM classifies each evidence sentence as either "Supporting" or "Refuting" the claim based on logical entailment.
2. Evidence Consolidation (Justification Generation):
   • The same GLM consolidates the classified evidence into two concise, natural language justifications:
     • A Supporting Justification (synthesizing all supporting evidence).
     • A Refuting Justification (synthesizing all refuting evidence).
   • If evidence is missing for a category, the GLM generates the justification based on its internal knowledge.
3. Veracity Prediction (Training ELMs):
   • An Encoder-Only Language Model (ELM) is fine-tuned using the Claim concatenated with the Supporting Justification and the Refuting Justification.
   • The ELM learns to predict the final veracity label (e.g., True, False, Half-True) based on these synthesized inputs rather than raw, potentially contradictory evidence.

Experimental Setup:
The authors conducted two types of experiments to validate this approach:

• Training-Based Experiments (TBEs): Fine-tuning ELMs (RoBERTa, XLNet) and GLMs (Mistral, Llama, Gemma, Qwen, Falcon) using LoRA/LoRA+ adapters.
  • TBE-1: Training on raw evidence.
  • TBE-2: Training on "overall understanding" of claim-evidence.
  • TBE-3: Training on entailed justifications (the proposed method).
• Inference-Based Experiments (IBEs): Zero-shot and few-shot prompting strategies (including Chain-of-Thought) using GLMs directly.

Datasets:
The study evaluated performance on:

• Monolingual: LIAR-RAW (6 classes) and RAW-FC (3 classes).
• Multilingual: X-FACT (25 languages) and RU22Fact (4 languages).
• Multimodal: Factify-2, MOCHEG, and VERITE.

3. Key Contributions

• Novel Training Paradigm: The paper introduces a method where GLMs act as "reasoning engines" to preprocess evidence into structured justifications, which are then used to train smaller, more efficient ELMs. This bridges the gap between the reasoning power of large models and the efficiency of encoder-only models.
• Evidence Discrimination via Entailment: Unlike prior works that use attention scores or simple retrieval, this approach explicitly uses the GLM's entailment capability to separate evidence into supporting and refuting groups before prediction.
• Comprehensive Evaluation: The study includes rigorous ablation studies, error analysis, quality evaluation of model explanations (via lexical overlap and subjective scoring), and domain generalization tests across multimodal and multilingual settings.
• State-of-the-Art Results: The proposed method significantly outperforms existing baselines (HiSS, FactLLaMa, RAFTS, L-Defense) and standard prompting strategies.

4. Key Results

• Performance Gains (TBE-3):
  • On LIAR-RAW, the proposed method (TBE-3) improved Macro-F1 (MF1) by up to 28.57% over the best baseline.
  • On RAW-FC, it achieved an improvement of up to 44.26% over the baseline.
  • Specifically, RoBERTa fine-tuned with Llama-based entailed justifications achieved an MF1 of 0.88 on RAW-FC.
• Inference vs. Training: Inference-based methods (IBEs) using prompting alone performed poorly compared to the training-based approach, highlighting that GLMs struggle to utilize justifications effectively without fine-tuning in a zero-shot setting.
• Ablation Study: Removing either the supporting or refuting justifications caused a significant drop in performance (MF1 dropped by ~31% and ~9% respectively on LIAR-RAW), confirming that both types of synthesized evidence are crucial.
• Domain Generalization:
  • Multimodal: The approach improved MF1 by up to 54.10% on multimodal datasets (MOCHEG).
  • Multilingual: It improved MF1 by up to 19.70% on multilingual datasets (X-FACT), demonstrating robustness across languages and modalities.
• Explanation Quality: Subjective evaluations revealed that Llama generated the most informative, logical, and readable justifications. Models trained on Llama-generated justifications achieved the highest veracity prediction scores, showing a strong correlation between explanation quality and downstream performance.
• Attention Analysis: Linguistic analysis showed that ELMs trained on this paradigm focused more accurately on factual keywords for extreme labels (True/False) but exhibited scattered attention for borderline cases (Half-True), indicating a remaining challenge in handling ambiguity.

5. Significance

This paper makes a significant contribution to the field of automated fact-checking by demonstrating that decomposing the fact-checking task into evidence classification, justification generation, and final prediction yields superior results compared to end-to-end training or direct prompting.

• Efficiency: It allows for the deployment of high-performance fact-checking systems using smaller, open-source ELMs (which are cheaper and faster to run) by leveraging the reasoning of larger GLMs only during the data preparation/training phase.
• Robustness: The method effectively handles contradictory evidence, a major failure point in previous systems, by forcing the model to synthesize opposing arguments before making a decision.
• Scalability: The approach is scalable across different languages and modalities, offering a unified framework for real-world misinformation detection.

In conclusion, the authors prove that "entailed opinions" (structured justifications derived from entailment) serve as a powerful supervision signal, enabling smaller models to achieve state-of-the-art fact-checking performance.