Technical Summary: DS@GT ARC at CheckThat! 2026

Problem Statement

The paper addresses the challenge of automated verification for numerical claims, which require both language understanding and fine-grained quantitative reasoning. While previous iterations of the CheckThat! task focused on classifying claims (True, False, Conflicting) based on evidence, the CLEF 2026 CheckThat! Task 2 introduces a more complex, test-time scaling framework. The core objective is twofold:

1. Reasoning Trace Ranking: Given a claim, evidence, and multiple reasoning traces generated by Large Language Models (LLMs) at varying temperatures, the system must rank these traces by their utility in arriving at the correct verdict.
2. Verdict Prediction: Derive a final verdict from the top-ranked traces.

The task is evaluated on English and Arabic datasets, utilizing metrics such as Macro-averaged F1 for verdict accuracy and Recall@5/MRR@5 for trace ranking quality. A specific challenge highlighted is the "numeric-truth effect," where the presence of numbers increases a statement's perceived credibility regardless of accuracy, making verification difficult for current automated systems.

Methodology

The authors propose and evaluate two distinct approaches to solve the task, both sharing a core supervision principle: assigning a binary utility label to each reasoning trace (1 if the trace's verdict matches the gold label, 0 otherwise).

Approach I: LLM-Based Verifier with LoRA

This approach frames trace ranking as a binary classification problem using fine-tuned LLMs.

• Architecture: The model takes concatenated inputs of the claim text and the trace verdict. It is fine-tuned using LoRA (Low-Rank Adaptation) to efficiently update model parameters while keeping base weights frozen.
• Backbones: Several models were evaluated, including Llama-3.2-1B, ModernBERT-large, and Qwen2.5-Math-7B. The latter was selected as the final backbone due to its pretraining on mathematical reasoning tasks.
• Sub-Claim Decomposition: The authors experimented with an adaptive decomposition strategy where an instruction-tuned LLM (Llama-3.2-3B-Instruct) splits complex claims into up to two independent sub-claims. These sub-claims are verified individually, and scores are aggregated (via averaging) to determine the final verdict.
• Multilingual Handling: For Arabic, the authors compared a general cross-lingual model against AraBERT, a model pre-trained specifically on Arabic corpora.

Approach II: Grouped Reward-Based Trace Ranking

This approach utilizes a lightweight, interpretable model to score traces without fine-tuning a neural re-ranker.

• Feature Representation: The model uses TF-IDF vectorizers (word-level and character-level) combined with a custom feature block. This block extracts numeric and temporal features (years, months, percentages, currency) using regular expressions to identify overlaps and mismatches between the claim, evidence, and trace.
• Grouped Aggregation: Instead of selecting a single top trace, traces are grouped by their predicted verdict (True, False, Conflicting). Scores within each group are aggregated, incorporating a "count bonus" based on the number of supporting traces.
• Final Prediction: The final verdict is the group with the highest aggregated score. Empty groups are penalized with a score of -1.0 to prevent selecting unsupported verdicts.

Key Results

English Performance

• Overall: Approach I (LLM-based) outperformed Approach II on most metrics, achieving a composite score of 41.88 (average of Macro-F1 and Recall@5) compared to 40.50 for Approach II.
• Recall@5: Approach I achieved a Recall@5 of 24.77, significantly higher than Approach II's 22.21.
• Conflicting Class: Approach II demonstrated superior performance on the "Conflicting" class (F1: 18.02 vs. 12.74 for Approach I). The authors attribute this to the grouped aggregation method's ability to capture distributed evidence support across multiple traces, which is characteristic of ambiguous claims.
• Decomposition Impact: The sub-claim decomposition strategy degraded performance in Approach I (Macro-F1 dropped from 58.98 to 55.79). The authors conclude that splitting claims introduced noise rather than aiding reasoning.
• Model Selection: Qwen2.5-Math-7B was the best-performing backbone, likely due to its mathematical pretraining.

Arabic Performance

• Language Specificity: AraBERT significantly outperformed the general multilingual BERT baseline across all metrics (Macro-F1: 84.85 vs. 82.32; Recall@5: 33.41 vs. 29.54).
• Conflicting Class: No "Conflicting" labels were present in the Arabic test set, so this metric was not reported.

Key Contributions and Significance

The paper makes the following contributions to the field of automated fact-checking:

1. Trace-Level Supervision: It shifts the focus from claim-level classification to reasoning trace ranking, evaluating the quality of the reasoning process itself, not just the final output.
2. Comparative Analysis of Architectures: It provides a direct comparison between heavy, fine-tuned LLM verifiers (Approach I) and lightweight, feature-driven reward models (Approach II). The results suggest that while LLMs are generally superior, lightweight models with grouped aggregation offer a complementary advantage for ambiguous ("Conflicting") cases.
3. Multilingual Insights: The study demonstrates that for Arabic numerical claim verification, language-specific pretraining (AraBERT) yields significant gains over general multilingual models.
4. Decomposition Findings: The paper provides empirical evidence that, in this specific context, automatic sub-claim decomposition can introduce noise and degrade performance, challenging the assumption that decomposition is universally beneficial for complex claims.

Future Directions

The authors identify several avenues for future work:

• Ensemble Methods: Combining the strengths of Approach I (overall performance) and Approach II (Conflicting class performance) into a unified ensemble.
• Data Expansion: Training the lightweight reward model (Approach II) on larger, more diverse external datasets to improve generalization.
• Advanced Decomposition: Exploring learned or retrieval-guided decomposition methods to reduce the noise observed in prompt-based splitting.
• Explainability: Enhancing systems to generate human-readable justifications alongside verdicts to improve interpretability.

The paper concludes that while LLM-based approaches are currently the most effective for overall numerical claim verification, grouped reward modeling offers a valuable, interpretable signal for handling the most difficult, ambiguous cases.