ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models

1. Problem Statement

The paper addresses the challenge of grounded claim verification, where a model must determine if a claim is supported by a given document. While Large Language Models (LLMs) are powerful, they suffer from:

• Hallucinations and reasoning errors: Leading to incorrect verifications.
• Opacity: Lack of explainability makes them unsuitable for high-stakes domains (e.g., healthcare, science).
• Resource constraints: Existing high-performance verifiers (like MiniCheck-7B) or closed-source models (GPT-4) are computationally expensive, raising privacy and cost concerns.

The authors aim to create a compact (1B parameter), resource-efficient, and interpretable verifier that explicitly generates reasoning before making a decision, proving that "thinking" is necessary even for small models.

2. Methodology

A. Model Architecture

• Base Model: A 4-bit quantized Gemma3-1B model.
• Output Format: Unlike standard classifiers that output only a label, ThinknCheck generates a structured output consisting of:
  1. A concise, structured rationale (reasoning trace).
  2. A binary verdict (Supported/Not Supported).
• Training Strategy: The model is fine-tuned using Supervised Fine-Tuning (SFT) on a custom dataset. The authors explicitly tested and rejected Preference Optimization (GRPO) for this scale, finding SFT superior.

B. Dataset Construction

The authors created two key datasets to train and evaluate the model:

1. LLMAggreFact-Think:
   • Derived from the LLMAggreFact benchmark (24.1k examples).
   • Augmented with reasoning chains generated by GPT-4o-mini via zero-shot prompting.
   • Quality Control: Instances where the generated reasoning label mismatched the original ground truth were filtered out.
   • Structure: 4-tuples of (Claim, Document, Verification Label, Reasoning).
2. GSMClaims:
   • A novel benchmark created to test arithmetic reasoning in verification.
   • Reformulated from the GSM8K math dataset.
   • Converts math problems into "Claim + Document" pairs where the claim is either a correct calculation or a plausible incorrect one.
   • Contains 2,634 balanced items.

C. Model Variants & Ablations

To isolate the impact of reasoning, the authors trained several variants:

• ThinknCheck-1B: The full model with supervised reasoning.
• ThinknCheck-nothink-1B: Same architecture/data but trained to output only the label (no reasoning).
• ThinknCheck-Science: A specialized variant fine-tuned on additional SciFact and GSMClaims data to handle scientific and mathematical claims.
• Baselines: Compared against MiniCheck-7B, AlignScore, and zero-shot closed models (GPT-4, Claude).

3. Key Contributions

1. ThinknCheck Model: A 1B-parameter verifier that outperforms 7B-parameter specialized models (MiniCheck) and matches zero-shot closed-source models, while being 7x smaller.
2. LLMAggreFact-Think Dataset: A reasoning-augmented training set that enables small models to learn structured verification chains.
3. GSMClaims Benchmark: A new evaluation suite specifically designed to test arithmetic reasoning capabilities in claim verification, addressing a gap in existing benchmarks.
4. Empirical Insights on Reasoning:
   • Demonstrated that explicit, supervised reasoning is critical for small models; zero-shot Chain-of-Thought (CoT) on the base model actually harms performance.
   • Found that Preference Optimization (GRPO) with simple rewards underperforms SFT at the 1B scale, often leading to "lexical overlap" shortcuts.
   • Identified an inverted-U relationship between reasoning length and accuracy: mid-length rationales perform best, while very short chains lead to false positives (lexical overlap) and very long chains lead to over-cautiousness.

4. Experimental Results

The model was evaluated on LLMAggreFact, SciFact, and GSMClaims using Balanced Accuracy (BAcc).

Key Findings:

• Performance: ThinknCheck-1B (78.1 BAcc) surpasses MiniCheck-7B (77.4 BAcc) on LLMAggreFact despite having 7x fewer parameters.
• Generalization: On SciFact, ThinknCheck-1B achieves 64.7 BAcc, a massive +14.7 point gain over MiniCheck-7B.
• Ablation Impact: Removing the reasoning step (ThinknCheck-nothink) causes a catastrophic drop in performance (e.g., from 78.1 to 57.5 on LLMAggreFact, and 64.7 to 21.7 on SciFact), proving that reasoning drives generalization.
• Arithmetic: The specialized ThinknCheck-Science variant achieves 61.0% accuracy on GSMClaims, a 17% relative improvement over the standard variant.

5. Significance and Conclusion

• Efficiency vs. Performance: The paper proves that for grounded verification, small, reasoning-supervised models can outperform larger, non-reasoning models. This is crucial for deploying AI in resource-constrained or privacy-sensitive environments.
• Interpretability: By forcing the model to generate a rationale before the verdict, the system provides human-readable explanations, increasing trust in high-stakes applications.
• Methodological Shift: The results suggest that for small models (1B scale), Supervised Fine-Tuning (SFT) on structured reasoning data is a more effective strategy than zero-shot prompting or Preference Optimization (GRPO).
• Future Directions: The authors highlight the need for adaptive reasoning budgets (to avoid over/under-thinking) and the integration of external tools (like calculators) for complex arithmetic tasks.

In summary, ThinknCheck demonstrates that "thinking" is not just a feature of massive models but a necessary component for accurate, interpretable, and efficient claim verification in compact models.