ClaimDB: A Fact Verification Benchmark over Large Structured Data

1. Problem Statement

Real-world fact-checking increasingly relies on verifying claims grounded in large-scale structured data (e.g., government databases, healthcare records, financial reports). However, existing fact-verification benchmarks (like FEVER, TabFact, or BIRD) suffer from a critical limitation: the evidence provided is small enough to fit within a Large Language Model's (LLM) context window (typically thousands of tokens).

In reality, high-stakes claims often require reasoning over millions of records across multiple tables. For example, verifying a claim about national inflation rates or crime statistics involves aggregating data from datasets that, when converted to text, can exceed 100 million tokens.

• The Gap: Current benchmarks assume evidence can be "read" in-context. This assumption fails at scale, forcing a shift from simple retrieval to executable reasoning (e.g., writing and executing SQL queries).
• The Challenge: Existing models struggle to navigate, aggregate, and reason over data volumes that far exceed their context windows, and they often fail to correctly identify when information is insufficient (the "Not Enough Info" or NEI case).

2. Methodology: The CLAIMDB Benchmark Construction

The authors introduce CLAIMDB, a benchmark designed to bridge the gap between small-scale table verification and real-world large-scale data analysis.

A. Data Source & Scale

• Source: Derived from the BIRD (Big Bench for Large-Scale Database Grounded Text-to-SQLs) benchmark, which contains realistic, "dirty" databases with millions of rows.
• Scale: The benchmark consists of 80 unique real-life databases covering 11 domains (e.g., Governance, Healthcare, Sports, Economy).
• Statistics:
  • Total Claims: 53,368.
  • Evidence Context: Each claim is paired with a database averaging 11.3 tables and 4.6 million records.
  • Token Count: Approximately 110 million tokens of evidence per claim (far exceeding the 1M–2M token limits of even the most advanced long-context LLMs).

B. Claim Generation Pipeline (Figure 1)

The construction process involves four key stages:

1. Pre-Filtering (Compositional Queries): Starting with 11k NL-to-SQL pairs from BIRD, the authors filter for compositional queries. They use Abstract Syntax Trees (ASTs) to retain only queries involving aggregations (AVG, SUM), orderings (MAX/MIN), window functions, or multi-table joins (3+ tables). Simple lookups are discarded because they don't require complex reasoning.
2. Execution: SQL queries are executed on the real databases to generate ground-truth Question/Answer pairs.
3. Claim Generation: Using gpt-5, the system generates three types of claims for each Q/A pair:
   • Entailed: Supported by the answer.
   • Contradicted: Factually incompatible with the answer.
   • Not Enough Info (NEI): Claims that cannot be verified. These are further categorized into:
     • Out-of-Schema: Concepts not present in the database schema.
     • Counterfactual: Hypothetical "what-if" scenarios.
     • Subjective: Opinions or value judgments.
4. Quality Control (LLM Judges): A panel of diverse LLMs (Phi-4, grok-3-mini, mistral-small) acts as judges to filter out low-quality claims. They check for:
   • Label correctness.
   • Self-containment (no opaque references to "the question above").
   • Schema leakage (ensuring NEI claims don't accidentally reveal database structure).
   • Result: Approximately 14% of generated claims were filtered out, leaving a high-quality final set.

C. Benchmark Splits

The dataset is split into Training, Public Test, and Private Test sets. The test sets are balanced across labels (Entailed, Contradicted, NEI) and subcategories. For NEI claims, the test set is restricted to the top quartile of similarity scores (semantic similarity between the claim and the database concepts) to ensure the claims are challenging and not trivially obvious.

3. Evaluation Setup

The authors evaluated 30 state-of-the-art LLMs (both proprietary and open-source, mostly <70B parameters) using a neuro-symbolic architecture:

• Agent Setup: Models act as agents with a single tool: the ability to execute arbitrary SQL queries against the database.
• Constraint: Models are limited to 20 tool calls per claim.
• Goal: The model must reason through the data by composing SQL queries to verify the claim, rather than reading the data directly.
• Metrics: Accuracy and Macro-F1 across the three classes (Entailed, Contradicted, NEI).

4. Key Results

The experiments revealed significant challenges for current LLMs in this setting:

• Overall Performance: More than half of the evaluated models (17 out of 30) scored below 55% accuracy.
• Top Performers:
  • gpt-5-mini achieved the highest accuracy (82.7%) and Macro-F1 (82.8%).
  • claude-haiku-4.5 (80.9%) and gemini-3-flash (80.1%) followed closely.
  • gpt-oss-20b (an open-source model) performed on par with top proprietary models (74.0%), showing promise for open-source reasoning.
• Scaling Trends: Larger models generally performed better, but improvements were marginal and roughly log-linear.
• The "NEI" Problem (Abstention):
  • Proprietary Models: Showed a strong bias against predicting NEI. They rarely abstained, often guessing "Entailed" or "Contradicted" even when evidence was missing.
  • Open-Source Models: Showed the opposite bias, over-predicting NEI (predicting "Not Enough Info" roughly 50% of the time even for clear Entailed/Contradicted cases).
  • Significance: Neither behavior is ideal for high-stakes fact-checking, where admitting uncertainty is crucial.
• Tool Usage:
  • Top-performing models averaged 4–8 SQL tool calls.
  • Models that made too few calls (guessing without data) or too many calls (losing focus) performed poorly.
  • Failure Analysis: Failures were rarely due to SQL syntax errors (top models had >90% query success rates). Instead, failures stemmed from reasoning breakdowns—the inability to correctly compose a sequence of queries to navigate the database logic.
• Data Contamination: When tested without tool access (relying only on parametric knowledge), the best model dropped to near-random performance (36.7% accuracy), confirming the benchmark is not solvable via memorization.

5. Key Contributions

1. CLAIMDB Benchmark: The first fact-verification benchmark specifically designed for large-scale, multi-table structured data (millions of records), forcing a shift from in-context reading to executable reasoning.
2. Realistic Scale: By utilizing 80 real-world databases with ~110M tokens of context per claim, it exposes the limitations of current context-window-based approaches.
3. Comprehensive Evaluation: A rigorous evaluation of 30 models revealing that current SOTA models struggle significantly with compositional reasoning over large data and, critically, with abstention (knowing when they don't know).
4. Open Resources: Release of the dataset, code, and a public leaderboard to drive future research in neuro-symbolic fact verification.

6. Significance

CLAIMDB addresses a critical blind spot in AI research. While LLMs are excellent at processing text and small tables, they are currently unreliable for verifying claims based on massive, real-world datasets. The benchmark highlights that:

• Reasoning > Reading: Future fact-checking systems must rely on executable programs (SQL, code agents) rather than just context retrieval.
• Reliability Gap: The inability of models to correctly handle "Not Enough Info" cases poses a severe risk for deploying LLMs in high-stakes domains (e.g., legal, medical, financial) where hallucination or forced guessing can have serious consequences.
• Open-Source Potential: The strong performance of models like gpt-oss-20b suggests that open-source models can catch up to proprietary ones in reasoning tasks if trained or prompted correctly.

In summary, CLAIMDB serves as a necessary stress test for the next generation of AI agents, pushing the field beyond "reading" data toward "reasoning" with data at scale.