🤖 AI

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

DeepSciVerify is a two-stage, LLM-driven pipeline that enhances scientific claim-citation verification accuracy and efficiency by combining abstract-level reasoning with selective escalation to full-text evidence only when necessary.

Original authors: Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari, Alexander Tessier

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari, Alexander Tessier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to verify a bold statement made in a new book. The author says, "This study proves that coffee cures headaches," and points to a specific research paper as proof.

Your job is to check if that research paper actually supports the claim. This is the core problem the paper DEEPSCIVERIFY tries to solve.

The Problem: The "Blurry Photo" Trap

Usually, when we check a citation, we only look at the abstract (the short summary at the beginning of a paper). Think of the abstract like a blurry thumbnail photo of a painting. It gives you the general idea, but it often misses the fine details, the specific numbers, or the limitations that are crucial to knowing if the claim is actually true.

If you only look at the thumbnail, you might guess the painting is a sunset, but when you finally see the whole picture, you realize it's actually a storm. In scientific writing, this leads to "hallucinations" where AI or authors cite a paper that sounds relevant but doesn't actually prove their point.

The Solution: A Two-Stage Detective System

The authors built a system called DEEPSCIVERIFY that acts like a smart, two-step detective. Instead of immediately reading the entire 50-page research paper (which is slow and expensive), it uses a "lazy but smart" approach:

Stage 1: The Quick Glance (Abstract Level)

First, the system looks at the "blurry thumbnail" (the abstract).

The Detective: It uses a specific AI model (let's call it "The Cautious One") that is very good at saying, "I'm not sure yet."
The Decision:
- If the abstract clearly proves the claim, the detective says, "Yes, this is true!" and stops.
- If the abstract clearly disproves it, it says, "No, this is false!" and stops.
- The Magic: If the abstract is too vague or confusing, the detective says, "I can't tell from this picture," and escalates the case. It doesn't guess; it asks for more evidence.

Stage 2: The Deep Dive (Passage Level)

Only when the first detective is unsure does the system wake up the second team.

The Detective: It pulls out the full research paper (the high-resolution painting).
The Search: Instead of reading every single word, it uses a smart search tool (like a high-tech highlighter) to find the specific paragraphs that talk about the claim.
The Verdict: A second AI model (let's call it "The Balanced One") reads just those specific paragraphs and makes the final call: True, False, or "Still Not Enough Info."

Why This Works: The "Personality" of the AI

The paper discovered that different AI models have different "personalities" when they are unsure:

The Cautious AI (GPT-5.4): This model is like a nervous librarian. If it sees even a tiny gap in the evidence, it immediately says, "I don't know!" This is annoying if you just want a quick answer, but perfect for Stage 1. You want it to say "I don't know" so you don't waste time reading the whole book unnecessarily.
The Decisive AI (Claude Sonnet): This model is like a confident detective who likes to make a call. It's great for the final verdict once it has all the facts, but it might guess too quickly if it only has the blurry thumbnail.

DEEPSCIVERIFY uses the Cautious AI for the quick check and the Decisive AI for the final deep dive. They work together like a perfect team.

The Results

The system was tested on a dataset called SCITANCE (a collection of scientific claims and their citations).

Efficiency: It solved 67% of the cases just by looking at the abstracts. It only had to do the "deep dive" for the remaining 33%. This saves a lot of time and computing power.
Accuracy: By using this two-step method, it was 4.5 points more accurate than systems that only looked at abstracts. It also beat previous record-holders by a significant margin.

The Bottom Line

The paper argues that to verify scientific claims reliably, we shouldn't just throw the whole book at an AI and hope for the best. Instead, we should use a staged approach:

Check the summary first.
If the summary is clear, stop there.
If the summary is fuzzy, then go find the specific pages in the full text that matter.

This method makes scientific verification faster, cheaper, and much more trustworthy, ensuring that when a claim is made, the evidence behind it is actually there.

Technical Summary: DEEPSCIVERIFY

Problem Definition
The paper addresses the critical issue of misalignment between scientific claims and their cited evidence in reports generated by Large Language Models (LLMs). While a citation may be bibliographically valid, it may not semantically support the specific claim it is attached to. Existing verification systems, such as SciFact and SCIVER, primarily rely on abstract-level evidence. However, abstracts often omit crucial methodological details, quantitative results, or limitations necessary to determine if a cited work truly supports a claim. Consequently, abstract-only verification can be insufficient even when the full text contains the relevant evidence. The goal is to verify whether a pair $(c, r)$ —a scientific claim $c$ and a cited reference $r$ —is supported, contradicted, or if there is not enough information (NEI) to make a determination.

Methodology: DEEPSCIVERIFY
The authors propose DEEPSCIVERIFY, a two-stage pipeline designed to balance efficiency and accuracy by combining abstract-level reasoning with selective escalation to full-text evidence.

Phase 1: Abstract-Level Verification with Early Exit
- Retrieval: The system parses the input citation into structured signals (arXiv ID, DOI, URL, title) using an LLM-based parser. It then executes a multi-source retrieval cascade (arXiv API, Semantic Scholar, OpenAlex, etc.) to fetch the paper's abstract.
- Verification: An LLM verifier analyzes the claim against the retrieved abstract.
- Decision Logic: If the model outputs a definitive label (SUPPORTS or CONTRADICTS), the system returns this result immediately (early exit). If the model determines the evidence is insufficient (NEI), the system escalates to Phase 2. This design leverages the observation that some LLMs are naturally more conservative, making them suitable for identifying cases where abstracts are inadequate.
Phase 2: Passage-Level Verification (Selective Escalation)
- Trigger: This phase is activated only when Phase 1 yields an NEI verdict.
- Full-Text Retrieval: The system retrieves the complete paper using a similar cascade strategy (Direct URL, arXiv, Open-access repositories, or LLM-assisted web search).
- Passage Extraction: Using a Retrieval-Augmented Generation (RAG) approach, the full text is segmented into chunks. The system embeds the claim and the text chunks, retrieving the top- $k$ most relevant passages based on cosine similarity.
- Final Verdict: A second LLM verifier analyzes the retrieved passages against the claim to produce the final label.

Key Contributions

Two-Stage Pipeline: The introduction of a system that performs abstract-level verification with an early-exit mechanism, escalating to passage-level evidence only when necessary.
Robust Retrieval Cascades: Development of multi-source retrieval strategies for citation parsing, abstract retrieval, and full-text retrieval, enabling verification even with noisy or incomplete metadata.
Model Calibration Analysis: A detailed analysis revealing that different LLMs exhibit distinct biases under uncertainty. Specifically, GPT-5.4 is found to be overly cautious (frequently defaulting to NEI), while Claude Sonnet 4.6 is overconfident (tending to commit to SUPPORTS/CONTRADICTS even with incomplete evidence). The pipeline leverages these complementary behaviors by using the conservative model for the initial filter and a balanced model for final decisions.
Evidence of Selective Escalation: Demonstration that selectively escalating only uncertain cases improves both accuracy and efficiency compared to baselines that rely solely on abstracts or naively process full text.

Experimental Results
The system was evaluated on the SCITANCE benchmark, a dataset of 656 instances derived from citation sentences.

Performance: DEEPSCIVERIFY achieved 86.7 Micro-F1 and 81.5 Macro-F1 in the three-class setting. This outperforms the strongest abstract-only baseline (GPT-4) by +4.5 Micro-F1 and improves upon the best prior result on SCITANCE by +6.6 Micro-F1.
Efficiency: The system resolves 67.0% of instances at the abstract level, avoiding full-text retrieval for the majority of cases.
Ablation Studies:
- RAG vs. LLM Extraction: The RAG-based passage extraction method (using embedding similarity) significantly outperformed an LLM-based extraction method. The RAG approach improved Micro-F1 by +4.5 points over the abstract-only baseline, whereas the LLM extractor only yielded a +1.1 point gain and degraded the "Support vs. Not Support" metric, likely due to selecting semantically relevant but evidentially weak passages.
- Full-Text Usage: Naive use of full text by single models (without the staged pipeline) resulted in limited or negative gains, highlighting that the method of evidence selection is as important as the volume of evidence.

Significance and Claims
The paper claims that selective evidence escalation is a promising direction for improving scientific claim–citation alignment. The authors argue that the primary challenge is not merely retrieving more information, but identifying the specific parts of a document most informative for a claim. By combining the complementary calibration behaviors of different LLMs with a staged verification process, DEEPSCIVERIFY achieves higher reliability without incurring the computational cost of full-text processing for every instance. The results suggest that this approach effectively mitigates the limitations of abstract-only verification while avoiding the noise associated with processing entire documents indiscriminately. The authors remain modest regarding the scope, noting that their evaluation is limited to the SCITANCE dataset and a single domain, but the findings offer a generalizable strategy for high-stakes scientific verification.