💬 NLP

Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification

This paper introduces NEI-CAP, a diagnostic protocol revealing that fact verification models often fail to generalize "Not Enough Information" competence across different evidence construction methods because aggregate scores can mask underlying reliance on shortcut cues and construction-specific artifacts.

Original authors: Jingxi Qiu, Zeyu Han, Cheng Huang

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingxi Qiu, Zeyu Han, Cheng Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a detective to solve mysteries. Your goal is to see if they can tell the difference between three things:

The case is solved (Evidence supports the claim).
The case is cracked (Evidence refutes the claim).
We don't have enough info yet (The "NEI" label).

This paper argues that while we've been testing detectives on how well they solve cases, we've been tricking them on the "We don't have enough info" test. We've been giving them clues that are so obviously missing that the detective doesn't need to be smart to spot the problem; they just need to spot the format of the missing clue.

Here is the breakdown of the paper's findings using simple analogies.

1. The Problem: The "Empty Box" Trick

In fact-checking, when a computer model says "Not Enough Information" (NEI), it means the evidence provided isn't enough to prove or disprove a statement.

The paper found that many datasets (the training tests for these models) create "NEI" examples in a lazy way.

The "Easy" Trick: They give the model a claim like "The Great Wall is in Brazil" and then give it no evidence at all (an empty box) or a piece of text about pizza (totally unrelated).
The Result: The model learns a shortcut. It thinks, "Oh, if there's no text or the text is about pizza, I should say 'Not Enough Info'." It's not actually checking if the evidence is insufficient; it's just checking if the evidence is missing or weird.

2. The Real Test: The "Half-Full Box"

The paper introduces a new way to test called NEI-CAP. Instead of just giving the model an empty box, they give it a half-full box.

The Scenario: The claim is "The Great Wall is in Brazil." The evidence provided is a long, detailed paragraph about the Great Wall, but it only talks about its length and where it is in China. It doesn't mention Brazil, but it's clearly related to the topic.
The Challenge: A smart detective should realize, "This text is about the Great Wall, but it doesn't tell me if it's in Brazil. I need more info."
The Failure: The paper found that models trained on the "Empty Box" tricks (the easy NEI) completely fail this "Half-Full Box" test. They look at the related text, see that it looks supportive (because it's about the Great Wall), and confidently say, "This supports the claim!" or "This refutes it!" They miss the fact that the specific detail they need is missing.

3. The "Construction" Metaphor

The authors call the way these tests are built "Construction Families."

Easy Construction: Like building a wall out of cardboard. It looks like a wall, but it's flimsy. If you train a model on cardboard walls, it learns to spot cardboard, not real bricks.
Hard Construction: Like building a wall out of real bricks but leaving a hole in the middle. The model needs to understand the structure of the wall to know it's incomplete.

The paper shows that if you train a model on "cardboard" (easy, unrelated, or empty evidence), it gets a perfect score on the test. But the moment you switch to "real bricks with a hole" (semantically related but insufficient evidence), the model collapses. It scores zero.

4. The "Magic Score" Illusion

Currently, when we look at a model's report card, we see one big number: "NEI Score."

The Paper's Warning: This number is a lie. It's like a student getting an "A" on a math test because the teacher only asked them to add zeros. If you ask them to add fractions, they fail.
The paper proves that a model can have a perfect "NEI Score" on easy tests but be completely useless on real-world scenarios where evidence is related but incomplete. The "Average Score" hides the fact that the model is just memorizing the format of the test, not learning the skill.

5. The Solution: NEI-CAP

The authors propose a new protocol called NEI-CAP. Think of this as a new way to grade the detective:

Don't just give a score: Instead, report how the test was built. Did we use the "Empty Box" or the "Half-Full Box"?
Audit the tricks: Check if the model is just spotting shortcuts (like "no text = NEI") or if it actually understands the content.
Human Check: For the hardest tests, humans verify that the evidence is truly insufficient, ensuring the model isn't being tricked by ambiguous data.

Summary

The paper concludes that the way we create "Not Enough Information" tests determines what the AI learns.

If we make the tests too easy (empty boxes), the AI learns to spot emptiness, not insufficiency.
If we make the tests hard (related but incomplete info), the AI actually has to think.
Currently, most AI models are "cheating" on the easy tests. They look smart, but they fail when the evidence is tricky. The paper urges us to stop hiding behind average scores and start reporting exactly what kind of "evidence construction" the model was tested on.

Technical Summary: Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification

Problem Statement

Fact verification systems typically classify claims into three categories: SUPPORTED, REFUTED, or NOT ENOUGH INFORMATION (NEI). The NEI label is intended to indicate that the available evidence set $E$ is insufficient to establish the truth or falsity of a claim $c$ . However, the paper argues that the operationalization of the NEI label is a design choice that varies significantly across benchmarks (e.g., FEVER, SciFact, HoVer) and is often left undefined in evaluation protocols.

The core problem identified is that "evidence absence" (e.g., empty fields, unrelated passages) is observationally distinct from "evidence insufficiency" (e.g., semantically related but incomplete evidence). Current benchmarks often conflate these conditions. Consequently, models may learn to recognize specific construction artifacts (such as missing text, topic mismatches, or positional biases) rather than learning the intended capability of identifying genuine evidence insufficiency. This leads to a situation where aggregate NEI performance metrics (like NEI-F1) can be high while the model fails completely on semantically related insufficient evidence.

Methodology: NEI-CAP

To address this, the authors introduce NEI-CAP (NEI Construction-Aware Protocol), a diagnostic framework that treats the construction of NEI evidence as an explicit variable in evaluation.

1. Construction Taxonomy

NEI-CAP categorizes NEI examples into specific construction families based on how the insufficient evidence was generated. The taxonomy distinguishes between:

Shortcut-prone constructions: Examples designed to expose format, topic, or position shortcuts.
- Placeholder: Fixed or empty evidence markers.
- Random Irrelevant: Evidence from unrelated topics.
- Position-biased: Non-rationale evidence from predictable document locations.
Semantically related insufficient evidence (Hard NEI): Examples where evidence is topically related but logically incomplete.
- BM25 near-miss: High lexical overlap but insufficient support.
- Cited non-rationale: Evidence from cited documents that is not the actual rationale.
- Same-document non-rationale: Non-rationale evidence from the same source document.
- Missing-hop: Multi-hop evidence with a required fact removed.

2. Diagnostic Workflow

The protocol follows five stages:

Construct: Assign a construction family ( $z$ ) and group ID ( $g$ ) to each example. The model is never exposed to $z$ ; it is used solely for auditing and stratified reporting.
Audit: Measure shortcut features (e.g., length, overlap, coverage) to identify surface-level cues that models might exploit.
Validate: Use human adjudication to separate "candidate" hard NEI from examples that are actually supported, refuted, or ambiguous. This creates a "human-adjudicated hard-NEI" subset.
Stress-test: Evaluate models trained on specific construction families against other families to test transferability.
Report: Release metrics stratified by construction family rather than aggregate scores.

3. Experimental Setup

The protocol is instantiated primarily on SciFact (scientific claim verification), with FEVER and HoVer serving as bounded external controls.

Models: Primary verifier is DeBERTa-v3-base, with RoBERTa and SciBERT as secondary backbones.
Training Regimes: Models are trained on single construction families (e.g., only placeholders) or mixed regimes (easy, hard, or balanced mixtures).
Evaluation: Uses construction-shift matrices (train on Family A, test on Family B) and human-adjudicated hard-NEI recall.

Key Results

1. Transfer Failure of Shortcut-Prone Training

Models trained on "easy" NEI constructions (e.g., placeholders) achieve perfect NEI-F1 on matched test sets but collapse to 0.000 NEI-F1 when evaluated on "hard" constructions (BM25 near-miss or cited non-rationale).

This failure is not a calibration artifact; probability mass shifts entirely to SUPPORT or REFUTE.
The collapse is stable across different model architectures (DeBERTa, RoBERTa, SciBERT) and random seeds.

2. Human-Adjudicated Hard NEI Exposes Model Weakness

On the human-validated hard-NEI subset (54 examples), models trained on placeholders or position-biased data achieve 0.000 recall. They overwhelmingly predict SUPPORT (false positives) with high confidence.

Even "random irrelevant" training, which is less extreme than placeholders, fails to generalize to hard insufficiency (0.216 recall).
Only models trained on hard constructions (BM25/cited) show partial recovery, but they still do not achieve perfect transfer.

3. Mixed Training Does Not Eliminate Construction Sensitivity

Training on mixed construction regimes (combining easy and hard examples) improves hard-NEI recall compared to single-family training but does not eliminate the performance gap.

Models trained on "balanced" mixtures still exhibit construction-stratified profiles, performing significantly better on easy conditions than on hard conditions.
This indicates that mixed training is a stronger stress test but not a replacement for construction-aware reporting.

4. Impact on Non-NEI Labels

Fixed-claim diagnostics reveal that the choice of NEI construction affects the model's confidence in SUPPORT and REFUTE labels. When reference evidence is swapped with insufficient evidence, models trained on easy NEI fail to reduce their confidence in the original label, whereas models trained on hard NEI show a probability drop. This suggests construction artifacts affect the entire three-way task, not just the NEI corner.

Contributions

Recasting NEI: The paper reframes NEI not as a single negative label but as a construction-sensitive evidence condition.
NEI-CAP Protocol: Introduces a diagnostic protocol that explicitly tracks construction families, audits shortcut features, and validates hard cases via human adjudication.
Empirical Evidence of Failure: Demonstrates through SciFact, FEVER, and HoVer experiments that shortcut-prone training fails to transfer to semantically related insufficient evidence, and that aggregate NEI-F1 scores can hide these failures.

Significance and Claims

The paper claims that aggregate NEI-F1 scores are insufficient for evaluating fact verification systems because they can hide whether a model has learned to recognize genuine evidence insufficiency or merely the artifacts of how NEI examples were constructed.

Modest Scope: The authors explicitly state that NEI-CAP is a diagnostic protocol, not a complete solution to evidence-insufficiency reasoning. It identifies construction sensitivity and provides audit tools but does not guarantee general-purpose sufficiency reasoning.
Reporting Discipline: The primary contribution is a shift in reporting discipline: every NEI result should be accompanied by the construction family that produced it.
Limitations: The strongest evidence is derived from SciFact-style scientific verification. FEVER and HoVer are used as bounded controls and are not human-validated hard-NEI benchmarks in this context. The paper acknowledges that different domains may require additional construction families and that human validation reduces but does not eliminate label validity risks.

In conclusion, the paper argues that the construction of insufficient evidence must be treated as a core part of the evaluation protocol, alongside labels and metrics, to ensure that fact verification systems are learning the intended capabilities rather than dataset-specific shortcuts.