💬 NLP

When Verification Fails: How Compositionally Infeasible Claims Escape Rejection

This paper reveals that current scientific claim verification benchmarks fail to distinguish rigorous reasoning from a "salient-constraint" shortcut because they only test single-element perturbations, leading models to consistently over-accept compositionally infeasible claims where non-salient constraints are contradicted despite the salient one being supported.

Original authors: Muxin Liu, Delip Rao, Grace Kim, Chris Callison-Burch

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muxin Liu, Delip Rao, Grace Kim, Chris Callison-Burch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a bouncer at an exclusive, high-stakes club. The club's rule is strict: to get in, a guest must prove they meet every single requirement on the list. If they miss even one tiny detail, they are turned away. This is how science is supposed to work: a claim is only true if all the evidence supports it.

However, a new paper by Muxin Liu and colleagues reveals that our current "AI bouncers" (Large Language Models) are cheating. They aren't checking the whole list; they are just looking at the most obvious thing and letting people in if that one thing looks good.

Here is the breakdown of the problem, the experiment, and the solution, using simple analogies.

1. The Two Types of Bouncers

The paper compares two ways of checking a claim:

The Strict Bouncer (Closed-World Assumption): This bouncer reads the entire guest list. "Do you have a ticket? Yes. Are you over 21? Yes. Is your name on the VIP list? Yes." Only if all boxes are checked does the guest enter.
The Lazy Bouncer (Salient-Constraint Checking): This bouncer only looks at the most obvious thing. "Do you have a ticket?" (Yes). "Okay, come on in!" They ignore the age or the name because the ticket was the "salient" (most noticeable) part.

The Problem: For years, we tested AI bouncers with fake guests who had obvious flaws (like wearing a fake ticket). The AI correctly rejected them. But because the flaw was so obvious, the AI looked smart even if it was just using the "Lazy" method. We didn't realize they were skipping the rest of the list.

2. The "Trap" Test: The Invisible Flaw

To catch the Lazy Bouncers, the researchers built a special trap called a "Compositionally Infeasible Claim."

Imagine a guest says:

"I have a valid ticket, I am over 21, and I am a time-traveling alien from the year 3000."

The Salient Part: "I have a valid ticket." (True).
The Hidden Part: "I am a time-traveling alien." (False, but hidden in the middle of the sentence).

The Result:

The Strict Bouncer reads the whole sentence, sees the alien part, and says, "Nope, rejected."
The Lazy Bouncer sees the ticket, thinks, "Looks good," and says, "Welcome!"

The researchers created thousands of these "trap" claims using scientific data (clinical trials, charts, and tables). They found that almost every AI model, even the most advanced ones, fell for the trap. They accepted the "alien" claims because the "ticket" part looked correct.

3. Why Can't the AI Just "Think Harder"?

The researchers tried to fix this by giving the AI better instructions, like:

"Please check every single detail!" (Closed-World instructions).
"Break the claim down into small steps!" (Decomposition).

The Analogy: Imagine telling the Lazy Bouncer, "Hey, please check the whole list this time!" The bouncer nods and checks the list a bit more carefully, but they still miss the hidden alien part.

The paper shows that these instructions only shift the AI's threshold.

If you tell them to be stricter, they reject more people (including some who actually belong).
If you tell them to be looser, they let more people in (including the aliens).

The AI doesn't actually get better at connecting the dots; it just changes how cautious it is. The "reasoning bottleneck" is structural: the AI is good at finding the ticket, but bad at realizing that the ticket and the alien story don't fit together logically.

4. The "Self-Check" Experiment

The researchers also asked the AI to check its own work (like a student grading their own homework).

The Result: The AI was just as bad at catching its own mistakes as it was at catching others'. It would write a wrong answer, look at it, and say, "Yep, that looks right," because it only focused on the obvious parts of its own reasoning.

5. The Big Takeaway

The "Shortcut" is the Default:
AI models are trained on massive amounts of data where the "obvious" part usually tells the whole story. They learned that checking the "ticket" is usually enough. They never learned that sometimes, the "ticket" is real, but the "alien" part makes the whole claim false.

Current Benchmarks are Flawed:
The tests we use to grade AI are too easy. They only test if the AI can spot the "fake ticket." They don't test if the AI can spot the "fake alien."

The Future:
To make AI truly reliable for science and medicine, we can't just tell them to "be more careful." We need to fundamentally retrain them to understand that truth requires checking every single connection, not just the most obvious one. Until then, we have to be very careful about trusting AI when complex details matter.

Summary in One Sentence

AI models are currently like bouncers who only check your ID card and ignore your age; the researchers built a test where the ID is real but the age is fake, proving that even the smartest AIs are still "lazy" when it comes to checking the fine print.

1. Problem Statement

Scientific claim verification requires determining if a claim is entailed by evidence. The rigorous standard for this task is the Closed-World Assumption (CWA), which dictates that a claim is accepted if and only if every asserted constraint is positively supported by the evidence. If any constraint is unsupported or contradicted, the claim must be rejected.

However, the authors argue that existing Large Language Model (LLM) benchmarks fail to distinguish between models that strictly enforce CWA and those that rely on a simpler heuristic called Salient-Constraint Checking.

Salient-Constraint Checking: The model identifies the most prominent (salient) constraint in a claim, checks if it is supported, and makes a decision based solely on that. It ignores non-salient constraints that require compositional reasoning (combining multiple pieces of evidence).
The Gap: Existing benchmarks construct "infeasible" (false) claims by perturbing a single salient element (e.g., changing a number or reversing a trend). In these cases, both CWA and Salient-Constraint Checking yield the same result (rejection), making it impossible to detect if a model is actually performing rigorous verification or just using the shortcut.

2. Methodology

A. Theoretical Framework

The authors formalize verification standards:

CWA: Requires $\forall$ (for all) constraints to be supported.
OWA (Open-World Assumption): Requires $\exists$ (exists) a contradiction to reject; otherwise, accepts.
Salient-Constraint Checking ( $V_s$ ): A hybrid that applies CWA's rejection criterion but only to the single most salient constraint ( $c^*$ ).
Proposition: In the "single-constraint regime" of existing benchmarks, $V_s$ and CWA are behaviorally indistinguishable.

B. Data Generation: Compositionally Infeasible Claims

To expose the gap, the authors introduce Compositionally Infeasible Claims (Adversarial Negatives). These are claims where:

The salient constraint is supported by the evidence.
A non-salient constraint (requiring composition of multiple evidence units) is contradicted.
The claim is locally plausible but globally false.

Graph-Based Adversarial Construction:
The authors developed a pipeline to generate these claims from existing benchmarks (NLI4CT, SCITAB, SciVer):

Observation Layer: Extracts 5 directly retrievable facts from the evidence.
Context Layer: Combines observations into intermediate relationships.
Interpretation Layer: Combines context nodes into high-level conclusions.
Corruption: A single non-salient node in the Context or Interpretation layer is corrupted (e.g., scope swapping, overgeneralization). The corruption is propagated to generate a claim that preserves all observation-layer facts (keeping the salient constraint true) but violates the compositional logic.

C. Experimental Setup

Datasets: NLI4CT (clinical trials), SCITAB (scientific tables), and SciVer (charts).
Models: Evaluated 8 proprietary models (GPT-5, GPT-4o, Gemini, Claude families) and the Qwen3 family across various sizes.
Interventions: Tested prompting strategies to enforce CWA (Closed-World instructions, Decomposition + CWA) vs. OWA (Open-World instructions).
Trace Analysis: Used a rubric-based LLM judge to score reasoning traces on a scale of -2 (Salient-Checking) to +2 (Full CWA) to analyze how models reason, not just their final verdict.

3. Key Results

A. Performance Gap (The "Routing Failure")

Models that achieve high accuracy on standard benchmarks consistently fail on compositionally infeasible claims.

Over-Acceptance: Models accept adversarial negatives at significantly higher rates than standard negatives.
Magnitude: The accuracy drop on adversarial negatives ranges from ~5% to over 26% depending on the model and domain (e.g., Gemini-3.0-Flash dropped 26.7 points on SciVer).
Scaling: Increasing model size improves retrieval and direct falsification rejection but does not close the gap on compositional infeasibility. The failure persists even in the largest models (32B+).

B. Retrieval vs. Reasoning

An ablation study provided evidence nodes in different configurations:

No Graph: Raw evidence.
Graph-O: Isolated atomic observations (no structure).
Graph-O+C: Observations + Context (intermediate relationships).
Graph-All: Full graph including Interpretation (conclusions).
Finding: Providing isolated observations (Graph-O) did not improve performance. Providing context (Graph-O+C) helped, but the full graph (Graph-All) was required to reach ceiling performance. This confirms the bottleneck is compositional inference, not retrieval. Models can retrieve facts but cannot compose them to detect violations.

C. Strategy vs. Ability (ROC Curve Analysis)

By varying prompting strategies (OWA $\to$ Baseline $\to$ CWA), the authors mapped model behavior onto a Receiver Operating Characteristic (ROC) curve.

Shared Frontier: Different models and prompting strategies occupy different points on the same ROC curve.
Implication: The gap between model families reflects a difference in verification threshold (how strict they are by default), not a difference in underlying reasoning capability.
Limit of Prompting: Even with strict CWA prompting, models cannot perfectly distinguish feasible from compositionally infeasible claims. Stricter prompting reduces over-acceptance of false claims but increases the rejection of true claims (false positives), indicating a fundamental reasoning bottleneck that strategy guidance alone cannot overcome.

D. Trace Analysis

Reasoning traces revealed that models often terminate early at the salient constraint.

Models that successfully rejected adversarial claims showed higher "CWA scores" (checking non-salient constraints).
Models that failed showed early termination patterns similar to those on standard negatives.
Self-Verification: The same failure mode appeared when models verified their own reasoning, suggesting salient-constraint checking is a general verification tendency, not just a response to external evidence.

4. Key Contributions

Identification of a Structural Limitation: Demonstrated that existing benchmarks are insufficient because they cannot distinguish rigorous CWA verification from the simpler Salient-Constraint Checking shortcut.
New Benchmark Construction: Introduced a graph-based method to generate compositionally infeasible claims across text, tables, and charts, creating a robust test for compositional reasoning.
Diagnosis of the Bottleneck: Proved that the failure is a reasoning bottleneck (inability to perform $\forall$ -type compositional inference) rather than a lack of knowledge or retrieval capability.
Strategic vs. Capability Distinction: Showed that model differences are largely due to operating thresholds (strategy) rather than reasoning capacity, and that current prompting strategies cannot fully resolve the compositional inference gap.

5. Significance

Benchmark Validity: The paper challenges the validity of current claim verification benchmarks, suggesting that high accuracy scores may be inflated by models relying on shortcuts rather than genuine scientific reasoning.
Misinformation Risk: It highlights a critical vulnerability in AI systems: they may confidently accept scientifically false claims if the "obvious" part of the claim is true, even if the logical structure is flawed. This is a major risk for scientific misinformation.
Future Directions: The authors argue that simply prompting models to "check all constraints" is insufficient. Future work must address the training objective mismatch, where current training data rewards $\exists$ -type reasoning (finding any support) rather than the $\forall$ -type reasoning required for rigorous scientific verification. They suggest that post-training interventions (SFT/RL) on compositional data are necessary but currently yield limited results without addressing the fundamental training bias.