💬 NLP

ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations

The paper introduces ReFACT, a benchmark derived from Reddit's r/AskScience that reveals Large Language Models suffer from a fundamental semantic grounding deficit causing "salient distractor" errors and struggle with comparative judgment, thereby challenging the reliability of LLM-as-Judge paradigms for scientific factuality.

Original authors: Yindong Wang, Martin Preiß, Margarita Bugueño, Jan Vincent Hoffbauer, Abdullatif Ghajar, Tolga Buz, Gerard de Melo

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yindong Wang, Martin Preiß, Margarita Bugueño, Jan Vincent Hoffbauer, Abdullatif Ghajar, Tolga Buz, Gerard de Melo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot librarian named "LLM" (Large Language Model). This robot has read almost everything on the internet and can write stories, explain science, and answer questions with incredible fluency. It sounds like a genius.

But here's the catch: The robot is a confident liar.

When it doesn't know the answer, it doesn't say, "I don't know." Instead, it makes up a story that sounds perfectly plausible to a normal person, but is actually completely wrong. In the scientific world, we call this "confabulation." It's like the robot is daydreaming while speaking, mixing up facts with fiction so smoothly that you can't tell the difference.

The paper you're asking about, ReFACT, is a new "exam" designed to see if these robots can catch their own lies (or the lies of other robots) when it comes to science.

Here is the breakdown of what they found, using some simple analogies:

1. The Test: "Spot the Fake"

The researchers went to a popular online science forum (r/AskScience) where real experts answer questions. They took 1,001 of these real, correct answers and secretly "poisoned" them.

The Poisoning: They changed one tiny word or flipped a sentence.
- Real: "Your DNA gets copied."
- Poisoned: "Your RNA gets copied." (This is a huge scientific error, but it sounds very similar).
The Goal: They asked various AI models to look at the poisoned answer and say: "Is this fake? Where exactly is the lie? And can you fix it?"

2. The Big Surprise: The "Distractor" Problem

The researchers expected the robots to fail because science is hard. But they found something weirder.

The Analogy: Imagine you are playing a game of "Find the Fake Bill." You are handed a stack of $100 bills. One of them is a fake.

What you expect the robot to do: Look closely at the ink, the paper texture, and the serial numbers to find the fake.
What the robot actually does: It ignores the fake bill entirely. Instead, it points at a real bill in the stack and says, "This one is fake!" Why? Because that real bill has a shiny spot or a specific color that catches its eye.

The Finding:
61% of the time, when the AI tried to find the error, it pointed at a word that had nothing to do with the actual mistake.

If the error was changing "DNA" to "RNA," the AI might point at the word "cells" or "replicate" just because those words sound "science-y."
It's like a student taking a math test who doesn't know the answer, so they just circle the longest number on the page because it looks like the answer.

The Scary Part: This happened even with the biggest, most powerful robots (70 billion "brain cells" or parameters). Making the robot bigger didn't fix this. It just made the robot better at picking different shiny, science-sounding words to point at.

3. The "Side-by-Side" Paradox

Usually, we think comparing two things is easier than judging one thing alone.

Example: It's easier to spot which of two apples is rotten if you hold them side-by-side than if you look at just one apple in the dark.

The Finding:
The researchers tested this with the AI. They gave the AI two answers (one real, one fake) and asked, "Which one is the lie?"

Result: The AI got worse at this!
When the AI had to judge a single answer, it was okay. But when it had to compare two, it got confused. It seemed to think, "Well, both of these sound so smart and confident, I can't tell which one is lying!"
This is a huge problem because many people currently use AI to judge the quality of other AI's work. If the judge is confused by side-by-side comparisons, the whole system is unreliable.

4. The "Fix-It" Failure

Finally, they asked the robots to actually fix the lie.

The Result: Even the best robot (GPT-4o) could only fix the error correctly about 28% of the time.
It's like asking a mechanic who can't identify the broken part to also replace it. They just guess, and usually, they guess wrong.

The Bottom Line

This paper is a wake-up call.

Scaling isn't the cure: Just making AI models bigger and more expensive doesn't make them understand the truth. They are still "hallucinating" (confabulating) and getting distracted by shiny words.
AI Judges are flawed: We cannot blindly trust AI to fact-check other AI, especially in science. They are too easily fooled by their own confidence.
We need a new approach: The researchers suggest we need to teach these robots to stop looking at "shiny words" and start actually understanding the meaning of what they are reading.

In short: Our current AI robots are like confident tour guides who make up stories about history. They sound great, they know the right vocabulary, but if you ask them to point out a lie in their own story, they will likely point at the wrong thing and insist they are right. We need to be very careful about trusting them with scientific facts.

1. Problem Statement

Large Language Models (LLMs) are prone to scientific confabulation: generating text that is fluent, coherent, and contextually plausible but factually incorrect. Unlike obvious hallucinations, scientific confabulations often require expert domain knowledge to detect (e.g., confusing DNA replication mechanisms with RNA). Current evaluation benchmarks suffer from several limitations:

Synthetic Data Bias: Many existing benchmarks rely on LLM-generated errors, which lack the nuance of real-world scientific discourse.
Lack of Granularity: Most benchmarks focus on binary classification (True/False) rather than fine-grained span-level localization (identifying exactly where the error is) or correction.
Evaluation Reliability: The "LLM-as-Judge" paradigm is increasingly used to evaluate other models, but its reliability in detecting subtle scientific errors remains unproven.

2. Methodology: ReFACT Benchmark Construction

The authors introduce ReFACT (Reddit False And Correct Texts), a benchmark derived from authentic human-authored scientific discourse rather than synthetic generation.

Data Source: The dataset is built from r/AskScience, a highly moderated Reddit community with over 23 million members. The authors selected high-quality, top-rated question-answer pairs (10,282 initial pairs filtered to 1,001 final samples).
Confabulation Generation Pipeline:
1. Fact Extraction: Salient facts are extracted from original answers.
2. Transformation: Two strategies are used to create "confabulated" answers:
  - Negation: Logically reversing a fact (e.g., "there is a chance" $\to$ "there is no chance").
  - Entity Replacement: Substituting a specific scientific term with a plausible but incorrect alternative (e.g., "DNA" $\to$ "RNA"). An NLI (Natural Language Inference) classifier ensures the replacement is a distinct semantic error, not a synonym.
3. Human Verification: Three annotators verify that transformed answers are coherent, contextually relevant, and factually incorrect. Only samples with high inter-annotator agreement (72.56%) are retained.
Dataset Statistics: The final dataset contains 1,001 samples across 10 scientific domains, with fine-grained span-level annotations for error localization and error types.

3. Evaluation Framework

The benchmark evaluates LLMs on three distinct tasks:

Confabulation Judgment:
- Independent: Determine if a single answer contains a confabulation.
- Comparative: Given two answers, identify which one is factually correct.
Span Localization: Identify the exact character span containing the error (measured via Intersection-over-Union, IoU $\ge$ 0.5).
Correction: Generate the correct entity to replace the corrupted one (measured via Exact Match).

4. Key Results & Findings

The authors evaluated 9 state-of-the-art models (including Gemma-3, Llama-3, and GPT-4o) and discovered two critical failure modes:

A. The Salient Distractor Phenomenon (Scale-Invariant Failure)

Observation: When models fail to localize errors, they do not merely miss the boundary; they confidently point to semantically unrelated but contextually salient terms.
Data: 61% of incorrect span predictions are semantically unrelated to the actual error (e.g., highlighting "RNA" when the error was in a different sentence, or highlighting generic scientific terms).
Scale Independence: This failure mode persists across all model sizes, from 1B to 70B parameters. Larger models do not fix the issue; they simply become better at selecting "scientific-sounding" distractors.
Implication: Scaling alone cannot resolve the fundamental semantic grounding deficit in LLMs.

B. The Comparative Judgment Paradox

Observation: Contrary to the assumption that comparing two options is easier, Comparative Judgment is significantly harder than Independent Detection.
Data: GPT-4o's F1 score dropped from 0.67 (Independent) to 0.53 (Comparative). Llama-3.3-70B dropped from 0.73 to 0.39.
Implication: When presented with two plausible but subtly different answers, models struggle to discern the factual error, likely due to shallow comparative heuristics or overconfidence in surface-level coherence.

C. Task Performance

Localization: Models perform significantly better on negation localization (syntactic cues) than entity replacement (semantic understanding). GPT-4o achieved 0.66 accuracy on negation but only 0.47 on entity replacement.
Correction: This is the most difficult task. Even GPT-4o achieved only 0.28 accuracy in correcting entities.

5. Significance and Contributions

New Benchmark: ReFACT is the first benchmark to provide human-verified, span-level annotations for scientific confabulation, moving beyond binary classification.
Challenge to "LLM-as-Judge": The results directly challenge the reliability of using LLMs (even GPT-4o) as evaluators for scientific factuality, showing they often fail to detect errors when comparing outputs side-by-side.
Fundamental Limitation: The paper provides empirical evidence that current LLMs rely on surface-level saliency heuristics rather than true factual reasoning. The "Salient Distractor" failure mode suggests that increasing parameter count does not inherently improve scientific factuality.
Future Directions: The authors suggest that future improvements require contrastive training with hard negatives, hierarchical localization objectives, and explicit entity grounding mechanisms rather than simple scaling.

6. Conclusion

ReFACT reveals that while LLMs are fluent, they lack robust semantic grounding for scientific facts. The dominant failure mode involves fixating on plausible but irrelevant terms, a problem that persists regardless of model size. These findings necessitate a re-evaluation of how scientific AI is developed and, crucially, how it is evaluated.