💬 NLP

Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild

This paper introduces a dual-axis framework (JudgeGPT and RogueGPT) and employs Structural Causal Models to demonstrate that human susceptibility to foundation model hallucinations is driven more by familiarity with fake news and the "fluency trap" of high-quality AI text than by political orientation, suggesting that effective interventions should target cognitive source monitoring rather than demographic segmentation.

Original authors: Alexander Loth, Martin Kappes, Marc-Oliver Pahl

Published 2026-02-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Loth, Martin Kappes, Marc-Oliver Pahl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🕵️‍♂️ The Big Picture: The "Too Good to Be True" Trap

Imagine you are walking through a forest. For thousands of years, you've learned to spot fake berries by looking at their color and texture. But now, someone has invented a perfect plastic berry. It looks exactly like a real one, smells like one, and even feels like one. You can't tell the difference just by looking.

This is exactly what this paper is about. Foundation Models (like the AI behind ChatGPT) have become so good at writing that they are creating "plastic berries" (fake news or AI text) that look indistinguishable from "real berries" (human writing).

The authors, Alexander, Martin, and Marc-Oliver, built a digital playground to study how humans react to these plastic berries. They wanted to answer: Can we tell them apart? And if we can't, why?

🛠️ The Tools: The "Judge" and the "Rogue"

To study this, they built two digital tools:

JudgeGPT (The Referee): This is a website where real people read short news snippets and play a game. They have to answer two questions:
- Is this story true or fake? (Authenticity)
- Did a human or a robot write this? (Source)
RogueGPT (The Magician): This is the engine that creates the stories. It uses different "magicians" (AI models like GPT-4, Llama-2, etc.) to write the news snippets.

They put 154 people through this game, asking them to judge 918 different stories.

🔍 The Big Discoveries

Here are the three main things they found, explained simply:

1. The "Fake News Familiarity" Shield 🛡️

The Myth: Many people think that if you are very political or have strong opinions, you are easily tricked by fake news.
The Reality: The study found that politics doesn't matter much. Whether you lean left or right, your ability to spot a robot didn't change much.

The Real Hero: The people who were best at spotting fakes were the ones who already knew a lot about fake news.

The Analogy: Think of fake news exposure like vaccination. If you've seen a lot of "bad actors" before, your brain has built up antibodies. You've seen the tricks, so you spot them faster. The more you've been exposed to fake news, the better you get at spotting it. It's like "adversarial training" for your brain.

2. The "Fluency Trap" (The GPT-4 Problem) 🤖

The Finding: The AI model GPT-4 was so good at writing that humans couldn't tell it apart from a human at all.

The Score: On a scale where 0 is "Definitely Human" and 1 is "Definitely Robot," GPT-4 scored 0.20. Other models scored around 0.50 or higher.
The Trap: GPT-4 writes so smoothly and perfectly that our brains stop checking the source. We fall into a "Fluency Trap."
The Analogy: Imagine a singer with a perfect voice. You are so impressed by how smooth the singing is that you forget to ask, "Is this a real person or a recording?" The smoothness tricks your brain into thinking it's real, even when it's not.

3. We Get Better with Practice (The "Pre-Bunking" Effect) 📈

The Finding: As people played the game more, they got better at spotting the AI.

The Analogy: This is like learning to ride a bike. At first, you wobble and fall. But after a few tries, your brain learns the pattern. The study suggests that if we expose people to AI-generated text before they encounter real fake news (a concept called "Pre-bunking"), it acts like a vaccine. It teaches their brains what to look for, making them immune to future attacks.

💡 What Should We Do About It?

The authors say we can't just rely on people to "be smarter" or "check the source" because the AI is too good at hiding. Instead, we need a three-part plan:

Digital Watermarks (The "Hologram"): Just like money has a hologram to prove it's real, AI text needs a hidden, unbreakable digital tag (like C2PA) that proves it came from a robot. We can't trust our eyes; we need to trust the code.
Train the Brain, Not the Fact-Checker: Instead of teaching people to check who wrote a story (which is impossible now), teach them to check the logic. Does the story make sense? Are the facts connected? This targets the "Fluency Trap" directly.
Adversarial Training: Platforms should occasionally show users "fake" AI content and say, "Hey, this was made by a robot." This keeps our brains sharp, like a fire drill.

🏁 The Bottom Line

The paper concludes that AI is winning the "sneakiness" contest. It writes so well that we can't tell it apart from humans.

However, there is hope. If we treat our brains like muscles that can be trained, and if we use technology to tag AI content automatically, we can build a safer internet. The key isn't to stop the AI; it's to upgrade our own "source monitoring" software so we don't get fooled by the perfect plastic berries.

1. Problem Statement

As Foundation Models (FMs) and Large Language Models (LLMs) achieve human-level fluency, distinguishing synthetic content from organic human writing has become a critical challenge for Trustworthy Web Intelligence. While technical detection methods exist, there is a gap in understanding the causal mechanisms behind human susceptibility to AI-generated misinformation. Specifically, it remains unclear:

Which demographic or experiential factors causally influence a user's ability to detect AI text?
Do different foundation models vary significantly in their "human-likeness" and ability to bypass human source monitoring?
Can exposure to AI content serve as "adversarial training" (pre-bunking) to improve human detection over time?

2. Methodology

The authors developed a dual-axis framework combining two open-source platforms to conduct empirical studies and causal analysis:

A. Data Collection Platforms

JudgeGPT: An interactive, multilingual (English, German, French, Spanish) web interface where users evaluate news fragments. It captures two continuous dimensions:
1. Perceived Authenticity (LegitFakeScore): How believable the content is.
2. Source Attribution (HumanMachineScore): Whether the content is perceived as human-written or machine-generated.
RogueGPT: A generative engine that produces diverse news fragments using five different foundation models:
- OpenAI: GPT-4, GPT-4o, GPT-3.5-turbo
- Meta: Llama-2-13b-chat
- Google: Gemma-7b-it
- Mistral AI: Mistral-7b-instruct
- Microsoft: Phi-3-mini-4k-instruct

B. Experimental Design & Causal Framework

Dataset: 918 valid evaluations from 154 participants across 2,301 news fragments.
Structural Causal Models (SCMs): The authors employ Directed Acyclic Graphs (DAGs) to move beyond simple correlation. They formulate testable causal hypotheses where:
- Treatment ( $T$ ): Model Architecture (e.g., GPT-4 vs. Llama-2).
- Mediator ( $M$ ): Textual features (Surface Fluency, Discourse Coherence, Causal Consistency).
- Confounders ( $C$ ): User demographics (Political orientation, Fake news familiarity, Age, Education).
- Outcome ( $Y$ ): Source Attribution accuracy.
Analysis: The study uses correlation analysis to identify patterns and SCMs to propose causal pathways, acknowledging that observational data requires validation via randomized experiments.

3. Key Contributions

JudgeGPT & RogueGPT Platforms: The first open-source tools enabling continuous, dual-axis (authenticity + attribution) human evaluation of foundation model outputs across multiple languages and models.
Causal Analysis of Susceptibility: Empirical evidence identifying "fake news familiarity" as a significant predictor of detection accuracy, while debunking the strong influence of political orientation.
Model-Specific Vulnerability Quantification: Quantitative evidence showing that GPT-4 produces text significantly more indistinguishable from human writing than other tested models, creating a "fluency trap."

4. Key Results

A. Causal Factors in Susceptibility

Fake News Familiarity: Shows a moderate positive correlation with detection accuracy ( $r = 0.35$ for source attribution). This suggests that prior exposure to misinformation acts as adversarial training, improving human discriminators.
Political Orientation: Shows a negligible association with detection performance ( $r = -0.10$ ). Contrary to partisan narratives, political bias is not the primary driver of susceptibility; analytical thinking and experience are more critical.

B. The "Fluency Trap" & Model Comparison

GPT-4 Dominance: GPT-4 achieved an average HumanMachineScore of 0.20 (where lower scores indicate "more human-like"), significantly outperforming all other models (the next best, Mistral-7b, scored 0.44).
Indistinguishability: Statistical $t$ -tests revealed no significant mean difference between human-written and AI-generated content at the population level for either authenticity or source attribution. Humans generally cannot reliably distinguish high-quality FM outputs from human text.

C. Learning Effects & Cognitive Profiles

Pre-bunking Potential: Participants showed a downward trend in HumanMachineScore (improving detection) early in the session, suggesting that exposure acts as cognitive inoculation.
Judging Profiles: Clustering analysis identified three distinct user types:
- Skeptical Judges: Consistently rate content as fake/machine.
- Trusting Judges: Consistently rate content as legitimate/human (vulnerable to fluency).
- Variable Judges: Inconsistent patterns.
Cognitive Fatigue: While AI source detection remained stable, accuracy in detecting "fake news" (legitimacy) degraded over time due to cognitive resource depletion (System 2 fatigue).

5. Significance and Implications

Theoretical Impact

Erosion of Truth-Default: The study confirms that high fluency in FMs bypasses human "Source Monitoring" heuristics, leading to "validity illusions" where users assume AI outputs are accurate.
Shift from Demographics to Cognition: Interventions should target cognitive source monitoring and media literacy rather than demographic segmentation (e.g., political groups).

Policy and Technical Recommendations

Mandatory Provenance (C2PA): Since human detection of GPT-4 is failing, reliance on "user vigilance" is insufficient. Cryptographic watermarking and provenance standards are essential.
Cognitive Security Education: Training must shift from "checking sources" to "checking logic" (identifying causal inconsistencies and hallucinations) to counter the fluency trap.
Adversarial User Training: Platforms should implement "pre-bunking" interventions where users are exposed to synthetic content with immediate feedback to calibrate their Source Monitoring heuristics.
Model-Side Transparency:
- Stylistic Fingerprinting: Train models to retain detectable stylistic signatures.
- Uncertainty Surfacing: Models should use hedging language for low-confidence claims.
- RLHF for Detectability: Incorporate a "detectability reward" in Reinforcement Learning from Human Feedback to explicitly trade off deceptive capability against task performance.

Future Work

The authors propose a "Trustworthy Cycle": Human feedback $\to$ Causal Discovery $\to$ Model Improvement (Safety Classifiers/RLHF) $\to$ Re-evaluation. Future research will expand to more diverse populations, longitudinal studies on skill retention, and testing specific "causal flaws" in AI text to verify if humans can detect logical errors better than stylistic ones.