🤖 AI

AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized

Five experiments demonstrate that merely having access to AI advice, even when it is incorrect and accuracy is incentivized, drastically reduces people's willingness to admit uncertainty, leading them to answer more questions with inflated confidence but significantly lower accuracy.

Original authors: Chiara Marcoccia, Walter Quattrociocchi, Valerio Capraro

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chiara Marcoccia, Walter Quattrociocchi, Valerio Capraro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Art of Knowing When to Stop

Imagine you are playing a trivia game with a friend. Sometimes, you know the answer immediately. Other times, you're not sure, so you might say, "I don't know," or take a wild guess. In the world of science, this ability to pause and admit uncertainty is called "suspending judgment." It's a superpower that keeps us from believing things that aren't true. For a long time, humans have been the only ones with this superpower; if we didn't know something, we stayed quiet. But now, we have a new teammate: Artificial Intelligence (AI). These AI assistants are like incredibly fast, confident librarians who can answer almost any question instantly. The big question scientists are asking is: If we have a super-fast, super-confident friend who always has an answer ready, will we stop saying "I don't know"? Will we trust the AI so much that we forget how to pause and think for ourselves, even if the AI is wrong?

The Experiment: When the AI Lies

A team of researchers decided to test this by setting up a tricky game. They asked people 6 difficult questions about movies—things like the color of a turtle in an old cartoon or the specific car a character drove. These questions were chosen carefully because they are the kind of things that make AI "hallucinate," or make up confident-sounding but completely wrong answers. The researchers gave some people the option to ask an AI for help, while others had to answer on their own.

Here is the twist: The AI was almost always wrong. It was like having a friend who is very loud and very sure of themselves, but who is actually guessing the wrong answers. The researchers wanted to see if people would still say "I don't know" when they had the option to ask this unreliable friend, or if the mere presence of the AI would make them stop pausing and just start answering.

What They Found: The "Cognitive Surrender"

The results were surprising and a bit scary. When people had access to the AI, they almost completely stopped saying "I don't know." It was as if the AI's fluent, confident voice lowered the bar for what it took to feel ready to answer. Even though the AI was wrong most of the time, people who used it answered way more questions than those who didn't. But here's the catch: they got the answers right only about one-third as often as the people who didn't use the AI.

It's like a student who stops raising their hand to say "I'm not sure" because they have a cheat sheet that gives them answers. They fill out the whole test, but they get most of it wrong because they trusted the cheat sheet too much. The study found that having the AI available made people feel much more confident in their answers (their confidence nearly doubled!), even though their actual accuracy dropped significantly. They had essentially surrendered their own judgment to the machine.

Does Money Change the Game?

The researchers then wondered: What if getting the answer wrong actually cost you money? They set up a new round of the game where people could earn a small amount of money for correct answers and lose a small amount for wrong ones. They hoped this would make people think harder and maybe start saying "I don't know" again, even with the AI around.

The money did help a little. People asked the AI for help less often, and they got slightly more answers right. However, the money didn't fix the main problem. Even when there was a penalty for being wrong, people who had the AI still said "I don't know" far less often than people who didn't have the AI at all. The AI's pull was just too strong. It turned out that the AI didn't just change what people answered; it changed whether they felt they were allowed to answer in the first place.

The Takeaway

This research suggests that as AI becomes more common and starts giving us answers without us even asking, we might lose a crucial skill: the ability to recognize when we don't know something. The AI doesn't just fill in the blanks; it seems to convince us that we don't need to pause and think. The study shows that while we can be motivated to check our work if there's a cost to being wrong, the simple presence of a confident, fluent answer from a machine makes us much less likely to admit our own uncertainty. In a world full of AI, knowing when to say "I don't know" might be the most important skill of all.

Technical Summary: AI Advice Suppresses Willingness to Say "I Don't Know"

Problem Statement

Human judgment relies fundamentally on the metacognitive ability to recognize uncertainty and suspend judgment (i.e., say "I don't know") when knowledge is insufficient. This capacity protects against false beliefs and is a prerequisite for calibrated reasoning. However, Large Language Models (LLMs) are designed to provide fluent, instant answers to almost any query and are notoriously reluctant to admit uncertainty.

The central empirical question addressed is whether the mere availability of an AI-generated answer alters the human decision criterion for answering under uncertainty. Specifically, does access to AI advice cause individuals to override their own uncertainty and commit to an answer they would otherwise withhold? Previous literature has examined cognitive offloading, deskilling, and the erosion of agency, and the concept of "cognitive surrender" (Shaw & Nave, 2026) suggests people may defer to AI even when it is wrong. However, prior studies have not isolated the specific willingness to withhold judgment as a primary outcome, often confounding AI reliance with rational delegation to accurate tools.

Methodology

The authors conducted five experiments (total $N = 3,132$ ; four preregistered, one direct replication) to isolate the behavioral effect of AI availability from its accuracy.

Experimental Design & Stimuli:

Task: Participants answered six difficult questions regarding fine visual details in films (e.g., the color of a turtle in an obscure animated film, the vehicle in a specific movie scene).
AI Engineering: The questions were specifically selected to induce hallucinations in LLMs. The primary model used, Step 3.5 Flash, answered these questions incorrectly almost without exception. State-of-the-art models (GPT-5.5, Claude 4.6 Sonnet, Gemini 3.5 Flash) also failed on the most difficult items. This design ensured that following AI advice was irrational, allowing the researchers to measure a tendency to defer to AI output rather than sensible delegation.
Conditions:
- AI Access: Participants could either actively request AI advice (Studies 1a–3) or receive it automatically without choice (Study 4).
- Stakes: Monetary incentives were introduced in Studies 2–4. Participants earned \ $0.10 for correct answers, lost \$ 0.10 for incorrect answers, and received $0 for suspending judgment.
- Replication Controls: Study 1b used pre-generated incorrect AI responses to rule out system malfunctions. Study 3 increased incentive salience by reminding participants of stakes before every question and removed confidence elicitation to prevent priming guessing. Study 4 removed participant control, displaying AI advice automatically.

Measures:

Judgment Suspension: The primary dependent variable, defined as the proportion of questions where participants explicitly withheld an answer (e.g., "I don't know," blank response).
Secondary Measures: Answer correctness, self-reported confidence (Study 2), and frequency of AI advice requests.

Key Results

1. Suppression of Judgment Suspension
The mere availability of AI advice nearly eliminated participants' willingness to suspend judgment, even when the advice was systematically wrong.

Study 1a/1b: In the baseline (no AI), suspension rates were 36% and 44%, respectively. With AI access, these rates dropped to 6% and 3%.
Studies 2–4: This effect persisted across all subsequent studies. In Study 3, suspension dropped from 32% (no AI) to 2% (AI, no stakes) and from 41% (no AI) to 8% (AI, with stakes).
Automatic vs. Active: The effect held even when AI advice was presented automatically (Study 4), suggesting the mere presence of a fluent answer is sufficient to alter the decision to answer.

2. The Role of Monetary Incentives
Monetary incentives (accuracy rewards/penalties) had independent but limited effects:

On Suspension: Incentives modestly increased suspension rates (e.g., from 2% to 8% in the AI condition in Study 3), but suspension remained far below the levels observed when AI was unavailable. Contrary to the preregistered hypothesis, there was no significant interaction between AI availability and stakes; the two factors acted largely independently.
On Accuracy: Incentives significantly improved accuracy, but only when AI was present. In the absence of AI, incentives did not significantly change correctness. This suggests incentives motivated participants to override the AI advice rather than simply improving general performance.
On AI Reliance: Participants sought AI advice less frequently when stakes were present (e.g., 4.53 out of 6 questions vs. 5.27 in Study 3), indicating that financial consequences reduced reliance on the tool.

3. Confidence and Accuracy Divergence
Access to AI advice created a stark divergence between confidence and accuracy:

Confidence: AI access nearly doubled reported confidence (e.g., from ~30% to ~76% in Study 2 without stakes).
Accuracy: Participants answered more questions overall but were correct only about one-third as often as those without AI access (pooled correctness: ~9% with AI vs. ~27% without AI in no-stakes conditions).
Interaction: Incentives increased confidence only when AI was absent; when AI was present, confidence remained high regardless of stakes.

Key Contributions

Isolation of Metacognitive Shift: The study demonstrates that AI availability alters the metacognitive threshold for deciding whether to answer, not just the content of the answer. It reveals a "cognitive surrender" where the fluency of an AI response lowers the perceived cost of committing to a guess.
Decoupling Reliance from Accuracy: By engineering questions where AI is reliably wrong, the authors show that the reduction in judgment suspension is not a rational division of labor but a behavioral tendency to defer to plausible-sounding outputs.
Limits of Incentives: The research shows that while monetary stakes can partially restore independent judgment and reduce reliance on AI, they do not fully counteract the suppressive effect of AI availability on the willingness to say "I don't know."
Automatic vs. Active Influence: The effect persists even when AI advice is unsolicited and automatic, highlighting a risk for real-world scenarios where AI summaries and suggestions are increasingly integrated into user interfaces by default.

Significance and Claims

The authors argue that these findings suggest AI assistance influences not only the answers people give but also the fundamental decision of whether to answer under uncertainty. The "pull" to defer to a fluent answer appears strong enough to survive small monetary incentives.

The paper posits that this phenomenon can be understood through the lens of epistemia—the tendency to accept AI outputs for their surface plausibility and syntactic coherence rather than verification. Unlike human advisors, who typically underweight advice to protect independent judgment, participants in these studies over-weighted AI advice, converting potential correct answers into errors.

The authors conclude that as AI-generated answers become ubiquitous and unsolicited, the willingness to say "I don't know" may be among the first casualties of human-AI interaction. The significance lies in the observation that preserving human judgment may depend less on making AI more accurate and more on preserving people's readiness to recognize and act on the limits of their own knowledge. The study does not claim that AI eliminates the capacity for judgment, but rather that it drastically reduces the willingness to exercise it under uncertainty.