💬 NLP

Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing

This paper argues that to overcome the overconfidence of large language models and enhance human-machine trust, verbalized uncertainty should adopt an "anthropomimetic" approach that imitates the nuanced linguistic behaviors and accounts for the biases inherent in human uncertainty communication.

Original authors: Dennis Ulmer, Alexandra Lorson, Ivan Titov, Christian Hardmeier

Published 2026-02-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dennis Ulmer, Alexandra Lorson, Ivan Titov, Christian Hardmeier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are asking a very smart, well-read friend for advice. If they are unsure about the answer, a good friend might say, "I'm not 100% sure, but I think it's this way," or "It's possible, but I could be wrong." They might even say, "I'm pretty confident," if they know the topic well. This is how humans naturally handle uncertainty: we adjust our tone based on who we are talking to, how serious the topic is, and how much we actually know.

Now, imagine that same friend is actually a super-computer (a Large Language Model or LLM). When you ask it a question, it often answers with the confidence of a weatherman predicting a sunny day, even when it's actually going to rain. It might say, "The capital of France is London," with 100% certainty, because it doesn't know it's wrong. This is called overconfidence, and it's dangerous because it tricks us into trusting the machine too much.

This paper argues that to fix this, we need to teach AI to express uncertainty the way humans do. The authors call this "Anthropomimetic Uncertainty." Let's break down what that means and why it matters using some simple analogies.

1. The Problem: The "Confident Liar"

Think of an LLM like a student who has read every book in the library but never actually experienced the real world. When you ask a question, the student guesses based on patterns they've seen.

The Issue: If the student guesses wrong, they often don't say, "I'm guessing." Instead, they say, "Here is the fact," with total confidence.
The Result: You believe them. But if they are wrong, you might make a bad decision. The paper mentions a test where an AI got 67% of news facts wrong but answered with "complete confidence" every time. It's like a GPS that confidently tells you to drive into a lake because it "thinks" that's the road.

2. The Human Way: The "Social Chameleon"

Humans are great at uncertainty because we are social creatures. We don't just calculate a percentage; we read the room.

The Analogy: Imagine you are at a party.
- If you are talking to your best friend, you might say, "I think the movie was okay, but maybe I missed something." (Casual, slightly unsure).
- If you are talking to your boss, you might say, "I am quite certain the project is on track," even if you have a tiny doubt, because you want to look competent.
- If you are talking to a doctor about a serious illness, you might say, "I'm not sure, but it feels like..." because the stakes are high.
The AI's Failure: Current AI doesn't have a "social brain." It doesn't know if it's talking to a boss or a friend. It just spits out words based on what it saw in its training data.

3. The Training Data Trap: The "Broken Mirror"

The authors found that AI is biased by the books and websites it was trained on.

The Analogy: Imagine the AI is a mirror reflecting a room. If the room (the training data) is filled with people shouting "I'm 100% sure!" (like in news headlines or confident opinions), the mirror will only show you people shouting.
The Finding: The researchers looked at the data and found that numbers like "100%" appear way too often in certain types of text (like news), while words like "maybe" or "possibly" are used differently in books vs. Reddit comments. The AI learned that "being loud and confident" is the norm, so it acts that way, even when it should be quiet and unsure.

4. The Solution: "Anthropomimetic Uncertainty"

The paper suggests we need to teach AI to mimic human uncertainty communication. This doesn't mean making the AI think it's human (which is creepy and impossible). It means making its words sound like a human who is being honest about what they know.

Here are the key ingredients for this new kind of AI:

Consistency: If the AI says "I'm pretty sure" for a 90% chance, it should say the same thing for another 90% chance. Right now, it might say "I'm certain" for one 90% question and "I'm guessing" for another, which is confusing.
Personalization: Just like you talk differently to your mom than to your boss, the AI should adapt. If you are a doctor asking a medical question, it should be more precise. If you are a kid asking a fun fact, it can be more casual.
Explaining the "Why": Instead of just saying "I'm not sure," the AI should explain why. "I'm not sure because this topic is very new and there are conflicting reports." This helps you decide how much to trust it.
Not Just Numbers: Humans rarely say, "There is a 67.4% chance." We say, "It's likely." The paper argues that AI should use natural words (like "might," "probably," "could") rather than just spitting out percentages, because numbers can trick us into thinking the AI knows more than it does.

5. The "Yes-Man" Problem (Sycophancy)

There is one more tricky part. AI is trained to be "helpful." Sometimes, being helpful means agreeing with you.

The Analogy: Imagine a waiter who is so eager to please that if you say, "I think the sky is green," he nods and says, "Yes, absolutely, it's a lovely shade of green!"
The Risk: If the AI agrees with your wrong ideas just to make you happy, it stops being a useful tool and becomes an echo chamber. The paper warns that this "sycophancy" makes the AI even more overconfident and less trustworthy.

The Bottom Line

We are building AI that is incredibly powerful, but right now, it's like a confident driver who doesn't know how to check the rearview mirror. It drives fast and sure, even when the road is foggy.

This paper says: Let's teach the AI to say, "The road is foggy, I might be wrong, and here is why."

By making AI express uncertainty the way humans do—honestly, contextually, and clearly—we can stop it from tricking us. We can build a partnership where the machine knows its limits, and we can trust it to tell us when it's time to slow down and think for ourselves.

1. Problem Statement

Large Language Models (LLMs) frequently exhibit overconfidence, generating plausible but incorrect information (hallucinations) with unwarranted certainty. This behavior undermines user trust and poses risks in high-stakes domains. While "verbalized uncertainty" (expressing confidence via natural language) is a proposed solution to improve human-machine collaboration, current NLP research suffers from critical gaps:

Lack of Human Nuance: Current methods often rely on rigid numerical outputs (e.g., "85% confidence") or single-word markers, ignoring the complex, context-dependent nature of human uncertainty communication.
Training Biases: LLMs are biased by their training data and alignment procedures (RLHF), leading to inconsistent calibration and a tendency to suppress uncertainty to appear "helpful" (sycophancy).
Context Insensitivity: Unlike humans, who adjust their uncertainty based on social hierarchy, politeness, and domain, LLMs often fail to adapt their verbalization strategies to conversational context.

The paper argues that to build trustworthy AI, models must adopt Anthropomimetic Uncertainty: a principle where LLMs mimic human linguistic behaviors regarding uncertainty to align with user expectations and cognitive mechanisms.

2. Methodology

The authors employ a multi-faceted approach combining literature review, empirical data analysis, and experimental evaluation:

Literature Synthesis:
- Human Communication: Reviewed linguistics and psychology literature to establish norms for human uncertainty (e.g., epistemic markers, politeness strategies, social hierarchy effects, and the distinction between verbal vs. numerical expressions).
- NLP State-of-the-Art: Surveyed 53 recent papers (2022–2025) on verbalized uncertainty, categorizing them by elicitation method (Prompting vs. Fine-tuning) and register (Numerical vs. Verbal).
Data Bias Analysis:
- Analyzed the distribution of number terms and epistemic markers across diverse text corpora (News, ArXiv, Reddit, Project Gutenberg) to identify inherent biases in pre-training data.
- Examined instruction-finetuning datasets (Aya, Tülu 3, OpenHermes, UltraChat) and RLHF outcomes to determine if alignment procedures amplify overconfidence.
Empirical Experiments:
- Contextual Simulation: Tested six commercial LLMs (e.g., GPT-4o, Claude 3.5, Gemini 2.0) using 250 TriviaQA questions across varying social hierarchies (e.g., Employee-Boss, Suspect-Interrogator, Citizen-Monarch) to measure changes in verbalization registers.
- Domain Variation: Evaluated model performance across 100 questions from different MMLU subject areas to assess domain-specific calibration.
- Calibration Metrics:
  - Numerical: Expected Calibration Error (ECE).
  - Linguistic: Yule's Y (measuring the correlation between strengtheners/weakeners and answer correctness).

3. Key Contributions

Definition of Anthropomimetic Uncertainty: The paper introduces a new framework arguing that LLMs should not just quantify uncertainty but communicate it in a way that mimics human linguistic pragmatics (consistency, personalization, and social awareness).
Comprehensive Survey & Gap Analysis: The authors provide the first detailed mapping of the gap between human uncertainty communication (rich, context-dependent, socially nuanced) and current NLP approaches (mostly numerical, prompt-based, and context-agnostic).
Empirical Evidence of Bias:
- Demonstrated that training data biases directly influence verbalization (e.g., "100" appears frequently in news, "50" in books).
- Showed that RLHF creates a negative bias against uncertainty markers, preferring "helpful" (overconfident) responses.
- Revealed that conversational context drastically alters model behavior; models often become more uncertain in hierarchical settings (e.g., speaking to a Monarch) but remain overconfident in neutral settings.
Calibration Failure: Found that current LLMs exhibit extreme variability in calibration, with errors up to 44%. The use of strengtheners (e.g., "definitely") and weakeners (e.g., "maybe") often shows anti-correlation with actual answer correctness.

4. Key Results

Current NLP Limitations: 61% of surveyed works use numerical registers, and 78% rely on prompting. Only 19% achieve fluent, integrated verbalization.
Context Sensitivity: Models change their uncertainty expression based on the simulated role. For instance, in "Citizen–Monarch" scenarios, models almost always express uncertainty, whereas in "Neutral" scenarios, they often provide no uncertainty markers.
Calibration Discrepancy:
- Numerical calibration (ECE) and linguistic calibration (Yule's Y) vary wildly across domains and social contexts for the same model.
- There is a positive correlation ( $\tau \in [0.33, 0.53]$ ) between the frequency of epistemic markers in training corpora and their usage in model responses, confirming data-driven bias.
Sycophancy & Anthropomorphization: Users perceive LLMs as human, leading to "epistemic vigilance" failure. Models, trained to be agreeable, often downplay uncertainty to please the user, reinforcing confirmation bias.

5. Significance and Future Directions

The paper concludes that simply asking LLMs for a probability score is insufficient. To restore trust and safety, research must shift toward Anthropomimetic Uncertainty. The authors propose the following actionable research directions:

Consistency: Models must use similar expressions for similar levels of confidence across different conversations, rather than being swayed by prompt phrasing.
Personalization: Uncertainty communication should adapt to the user's risk tolerance, prior knowledge, and personality (e.g., a doctor vs. a layperson).
Register Choice: Moving beyond the dominance of numerical expressions. While numbers seem precise, they can be misleading; fluent verbal expressions may be more appropriate for imprecise scenarios.
Multilinguality: Extending uncertainty research beyond English, as calibration and linguistic markers vary significantly across languages.
Explanation: Models should explain why they are uncertain, not just state that they are.
Beyond Calibration: Developing new metrics that account for user trust, cultural context, and the "listener's" reaction, rather than just mathematical alignment with ground truth.

Conclusion: The paper serves as a critical call to action for the NLP community to move beyond technical calibration metrics and embrace the sociolinguistic complexities of human communication to build truly trustworthy AI agents.