💬 NLP

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

This paper reveals that large language models exhibit overconfidence in incorrect answers when those answers are highly popular or frequently co-occur with the query, and demonstrates that incorporating knowledge popularity signals effectively mitigates this overconfidence while significantly improving confidence estimation accuracy.

Original authors: Shiyu Ni, Keping Bi, Jiafeng Guo, Xueqi Cheng

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiyu Ni, Keping Bi, Jiafeng Guo, Xueqi Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet hum of a server farm, a new kind of intelligence has emerged, one that can write poetry, solve equations, and answer questions with a fluency that feels almost human. These large language models are trained on vast oceans of text, learning to predict the next word in a sentence by recognizing patterns in human knowledge. But there is a catch: these machines do not truly "know" facts in the way we do. They do not have a memory of the world, only a statistical sense of which words usually appear together. When they are unsure, they often do not admit it. Instead, they frequently produce incorrect answers with absolute certainty, a phenomenon researchers call overconfidence. This is a critical problem for anyone relying on these tools for safety, medicine, or finance, because a confident lie is far more dangerous than a hesitant truth. The core question for scientists has been why these models trust their own mistakes so deeply. Is it a random glitch, or is there a hidden pattern in how they learn that makes them certain of the wrong things?

A team of researchers set out to investigate this mystery by looking at the concept of popularity. They wondered if the models were simply favoring answers that were famous or frequently seen in their training data, even when those answers were wrong for the specific question asked. To test this, they focused on a specific type of task: factual questions about real-world entities, such as asking who directed a specific movie or where a basketball player was born. They gathered thousands of these questions and compared the correct answers against the wrong answers the models generated. They measured the "popularity" of the people and places involved by counting how many links pointed to them on the internet and how often they appeared together in written documents. This approach allowed them to see if the models were gravitating toward the most familiar names rather than the correct ones.

The researchers discovered that when these models make mistakes, their errors are far from random. Instead of guessing obscure or unlikely names, the models tend to hallucinate answers that are more popular than the correct facts. For instance, if a model does not know the director of a lesser-known film, it is more likely to confidently name a famous director it has seen many times before, rather than a random unknown. Furthermore, the models often choose answers that frequently appear in the same sentences as the question, even if that connection is incorrect. A model might answer a question about a movie director by naming the producer, simply because those two roles are often mentioned together in articles. The study found that these popular but wrong alternatives are not just common; they are also the ones the model trusts the most. Even when the answer is incorrect, the model assigns a higher level of confidence to the popular, familiar-sounding response than to a less famous, correct one.

This pattern holds true across different types of questions and different sizes of models, suggesting a fundamental flaw in how these systems process knowledge. The researchers found that the more a piece of information is repeated in the training data, the more likely the model is to generate it and the more confident it will be, regardless of whether it is the right answer for the specific query. This creates a dangerous feedback loop where the most familiar incorrect answer is treated as the most probable truth. The study highlights that substantial domain knowledge gaps are associated with even stronger popularity-biased generation, meaning that when models know less about a specific topic, they are more likely to rely on familiar but incorrect associations. The findings reveal strong systematic associations between popularity and overconfidence, though the researchers note that these are correlational rather than causal, meaning they show a clear link without proving that popularity alone causes the overconfident predictions.

To address this, the researchers developed a method to help the models recognize when their confidence might be misplaced. They created a system that looks at the popularity of the answer and the relationship between the question and the answer before accepting the model's confidence score. By factoring in how popular the answer is and how often it appears with the question, the system can adjust the model's confidence downward when it is too sure of a popular but likely wrong answer. When they tested this approach on six different models, the results were striking. The average confidence the models placed on their incorrect answers dropped dramatically, from a high of 0.765 down to 0.254. At the same time, the overall accuracy of the model's confidence improved significantly, meaning the model became much better at knowing when it was right and when it was wrong.

The study concludes that popularity is a key driver of overconfidence in artificial intelligence. The models are not just guessing; they are following a path of least resistance toward the most familiar names and associations. By understanding this bias, it is possible to build better safeguards that prevent these systems from sounding authoritative when they are actually mistaken. The researchers noted that while their work focused on factual questions about specific entities, the principle likely extends to other areas where the models might favor common patterns over specific truths. They also acknowledged that their measures of popularity were based on public internet data, which serves as a proxy for what the models saw during training, and that the relationship they found is a strong correlation rather than a proven cause-and-effect. Nevertheless, the ability to use these popularity signals to calm the machine's overconfidence offers a practical step toward making these powerful tools more reliable and trustworthy for everyday use.

Technical Summary: Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

Problem Statement

Large Language Models (LLMs) frequently exhibit overconfidence, assigning high generation probabilities to incorrect answers. While existing research has established that LLMs are often miscalibrated, the specific factors driving this overconfidence remain insufficiently understood. Prior studies have largely focused on "ground-truth-side" popularity (i.e., how frequently the correct answer appears in training data) to explain when models succeed. However, this approach does not explain why models confidently generate incorrect answers. This paper posits that the mismatch between answer correctness and assigned confidence stems from the frequency structure of the knowledge learned by LLMs, specifically the popularity of the generated answer itself, rather than just the correctness of the ground truth.

Methodology

Task and Data

The study focuses on entity-centric factual Question Answering (QA) derived from Wikidata triplets (Subject, Relation, Object). The models are queried to generate the object given a subject and relation.

Datasets: Three datasets constructed by prior work (Yuksekgonul et al., 2023) covering Movies (Directors), Songs (Performers), and Basketball (Player Birthplaces). These datasets vary in knowledge popularity and model accuracy.
Models: Experiments were conducted on six LLMs, including open-source models (Llama 3-8B, Qwen2-7B, Qwen2.5-7B/14B/32B) and a black-box model (ChatGPT/GPT-3.5-Turbo).

Knowledge Popularity Metrics

Since training data is inaccessible, the authors use external statistics as proxies for knowledge frequency:

Entity Popularity ($Pop$): Measured by the number of Wikipedia sitelinks pointing to an entity.
Co-occurrence Count ($RPop$): Measured by the number of Wikipedia documents where the question entity and the answer entity appear together.
Signals Defined:
- $Pop_Q$ : Popularity of the question entity.
- $Pop_{GT}$ : Popularity of the ground-truth answer.
- $RPop_{GT}$ : Co-occurrence between question and ground-truth.
- $Pop_{Ge}$ : Popularity of the generated answer.
- $RPop_{Ge}$ : Co-occurrence between question and generated answer.

Analysis and Calibration Approach

Correlational Analysis: The authors analyze the relationship between these popularity signals and model confidence (defined as the geometric mean of token generation probabilities) for both correct and incorrect predictions using partial Spearman correlation coefficients.
Calibration Strategy: To mitigate overconfidence, the authors propose a popularity-aware confidence estimator.
- Model: A lightweight Multi-Layer Perceptron (MLP) takes the model's raw confidence ( $c$ ) and popularity signals ( $p$ ) as input to predict the ground-truth correctness ( $y$ ).
- Signals: The system tests individual signals ( $Pop_Q, Pop_{Ge}, RPop_{Ge}$ ) and a combined setting.
- Self-Generated Popularity: To reduce reliance on external corpora, the authors also explore using the LLM itself to estimate popularity signals ( $\hat{Pop}$ ).

Key Findings

1. Hallucinations are Non-Random and Popular

When LLMs hallucinate, they do not generate arbitrary wrong answers. Instead, they tend to generate answers that are:

More popular than the ground-truth answer ( $Pop_{Ge} > Pop_{GT}$ ).
More strongly associated with the question entity than the ground truth ( $RPop_{Ge} > RPop_{GT}$ ).
This pattern is most pronounced in domains where the model has lower factual recall (e.g., the Basketball dataset), suggesting that when knowledge is sparse, models default to globally popular or highly co-occurring entities.

2. Popularity Drives Confidence in Errors

There is a strong positive correlation between the popularity of a generated answer and the confidence assigned to it, even when the answer is incorrect.

Incorrect Predictions: Confidence is strongly driven by question-answer co-occurrence ( $RPop_{Ge}$ ) and answer popularity ( $Pop_{Ge}$ ).
Correct Predictions: Confidence is uniformly high because correct answers typically involve highly popular and strongly associated knowledge, making correlations harder to detect due to low variance.
Scaling: Increasing model scale (e.g., from 7B to 32B parameters) does not eliminate this correlation; larger models remain overconfident in popular but incorrect answers.

3. Mitigation via Popularity Signals

Incorporating knowledge popularity signals significantly improves confidence calibration:

Overconfidence Reduction: The average confidence on incorrect answers ( $Conf_w$ ) dropped from 0.765 (vanilla probability) to 0.254 when using all popularity signals.
Calibration Quality: The Expected Calibration Error (ECE) decreased from 0.356 to 0.050.
Alignment: The ability to correctly distinguish between correct and incorrect answers (Alignment) improved from 77.08% to 83.72%.
Signal Effectiveness: Among individual signals, $RPop_{Ge}$ (question-answer co-occurrence) was the most effective standalone predictor.
Self-Generated Signals: Using the LLM to estimate popularity signals also improved calibration, though with slightly lower gains than using external corpora.

Significance and Claims

The paper claims to provide a systematic understanding of the "popular but wrong" phenomenon, revealing that overconfidence is not merely a failure of knowledge retrieval but a systematic bias toward familiar, high-frequency entities.

Mechanism: The findings suggest that LLMs assign high confidence to incorrect answers because those answers possess strong popularity signals (high entity frequency or high co-occurrence with the query), which the model interprets as a sign of correctness.
Practical Utility: The study demonstrates that these popularity signals can be leveraged as a post-hoc calibration mechanism. By explicitly modeling the relationship between popularity and correctness, it is possible to substantially reduce overconfidence and improve the reliability of confidence-based decisions without requiring access to internal model states or retraining.
Limitations: The authors modestly note that their findings are correlational, not causal, and that the study is limited to closed-book, entity-centric factual QA. The applicability to open-ended generation or complex reasoning tasks remains an open question.

In summary, the paper argues that knowledge popularity is a critical, previously under-explored factor in LLM overconfidence, and that accounting for the popularity of generated answers offers a viable path to better-calibrated models.