💻 computer science

Information Discernment in Large Language Models

This paper introduces the Learn2Discern framework to reveal that large language models consistently fail to appropriately weigh information based on source reliability or truthfulness, a critical shortcoming that user studies confirm erodes trust and which persists despite increases in model size, though it can be mitigated through simple inference-time interventions.

Original authors: Joshua Ashkinaze, Laura Kurek, Alina Faisal, Tongyuan Miao, Mariam Joseph, Ceren Budak, Eric Gilbert

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joshua Ashkinaze, Laura Kurek, Alina Faisal, Tongyuan Miao, Mariam Joseph, Ceren Budak, Eric Gilbert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have your own theory about what happened, but then you start getting tips from the public. Some tips come from a seasoned, honest police officer; others come from a known liar who just wants to cause trouble. Some tips help you get closer to the truth, while others send you running in the wrong direction. The big question is: does your detective brain know how to weigh these tips? Do you listen more to the honest officer, or do you get distracted by the loudest voice? Do you change your mind when a tip helps you, or do you stubbornly stick to your original guess even when it's wrong?

This is exactly the kind of puzzle that scientists are asking about modern "Large Language Models" (LLMs). These are the super-smart computer programs that power chatbots and search engines. They are trained on a massive amount of data from the internet, but they also need to look up new information to answer questions they don't know yet. The problem is, the internet is a noisy place. It's full of reliable news sites, but also full of fake news, rumors, and popular but wrong ideas. If a computer program can't tell the difference between a trustworthy source and a popular liar, or if it can't tell the difference between a helpful clue and a misleading one, it might start spreading misinformation to millions of people. This paper dives deep into whether these digital detectives are actually good at sorting out the truth from the noise.

The researchers, a team from the University of Michigan, created a new way to test this skill, which they call "information discernment." They set up a giant experiment involving 13 different AI models and nearly 670,000 trials. They asked the models questions with known answers, gave them a "prior" belief (what the model thought before), and then fed them a new claim from a specific source. Sometimes the source was highly reliable, sometimes it was a fringe website. Sometimes the new claim was closer to the truth, and sometimes it was wildly wrong. They wanted to see if the models would update their answers more for the good sources and the helpful claims.

The results were a bit of a shock. The paper found that, overall, these AI models are terrible at this kind of discernment. They perform almost no better than random guessing when it comes to deciding which sources to trust. In fact, the models were twice as likely to listen to a source just because it was popular (like a website with lots of visitors) rather than because it was reliable (like a fact-checked news outlet). It's as if the detective ignores the police officer and listens to the person with the biggest megaphone. Furthermore, the models didn't seem to care if a new piece of information helped them get closer to the truth or pushed them further away; they updated their beliefs roughly the same amount in both cases.

The team also checked if making the models bigger or newer would fix this. They found that newer and larger models got slightly better at spotting the truth in claims, but they still couldn't learn to distinguish between reliable and unreliable sources. This suggests that just making the AI "smarter" or training it longer doesn't automatically teach it to be a critical thinker about where information comes from. However, there was a silver lining: the researchers discovered that simple tricks, like asking the model to rate the source's reliability before answering, could significantly improve its performance.

To make sure this wasn't just a computer science problem, the researchers also asked 299 real human users what they thought. They found that people absolutely care about these skills. When users were told that a chatbot ignored reliable sources or changed its mind when it was already right, they said they would trust the bot less and use it less. This confirms that the ability to discern information isn't just a technical metric; it's a core requirement for building AI that people can actually rely on.

In short, the paper reveals that while our current AI models are great at retrieving information, they are currently quite blind when it comes to judging the quality of that information. They tend to follow popularity over truth and struggle to update their beliefs correctly. But the good news is that with a few simple instructions, we can help them see the light a little better, making them safer and more useful tools for everyone.

Technical Summary: Information Discernment in Large Language Models

Problem Statement

As Large Language Models (LLMs) increasingly function as front-end answer engines that retrieve and synthesize external information (e.g., via Retrieval-Augmented Generation or RAG), they face "knowledge conflicts" where external claims contradict their internal training priors. A critical vulnerability arises if models fail to weigh this external evidence appropriately. Specifically, models may propagate misinformation if they update their beliefs equally for high-credibility outlets and fringe websites (a failure of source discernment) or if they update equally for claims that bring them closer to the truth versus those that move them further away (a failure of truth discernment). Given that humans tend to over-trust LLM outputs, these failures in weighting external information pose a significant risk to the trustworthiness of AI-mediated queries.

Methodology

The authors introduce Learn2Discern (L2D), an experimental framework and benchmark designed to measure information discernment under knowledge conflicts.

Experimental Design

The core experiment involves a three-step elicitation procedure:

Prior Elicitation: The model is asked a question ( $Q$ ) with a numeric ground truth ( $T$ ) to establish a prior belief ( $P_1$ ).
Conflict Injection: A claim ( $C$ ) is generated by perturbing the ground truth ( $C = T \times F$ , where $F$ is a perturbation factor) and attributed to a source ( $S$ ) with known reliability ( $R_s$ ) and popularity ( $P_s$ ).
Posterior Elicitation: The model is prompted: "S said the answer is C. What do you think now?" to record the posterior belief ( $P_2$ ).

Data Construction

The benchmark, Learn2Discern, comprises:

Questions: 4,248 numeric questions derived from existing benchmarks (TriviaQA, NumerSense) and novel longitudinal datasets (General Social Survey, World Bank Development Indicators, TREC user queries) to mitigate data contamination.
Sources: 132 web news sources stratified by reliability (using the Lin et al. PC1 metric) and popularity (Open PageRank), creating six strata (3 popularity levels $\times$ 2 reliability levels).
Scale: The combinatorial design yields over 2.8 million potential experimental tuples, with approximately 670,000 trials conducted across 13 models.

Metrics

The framework operationalizes three normative axioms into interpretable metrics:

Source Discernment (SD): The Spearman rank correlation ( $\rho$ ) between the magnitude of the update ( $\Delta = |P_1 - P_2|$ ) and source reliability. A positive correlation indicates the model updates more for reliable sources.
Truth Discernment (TD): The Spearman rank correlation between the update magnitude and the "improvement" ( $I$ ) of the claim relative to the prior ( $I = |P_1 - T| - |C - T|$ ). A positive correlation indicates the model updates more when the claim brings it closer to the truth.
Correct Defense: The probability that a model maintains its answer when its prior is correct and the new claim is incorrect.

Validation

A pre-registered, quota-matched user study ( $n=299$ ) was conducted to validate the axioms. Users confirmed they endorse these principles and reported that violations would decrease their trust in and usage of chatbots.

Key Contributions

Axioms and Metrics: Formal definitions of source and truth discernment under knowledge conflicts, providing a standardized way to measure vulnerability to misinformation.
User Validation: Empirical evidence that real LLM users value these discernment properties and that violations negatively impact trust and adoption.
Benchmark Dataset: The release of Learn2Discern, a large-scale dataset pairing diverse numeric questions with stratified web sources, enabling causal testing of how models weight reliability and truth signals.
Empirical Findings: A comprehensive evaluation of 13 models (varying in size, recency, and architecture) revealing systematic failures in discernment.

Results

Across 13 models and nearly 670,000 trials, the study found:

Near-Chance Performance: Models perform near chance on both source and truth discernment metrics. They do not consistently update more for reliable sources or for claims that improve their accuracy.
Popularity Bias: Models rely on source popularity approximately twice as much as source reliability ( $\rho_{pop} \approx 0.07$ vs. $\rho_{rel} \approx 0.03$ ).
Model Scale and Recency: While larger and newer models show improvements in truth discernment and correct defense, they do not show improvements in source discernment. This suggests source discernment is a persistent blind spot not solved by increased model complexity.
Prior Accuracy Correlation: External evidence integration is most effective on datasets where the model's priors are already accurate. When models are initially wrong, they often fail to integrate external information correctly, sometimes updating further away from the truth.
Intervention Efficacy: Simple inference-time interventions (prompting strategies) can improve discernment. Specifically:
- Explicitly asking models to rate source reliability before updating improves source discernment.
- "Defense" prompts (evaluating if the original answer was correct) improve correct defense.
- Chain-of-Thought (CoT) reasoning improves truth-related metrics.

Significance and Claims

The paper claims that information discernment is a core alignment property that scales in importance as LLMs replace traditional search engines. The findings suggest that current LLMs lack the inherent ability to distinguish between high-quality and low-quality external evidence, relying instead on superficial signals like popularity.

The authors modestly claim that while current models exhibit systematic failures, these deficits can be partially addressed through low-cost, inference-time interventions (e.g., specific prompt modifications). They position Learn2Discern as a necessary testbed for evaluating and improving this property, noting that without such discernment, external evidence integration risks degrading rather than improving the trustworthiness of information users receive. The work highlights that source discernment remains a specific challenge that model scaling alone does not resolve, pointing to a need for future work in fine-tuning or architectural changes to encode source reliability knowledge.