💬 NLP

Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language

This study evaluates nine large language models as post-hoc explainers for probabilistic risk communication and finds that while they are generally consistent, they remain significantly miscalibrated—particularly regarding uncertainty—indicating they are not yet reliable zero-shot tools for translating numerical predictions into natural language.

Original authors: Diego Cerda-Mardini, Sarath Chandar, Sreenath Madathil

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Diego Cerda-Mardini, Sarath Chandar, Sreenath Madathil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a crystal ball that can predict the future, but instead of giving you a clear "Yes" or "No," it whispers numbers like "73% chance" or "very shaky prediction." Now, imagine you ask a super-smart robot (a Large Language Model, or LLM) to translate those numbers into plain English for a regular person. You want the robot to say, "It's very likely to rain," when the number is high, and "It's very uncertain," when the number is shaky.

The big question this paper asks is: Can these robots actually do that job without messing up?

The researchers set up a massive game of "translation" to find out. They didn't just ask the robots to guess; they fed them simulated predictions from a "crystal ball" (mathematically modeled using something called a Beta distribution) that had specific, known levels of likelihood and uncertainty. They then asked nine different AI models to pick the perfect phrase from a strict menu of options (like "very likely" or "highly uncertain") to describe the situation. They did this over and over again, changing the "temperature" (how creative or random the robot is allowed to be) and testing them in six different real-world scenarios, from predicting floods to guessing if a gambler will win.

Here is the verdict, straight from the lab: The robots are consistent, but they are terrible at calibration.

Think of it like a weather forecaster who is incredibly reliable at saying the same thing every time you ask, but is completely wrong about what that thing means.

Consistency (The "Reliable Robot"): If you ask the same robot the same question ten times, it will almost always give you the exact same answer. It's like a parrot that never forgets its script. Most of the models scored very high here, with one top-tier model (GPT-5.4) getting a perfect score of 1.0. It never wavered.
Calibration (The "Truth-Teller"): This is where the robots stumble. Calibration means matching the words to the actual math. If the math says "95% chance," the robot should say "virtually certain." If it says "10% chance," it should say "very unlikely." The study found that most robots are miscalibrated. They often pick the same middle-of-the-road phrases (like "somewhat certain") regardless of whether the math is screaming "almost impossible" or "guaranteed." They are like a person who always says "maybe" no matter what the dice roll is.

What the paper explicitly rules out:
You might think, "Maybe the robots just can't do the math! Maybe if we give them the final numbers directly instead of raw data, they'll get it right." The researchers tested this exact idea. They gave the robots the pre-calculated "mode" (the most likely number) and the "sample size" (how sure the math is) right in the prompt. The result? It didn't help. The robots still failed to map the numbers to the right words. This suggests the problem isn't that they can't do the math; the problem is in the translation step itself. They just don't have a solid internal dictionary that links numbers to words.

The "Temperature" Trade-off:
The researchers also played with the "temperature" setting, which controls how random the robot's choices are.

Low Temperature: The robot is a boring robot. It picks the same safe word every time. High consistency, but it gets stuck in a rut and doesn't match the numbers well.
High Temperature: The robot gets a bit wild. It starts picking different words, which actually helps it match the numbers better (improving calibration), but it becomes a flaky robot that gives you different answers for the same question (destroying consistency).

The One Star Performer:
There was one exception: GPT-5.4. This model was a beast at describing likelihood. It got a near-perfect calibration score of 0.96 and a perfect consistency score of 1.0. It seemed to have learned a perfect, rigid rulebook: "If the number is X, I say Y." However, even this super-robot failed when it came to uncertainty. It scored only 0.37 on uncertainty tasks, defaulting to vague, middle-ground phrases just like the others. It turns out, even the smartest robot doesn't know how to say "I'm really not sure" in a way that matches the math.

The Bottom Line:
The paper concludes that right now, these AI models are not ready to be standalone tools for explaining risk to the public. They are like a very consistent translator who speaks a language where the words don't quite match the meaning. If you ask them to explain a medical risk or a flood warning, they might give you a confident-sounding answer that is actually misleading.

The authors suggest that while these models are great at making fluent, easy-to-read sentences, we can't just plug them into a risk calculator and expect them to be accurate. The "bottleneck" isn't the math; it's the robot's inability to truly understand how to turn a number into a trustworthy word. Until we fix that, we need to be very careful about letting AI explain the odds of life-or-death situations.

Technical Summary: Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed as post-hoc explainers for AI-generated outputs in high-stakes domains such as healthcare, finance, and law. While calibrated uncertainty scores exist to indicate when a model's prediction should be trusted, end-users often struggle to interpret these numerical values due to a divergence between formal statistical definitions and intuitive understanding of probability.

The core problem addressed is whether LLMs can reliably translate probabilistic information (specifically likelihood and predictive uncertainty) into Natural Language Explanations (NLEs). For this role to be viable, models must satisfy two criteria:

Consistency: Producing identical verbal descriptors for identical inputs across repeated queries.
Calibration: Selecting descriptors that accurately reflect the magnitude of the underlying numerical quantities (e.g., assigning "very likely" to high-probability events and "unlikely" to low-probability ones).

Current literature suggests LLMs may prioritize helpfulness and decisiveness over expressing uncertainty, leading to systematic limitations in probabilistic verbalization. However, a controlled evaluation jointly measuring calibration and consistency across multiple architectures, domain contexts, and generation settings has been lacking.

2. Methodology

Experimental Design

The study formalizes risk communication as a descriptor selection task. The experimental pipeline involves:

Upstream Simulation: Predictions are simulated by sampling from Beta distributions parameterized by a mode ( $m$ , representing likelihood) and a prior sample size ( $k$ , representing uncertainty). A grid of 100 combinations ( $m \in [0.05, 0.95]$ , $k \in [50, 950]$ ) was used to generate systematic variations in likelihood and uncertainty.
Task Formulation: LLMs were prompted to select the best verbal descriptor for the likelihood and uncertainty of these simulated scenarios from a fixed, ordered set of NLEs (e.g., "virtually certain" to "exceptionally unlikely").
Variables:
- Models: Nine LLMs were evaluated, ranging from small open-source models (4B–8B parameters) to larger commercial and open models (up to 32B and GPT-5.4).
- Domain Contexts: Six contexts were used to test robustness, varying in stake level (high, medium, low) and event scope (individual vs. population): flood prediction, cancer prediction, drug anaphylaxis, gambling, restaurant recommendations, and aurora hikes.
- Temperature: Experiments were repeated across ten temperature settings (0.0 to 0.9).
- Repetitions: Each experiment was repeated ten times to measure consistency.
Ablation Studies:
- Precomputed Statistics: An ablation provided the mode ( $m$ ) and sample size ( $k$ ) directly in the prompt to determine if the bottleneck was statistical inference or verbalization.
- Metric Robustness: Six additional ablations tested metric sensitivity, including external validation against expert-elicited ranges, penalty parameter variations, and alternative uncertainty measures (variance, entropy).

Evaluation Metrics

Consistency Score: Measured the frequency of identical descriptor selection across ten repetitions for identical inputs (range [0, 1]).
Calibration Score: Adapted from the Jonckheere-Terpstra (JT) trend test. It evaluated whether the model's descriptor choices respected the numerical ordering of the underlying distributions. A violation occurred if a model assigned a "stronger" descriptor to a lower numerical value than a "weaker" descriptor elsewhere. The score was penalized for models that failed to utilize the full vocabulary of available descriptors.
Task Correctness (pass@k): Measured the probability of the model producing a correctly formatted output within $k=5$ attempts, ensuring that formatting errors did not artificially inflate calibration or consistency scores.

3. Key Results

Consistency vs. Calibration

General Finding: LLMs were found to be generally consistent but miscalibrated.
Consistency: Most architectures achieved high consistency scores (mean > 0.8) for likelihood tasks. GPT-5.4 achieved a perfect mean consistency score of 1.0 for both likelihood and uncertainty. However, consistency decreased significantly as temperature increased.
Calibration:
- Likelihood: Most models were miscalibrated. GPT-5.4 was the exception, achieving a mean likelihood calibration of 0.96. Other models ranged from 0.07 to 0.78.
- Uncertainty: Calibration was substantially weaker across all models. No architecture achieved a score above 0.40 for uncertainty tasks (with GPT-5.4 scoring 0.37 and others as low as 0.07).

Impact of Variables

Temperature: Increasing temperature created a trade-off: it slightly improved calibration (by encouraging the use of a wider vocabulary) but considerably decreased consistency.
Domain Context: Context influenced both metrics, but no single domain uniformly improved or worsened performance across all models. GPT-5.4 remained largely unaffected by domain context changes.
Precomputed Data: Providing explicit numerical summaries (mode and sample size) did not improve calibration or consistency for any model. This suggests the bottleneck lies in the verbalization step (mapping numbers to words) rather than in the statistical inference of raw samples.
Vocabulary Usage: Most models failed to select all available descriptors. They tended to default to a narrow set of moderate descriptors (e.g., "somewhat certain") regardless of the underlying mathematical uncertainty, rarely selecting extreme descriptors.

Robustness

Ablation studies confirmed that the findings were robust. External validation against expert-elicited ranges showed GPT-5.4 tracked expert consensus closely, while other models showed large absolute errors. The use of precomputed parameters did not alter the fundamental miscalibration, reinforcing that the issue is not the inability to calculate statistics but the inability to verbalize them accurately.

4. Contributions and Significance

The paper provides a systematic evaluation of LLMs as post-hoc explainers for probabilistic predictions, establishing that:

Current Limitations: Current LLMs do not constitute reliable zero-shot standalone risk communication tools. While they are consistent, they are significantly miscalibrated, particularly regarding predictive uncertainty.
The Verbalization Bottleneck: The inability to communicate risk accurately is not due to a failure in statistical reasoning (as providing ground truth parameters did not help) but resides in the mapping of known numerical quantities to calibrated verbal descriptors.
Scale and Architecture: Strong performance did not arise uniformly across architectures. While GPT-5.4 demonstrated near-perfect calibration for likelihood, this competence did not transfer to uncertainty tasks, suggesting that scale alone may not be sufficient for emergent understanding of probabilistic concepts.
Contextual Sensitivity: Models exhibit sensitivity to domain context, potentially using more cautious language in high-stakes medical or disaster scenarios, which complicates the distinction between genuine miscalibration and socially appropriate hedging.

5. Conclusion

The authors conclude that while LLMs remain promising for producing fluent and accessible language, they currently lack a sufficiently stable internal mapping between probabilistic information and NLEs. The study underscores that pairing LLMs with upstream statistical methods alone does not resolve calibration deficits, highlighting the need for targeted interventions specifically at the verbalization stage. The findings reinforce that risk communication is a difficult, specialized task, and current models are not yet ready to reliably translate probabilistic predictions into natural language for informed decision-making.