Technical Summary: Validating Elicited Beliefs from LLMs

Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes domains, such as medical diagnosis, where decision-making requires forming probabilistic beliefs about unknown outcomes. While researchers can prompt LLMs to output verbalized probabilities (e.g., "the probability of disease X is 80%"), it remains unclear whether these outputs reflect coherent internal beliefs that actually drive the model's decisions. A model might verbalize a probability that is disconnected from its decision-making process, potentially due to finetuning artifacts or preference optimization that distorts verbalized probabilities without altering underlying decision logic. Existing work focuses on calibration (accuracy of reported probabilities) but does not address whether the reported beliefs are the sufficient statistics governing the model's actions. The core challenge is to validate, in a black-box manner, whether an LLM's elicited beliefs are a meaningful explanation for its decisions without making strong assumptions about the model's utility function or requiring access to internal representations.

Methodology

The authors propose a decision-theoretic framework to test the consistency between elicited beliefs and observed decisions. The framework operates under the assumption of a "near-rational" decision-maker, formalized as a perturbed utility maximizer. This class of agents maximizes expected utility subject to a strictly convex regularizer (e.g., entropy), allowing for stochastic choices and unobserved utility components, which is more general than strict expected utility maximization.

The framework relies on two empirically testable conditions derived from this formalization:

1. Conditional Independence (Belief Sufficiency):

   • Hypothesis: If an agent's actions are driven solely by its subjective belief $P_S(\theta|x)$, then conditioning on this belief should render the action $A$ independent of the true outcome $\theta$. Formally, $A \perp \theta \mid P_S(\theta|x)$.
   • Test: The authors test the null hypothesis $I(A; \theta \mid p) = 0$ using a non-parametric k-Nearest Neighbors Conditional Mutual Information (CMI) test. Additionally, they measure the predictive improvement of the true outcome when adding the action $A$ to a model that already conditions on the elicited belief $p$. If the action provides significant additional predictive power, the belief is not a sufficient statistic.

2. Cyclic Monotonicity (Action-Guiding Beliefs):

   • Hypothesis: For a perturbed utility maximizer, the mapping from elicited beliefs to choice probabilities must be cyclically monotone with respect to the expected utility index. In binary state settings, this implies that as the elicited probability of a positive state increases, the probability of selecting actions more valuable in that state must increase monotonically.
   • Test: The authors bin elicited probabilities and estimate choice shares within bins. They formulate a signed-margin linear program (SignedLP) to find a utility-difference vector $d$ that satisfies strict monotonicity across bins. A positive statistic $\hat{\gamma}$ indicates strict monotonicity; a negative value indicates a violation.

The framework is applied to four medical diagnostic tasks (structural heart disease, diabetes, pediatric fever, and infant crying) using datasets from real-world records and expert-constructed Bayesian networks. The study evaluates multiple LLM families (GPT-5 variants, DeepSeek-R1, Llama-4) by eliciting beliefs and decisions in separate context windows to avoid contamination.

Key Contributions

• Formal Framework for Black-Box Validation: The paper introduces the first formal, decision-theoretic conditions to test if elicited LLM beliefs are coherent with their actions. Crucially, these tests are agnostic to the agent's utility function, relying only on the structure of perturbed utility maximization.
• Necessary and Sufficient Conditions: The authors prove (Proposition 5) that conditional independence and cyclic monotonicity are jointly necessary and sufficient for an observed distribution of beliefs and actions to be rationalized by a perturbed utility maximizer holding the elicited belief as its true subjective probability.
• Empirical Validation Protocol: The study operationalizes these theoretical conditions into practical statistical tests (CMI and SignedLP) that can be applied to any black-box LLM.
• Distinction Between Coherence Types: The work distinguishes between decision-theoretic coherence (beliefs guiding actions) and probabilistic coherence (adherence to axioms like the Law of Iterated Expectation), showing they are distinct properties.

Results

• Imperfect Sufficiency: Across all models and datasets, the Conditional Independence test rejects the null hypothesis. Elicited beliefs are not sufficient statistics; the models' actions contain residual information about the true outcome that is not captured in their verbalized probabilities. This suggests models "know more than they say" or that their decision-relevant information is not fully communicated in the elicited belief.
• Monotonicity Holds (Mostly): The Cyclic Monotonicity test shows that for most model-dataset combinations, the relationship between elicited beliefs and actions is monotone. Specifically, 9 out of 16 combinations showed strict monotonicity (positive $\hat{\gamma}$), and only one (Fever–Llama) showed a violation. This indicates that while beliefs are not perfectly sufficient, they generally move decisions in the correct direction.
• Model and Domain Variance:
  • Strongest Models: The strongest models (e.g., DeepSeek-R1, GPT-High) exhibit the smallest discrepancies between beliefs and actions, suggesting they are observationally closer to agents with coherent beliefs.
  • Domain Effects: The Diabetes dataset showed the weakest monotonicity, with no model showing a strictly positive margin.
• Calibration Does Not Fix Sufficiency: Applying isotonic calibration to the elicited beliefs improved marginal calibration but did not restore conditional independence, confirming that the sufficiency failure is structural, not just a calibration issue.
• Internal Consistency: Tests using the Law of Iterated Expectation (LIE) revealed that probabilistic coherence (adherence to probability axioms) does not necessarily align with decision-theoretic coherence. Models performing well on decision-consistency tests did not necessarily perform well on probabilistic consistency tests.

Significance and Claims

The paper claims that elicited probabilities from LLMs are imperfect but behaviorally informative summaries of decision-relevant information. The authors argue that while models do not act as if they hold perfectly coherent beliefs (as evidenced by the failure of sufficiency), the discrepancies are relatively small for state-of-the-art models.

The significance of this work lies in providing a principled, black-box method to validate whether an LLM's verbalized confidence is a trustworthy explanation for its decisions. This is critical for safety and transparency in high-stakes settings like medicine. The authors emphasize that their tests do not prove a model "truly" holds a belief in a philosophical sense, but rather that the model is observationally indistinguishable from a rational agent holding that belief. Consequently, the paper concludes that elicited beliefs should be empirically validated rather than assumed to faithfully represent model beliefs, offering a diagnostic tool for developers to understand failure modes (e.g., distinguishing between overconfidence and cost-perception issues in medical agents).