The strength of clinical evidence is recoverable from language model representations but not from their stated grades

Technical Summary: The Recoverability of Clinical Evidence Strength from LLM Representations

Problem Statement
Large language models (LLMs) are increasingly deployed to summarize clinical evidence, where the utility of a claim depends not merely on its truth but on the strength of the supporting evidence (e.g., randomized controlled trials vs. expert opinion). However, LLMs are known to communicate confidence poorly; their stated certainty is often weakly calibrated and fails to track correctness. A critical, untested question remains: do clinical LLMs internally register the strength of evidence for a claim, distinct from its factual truth, and if so, do they express this registration when prompted? Furthermore, it is unclear whether any internal signal regarding evidence strength represents a deep understanding of evidentiary weight or merely surface lexical cues.

Methodology
The study employed a comprehensive evaluation framework involving 45,134 clinical claims compiled from six public biomedical sources (including USPSTF, CPSTF, Trialstreamer, Evidence Inference, EBM-NLP, and medical reversals). Of these, 20,611 claims were harmonized into a four-level evidence grade (High, Moderate, Low, Very Low) under three independent grading frameworks. A subset was also labeled for factual truth (True, False, Uncertain), with specific design elements (e.g., "reversal" claims and balanced grids) used to decorrelate evidence grade from factual truth.

The authors tested 22 local, open-weight, instruction-tuned LLMs ranging from 0.6 to 70 billion parameters, spanning general-domain, medical domain-adapted, and reasoning-distilled families. The experimental pipeline involved:

1. Hidden-State Extraction: Extracting residual stream activation vectors at the final token of each claim from frozen models.
2. Linear Probing: Training L2-regularized multinomial logistic estimators to decode the evidence grade from these hidden states.
3. Verbalization Elicitation: Prompting models to explicitly state the evidence grade (zero-shot and few-shot) to compare stated grades against internal representations.
4. Control Analyses:
   • Lexical Controls: Comparing hidden-state decodability against TF-IDF bag-of-words classifiers and surface covariates (length, hedge density).
   • Generalization: Testing cross-topic and cross-framework transfer (leave-one-framework-out).
   • Separability: Constructing a balanced grade-by-truth grid to test if the grade signal is distinct from factual truth.
   • Steering: Attempting to modulate model output by adding the grade direction vector to the residual stream.
   • Flagging: Training a binary estimator to detect weakly supported claims.

Key Results

• Internal Recoverability vs. Stated Output: A linear estimator could recover the four-level evidence grade from the hidden states of every model tested, achieving a median peak-layer macro AUROC of 71.8 (significantly above chance). In stark contrast, the models' stated grades fell to chance levels (median accuracy ~22%), creating a large, systematic gap (25–27 percentage points) between what is recoverable internally and what is expressed verbally.
• Model Scale and Specialization: Contrary to the hypothesis that larger or more specialized models would possess cleaner internal structures, decodability did not improve with scale. On the Qwen3 ladder, decodability decreased as parameter count increased. Reasoning-distilled models performed the worst, while medical domain adaptation showed no significant effect.
• Nature of the Signal: The recoverable signal was found to be largely lexical. A TF-IDF classifier outperformed the hidden-state estimators (AUROC 78.9 vs. 71.8). Adding hidden features to lexical classifiers provided no benefit, and once lexical scores were partialled out, the hidden signal's correlation with the true grade dropped to near zero. The signal did not generalize across different clinical topics or grading frameworks, suggesting it reflects source-specific vocabulary rather than an abstract concept of evidence strength.
• Separability from Truth: On a balanced grid where grade and truth were statistically independent by construction, the grade and truth directions in the representation space were nearly orthogonal. The grade direction predicted truth no better than chance, confirming that the internal signal is distinct from factual correctness, even though they are strongly confounded in natural corpora.
• Steering and Flagging: Steering generation along the grade direction did not reliably alter the model's stated certainty, indicating the signal is not a functional lever for output control. However, an external estimator could successfully flag weakly supported claims (AUROC 69.2) using the hidden states, offering a practical mechanism to identify low-quality evidence without retrieval.

Significance and Claims
The paper claims that clinical LLMs carry an ordered signal of evidence strength in their internal representations that they fail to express in their outputs. This dissociation implies that the "grades" a model verbally provides are uninformative regarding the actual support for a claim. The authors argue that while the recoverable signal is largely lexical and not a deep, portable representation of evidence strength, its existence is critical for safety: the information needed to weigh a clinical assertion is present inside the model but absent from its output.

Consequently, the study concludes that relying on a model's stated confidence or grade is unsafe. Instead, safety in clinical deployment requires an external check—either reading the signal from the representation or using a text-based classifier—to flag weakly supported claims. The paper cautions that decodability results without lexical and out-of-distribution controls may overstate what a model truly encodes, and that the apparent identity of evidence grade and truth in natural data is an artifact of distribution rather than a shared internal code.