🤖 AI

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

This paper demonstrates that Large Language Models lack genuine epistemic self-awareness on clinical tabular data, as their verbalized confidence is uninformative and their accuracy inversely correlates with task difficulty, but proposes that cross-model attribution divergence analysis combined with few-shot prompting and feature evidence can effectively detect these blind spots and significantly improve both accuracy and calibration without retraining.

Original authors: Akshat Dasula, Prasanna Desikan, Jaideep Srivastava

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Akshat Dasula, Prasanna Desikan, Jaideep Srivastava

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Overconfident Intern"

Imagine you hire a brilliant medical student (the LLM) who has read every medical textbook in the world but has never actually worked in a hospital. You also have a seasoned, data-driven nurse (the XGBoost model) who has spent years tracking patient vitals and knows exactly which numbers predict a specific illness (Acute Kidney Injury).

The researchers asked a simple question: Does the medical student know when they are guessing?

They found that the student is dangerously overconfident. No matter how wrong they are, they say, "I am 90% sure!" The paper calls this "Epistemic Blind Spots"—the student doesn't know what they don't know.

The Experiment: Four Ways to Ask the Question

The researchers tested the medical student (using a model called Qwen 2.5) on 300 patient records. They tried four different ways of asking for a diagnosis:

The Blank Slate (Zero-Shot): Just giving the patient's data and asking, "Do they have kidney injury?"
- Result: The student guessed randomly (49% accuracy) but still claimed to be 85% confident.
The Cheat Sheet (Zero-Shot + SHAP): Giving the student the data plus a list of the top 5 numbers the nurse thinks are important.
- Result: The student still guessed poorly (52% accuracy) and still claimed 85% confidence. They looked at the right numbers but didn't understand how to use them.
The Examples (Few-Shot): Giving the student the data plus four examples of past patients with their correct diagnoses.
- Result: The student got much better (68% accuracy) and claimed 93% confidence.
The Super Combo (Few-Shot + SHAP): Giving the student the examples and the nurse's list of important numbers.
- Result: The student became very good (75% accuracy) and claimed 93% confidence.

The Four Big Discoveries

1. The "Broken Thermometer" (Confidence is Useless)

The most shocking finding is that the student's "confidence score" is a broken thermometer.

The Analogy: Imagine a thermometer that always reads "98.6°F" whether you are healthy or running a 104°F fever.
The Reality: Whether the student got the answer right or wrong, easy or hard, they always said they were roughly 85% to 93% sure. Their confidence was determined entirely by how the question was written, not by how good the answer was. It tells you nothing about whether the prediction is correct.

2. The "Inverse Difficulty" Effect (They Fail Where the Expert is Sure)

The researchers noticed a weird pattern:

When the Nurse was 99% sure (because the data was very clear and specific), the Student was only 65% correct.
When the Nurse was unsure (because the data was messy), the Student actually did just as well as the Nurse (about 74% correct).
The Analogy: The student is great at general medical theory but terrible at the specific, weird patterns of this specific hospital's data. The student shines when the data is fuzzy and general knowledge helps, but fails when the data requires specific, learned patterns.

3. The "Magic Combo" (1 + 1 = 3)

The researchers found that giving the student examples (Few-Shot) and giving them the nurse's list of important numbers (SHAP) worked better together than the sum of their parts.

The Analogy: Imagine trying to learn to drive.
- Examples alone are like watching a video of someone driving. You understand the goal but don't know which pedals to press.
- The List alone is like someone pointing at the pedals and saying "Press this." You know what to touch but not why or when.
- Together, you finally understand the whole car. The researchers found that combining these two methods fixed the student's reasoning much more than just adding the two improvements together.

4. The "Translator" (Fixing the Confidence)

Since the student's own confidence is useless, the researchers built a small "Translator" (a calibrator).

How it works: This Translator looks at the difference between what the Student thinks is important and what the Nurse thinks is important.
The Result: If the Student and Nurse disagree on what matters, the Translator says, "Hey, this prediction is risky." If they agree, it says, "This is likely safe."
The Impact: This replaced the student's fake confidence with a real, patient-specific reliability score. It didn't need to look inside the student's brain or run the test twice; it just compared the two models' reasoning.

The Conclusion: The "Cold Start" Problem

The paper concludes that putting an LLM on structured data (like spreadsheets of patient vitals) is like a Cold Start.

The student has the knowledge (they know medicine).
But they lack the direction (they don't know which numbers to look at).
And they lack self-awareness (they don't know when they are guessing).

The paper suggests that we don't need to replace the smart, data-driven nurse (XGBoost). Instead, we should use the nurse to guide the student (by showing them which features matter) and to calibrate the student (by telling us when the student's reasoning is drifting). This creates a system where the student learns to be genuinely aware of their own limits.

Technical Summary: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence

Problem Statement

While Large Language Models (LLMs) show promise in medical question answering and note summarization, their application to structured clinical tabular data (e.g., Electronic Health Records) remains under-explored regarding epistemic self-awareness. Prior research has established that tree-based models (e.g., XGBoost) consistently outperform LLMs on tabular prediction tasks, but this performance gap has largely been treated as a monolithic aggregate metric (e.g., lower AUROC) without examining the underlying epistemic structure.

The core problem addressed is whether LLMs can recognize the limits of their own knowledge on structured tasks. Specifically, the paper investigates if LLMs can distinguish between cases where they are likely correct versus incorrect, and whether their verbalized confidence signals are reliable indicators of prediction quality in clinical settings. The authors frame this as a "cold start" problem: LLMs possess relevant medical knowledge but lack the direction (feature attention) and self-assessment mechanisms required to apply this knowledge to structured data.

Methodology

Data and Task

The study utilizes the MIMIC-IV clinical database to replicate an Acute Kidney Injury (AKI) prediction task.

Dataset: A balanced cohort of 10,000 encounters (5,000 per class) derived from ~209,000 admissions.
Features: 321 features including vital signs, demographics, CCS diagnosis codes, and discretized laboratory values.
Split: 80/20 train-test split for XGBoost; a 15% subset (n=300) from the test set used for LLM evaluation.
Baseline: An XGBoost classifier trained on the top 50 most influential variables, achieving an AUROC of ~0.85 and accuracy of ~0.823 on the evaluation subset.

Experimental Setup

The authors evaluated Qwen 2.5 7B Instruct (locally hosted) across four prompting conditions:

Zero-Shot (ZS): Serialized patient features with a prediction prompt.
ZS + SHAP: ZS prompt augmented with the top-5 SHAP features, importance scores, and directional impacts from the XGBoost model.
Few-Shot (FS): ZS prompt with four class-balanced labeled examples.
FS + SHAP: Combination of few-shot examples and SHAP feature injection.

Output Requirements: For each case, the LLM was required to generate a binary prediction, a verbalized confidence score (0–1), and a ranked list of top-5 features with risk directions. Outputs were enforced via structured JSON.

Metrics and Analysis

Attribution Disagreement Score (ADS): Quantifies alignment between XGBoost's SHAP attributions and the LLM's self-reported feature rankings using Kendall's rank correlation ( $\tau$ ). $ADS = 1 - \tau$ .
Calibration: Measured via Expected Calibration Error (ECE) and AUROC for distinguishing correct vs. incorrect predictions.
Cross-Model Calibrator: A lightweight model (Logistic Regression or XGBoost) trained to predict the probability of LLM correctness ( $P(\text{LLM correct})$ ) using cross-model features (ADS, Jaccard overlap, directional agreement, confidence differences, etc.).

Key Findings

1. LLM Verbalized Confidence is Epistemically Vacuous

The study finds that LLM verbalized confidence is entirely determined by the prompt template, not prediction quality.

Observation: Confidence scores remained near-constant (0.856 for Zero-Shot; 0.937 for Few-Shot) regardless of whether accuracy was 49% or 75.3%.
Implication: On clinical tabular data, verbalized confidence carries zero information about correctness. It tracks prompt format rather than epistemic state, rendering it useless for downstream decision-making.

2. The Inverse Difficulty Effect

There is a counterintuitive relationship between data difficulty and LLM performance relative to XGBoost.

Observation: When XGBoost is highly confident (>0.85) and nearly always correct (99% accuracy), the LLM's accuracy drops to 64.8%. Conversely, when XGBoost is moderately uncertain (confidence 0.70–0.85), the LLM matches its performance (73.8% vs. 73.1%).
Implication: The models exhibit complementary failure modes. The LLM struggles with data-specific distributional patterns learned by XGBoost but adds genuine value in XGBoost's "uncertain zone" where general medical knowledge suffices. However, the LLM cannot identify these zones itself.

3. Super-Additive Attribution Alignment

Few-shot examples and SHAP-derived feature evidence act as orthogonal, super-additive interventions.

Observation:
- Few-Shot: Improves task comprehension (mapping features to predictions) but barely changes feature attention (Jaccard overlap increases only slightly).
- SHAP Injection: Redirects feature attention effectively (Jaccard overlap jumps to 0.658) but results in "shallow adoption" (LLM copies attributions verbally without adjusting predictions significantly).
- Combined (FS + SHAP): The combined effect on reducing ADS (1.158 reduction) exceeds the sum of individual effects (0.796). This combination achieves 75.3% accuracy and 0.378 ADS.
Implication: Solving the "cold start" requires both task demonstrations (direction) and structured model evidence (orientation).

4. Cross-Model Calibration via Attribution Divergence

Attribution divergence signals can power an effective external calibration system.

Observation: A cross-model calibrator using attribution divergence features reduced the Expected Calibration Error (ECE) from 0.254 (raw LLM) to 0.080. The calibrator achieved an AUROC of 0.71 in predicting whether the LLM would be correct.
Implication: Patient-specific reliability estimates can be generated without accessing LLM internals or requiring repeated inference, replacing the uninformative constant confidence with a meaningful reliability score.

Significance and Claims

The paper frames these findings as characterizing a cold start problem for LLMs on structured data. The LLM possesses latent medical knowledge but lacks the mechanism to orient itself within the feature space or assess its own reliability.

Diagnostic Foundation: The work establishes a diagnostic framework for identifying epistemic blind spots (misaligned attention, absent self-assessment) rather than just measuring performance gaps.
Practical Utility: It demonstrates that existing structured models (like XGBoost) can serve a dual role in clinical workflows: providing direction (via attribution injection) and calibration signals (via cross-model divergence) to LLMs.
Epistemic Self-Awareness: The authors argue that current LLMs lack genuine epistemic self-awareness on structured tasks. The proposed path forward involves using cross-model calibration targets as reward signals for Reinforcement Learning (RL), potentially teaching LLMs to internalize calibrated uncertainty in the future.

The study is modest in its scope, evaluating a single model (Qwen 2.5 7B) on a single task (AKI) with a limited sample size (n=300). It acknowledges limitations such as self-reported feature attributions and the preliminary nature of the calibrator, positioning this work as a foundational step toward broader research in epistemic intelligence for structured data.