💬 NLP

Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory

This paper introduces a Type-2 Signal Detection Theory framework using meta-d' and M-ratio to distinguish between an LLM's factual knowledge and its metacognitive efficiency, revealing that standard calibration metrics can mask significant variations in how well models truly "know what they know" across different domains and model architectures.

Original authors: Jon-Paul Cacioli

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jon-Paul Cacioli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of expert detectives to solve a series of mysteries. You need to know two things about each detective:

How good are they at solving the case? (Did they find the right answer?)
How good are they at knowing when they are right? (Do they confidently say "I solved it!" only when they actually did, or do they brag about solving cases they got wrong?)

For a long time, when we tested AI models (LLMs), we only looked at the first question. We asked: "How often is the AI right?" and "Does its average confidence match its average accuracy?"

This new paper argues that this is like judging a detective only by their win rate, ignoring whether they are overconfident or underconfident. The author introduces a new way to test AI called Metacognitive Efficiency.

Here is the breakdown using simple analogies:

1. The Problem: The "Confident Fool" vs. The "Honest Expert"

The paper uses a great example to show why current tests fail:

Detective A (The Confident Fool): Solves 90% of cases correctly. But for every single case, whether right or wrong, they shout, "I am 90% sure!"
- Current Test Result: This detective looks perfect. Their confidence matches their accuracy (90% confidence = 90% success).
- Reality: Their confidence is useless. If they say "90% sure," you have no idea if they are right or wrong.
Detective B (The Honest Expert): Solves 80% of cases correctly. But when they are right, they say "95% sure," and when they are wrong, they say "60% sure."
- Current Test Result: This detective looks "worse" because their average confidence doesn't perfectly match their average accuracy.
- Reality: Their confidence is gold. If they say "95% sure," you can trust them. If they say "60%," you should double-check their work.

The Paper's Insight: Current tests (like ECE) reward Detective A. This new test (Meta-d') rewards Detective B because it measures how well the confidence signal separates the right answers from the wrong ones.

2. The New Tool: The "Confidence Filter"

The author uses a concept from psychology called Signal Detection Theory. Imagine the AI's brain has two layers:

Layer 1 (The Solver): This part tries to answer the question.
Layer 2 (The Monitor): This part watches the Solver and says, "I think I got this right" or "I'm not sure."

The paper introduces a metric called the M-ratio. Think of it as a filter efficiency score:

M = 1.0 (Perfect Filter): The Monitor uses all the information the Solver has. If the Solver knows the answer, the Monitor knows it too.
M < 1.0 (Leaky Filter): The Solver knows the answer, but the Monitor is "noisy" or confused. It throws away useful information. The AI might be right, but it doesn't know it's right.
M > 1.0 (Super Filter): The Monitor is somehow even more sure than the raw data suggests (rare, but possible).

3. The Big Surprises

The author tested four popular AI models (Llama, Mistral, Gemma) on 224,000 questions. Here is what they found:

The "Best" Solver is the Worst Monitor:
One model (Mistral) was the absolute best at getting the right answers (high "Solver" score). But it had the worst M-ratio. It was like a genius detective who got the right answer 90% of the time but bragged about it with the same confidence whether they were right or wrong.
- Lesson: Just because an AI is smart doesn't mean it knows when to trust itself.
The "Worst" Solver is the Best Monitor:
Another model (Gemma) got fewer answers right overall, but its confidence was incredibly honest. When it said it was sure, it was usually right.
- Lesson: For safety-critical tasks (like medical diagnosis), you might prefer the "honest" model over the "confident" one.
Confidence Changes with "Temperature" (The Mood Ring):
AI models have a setting called "temperature" that makes them more random or more rigid. The study found that changing the temperature changes how bold the AI is (its "criterion"), but it doesn't change how well it can tell right from wrong (its "capacity").
- Analogy: Turning up the temperature is like making a person more talkative and loud. It doesn't make them smarter or more self-aware; it just makes them say "I'm 100% sure!" more often, even if they aren't.
Domain Specificity:
An AI might be a genius at Science but terrible at History, and its confidence might reflect that. Standard tests average everything out, hiding these weaknesses. This new test shows you exactly where the AI is blind.

4. Why This Matters for You

If you are building an app that uses AI, you need to know:

If you use the "Confident Fool" model: You might let the AI make decisions it shouldn't, because it sounds so sure even when it's wrong.
If you use the "Honest Expert" model: You can set a rule: "If the AI says it's less than 80% sure, a human must check the answer." This makes the system safer and more reliable.

Summary

This paper is a wake-up call. It says: "Stop just asking 'Is the AI right?' Start asking 'Does the AI know it's right?'"

By using this new "M-ratio" test, we can stop hiring the loud, overconfident AIs and start hiring the quiet, self-aware ones that actually know what they don't know. This is crucial for building AI that we can trust with real-world decisions.

1. Problem Statement

Current evaluation metrics for Large Language Model (LLM) confidence, such as Expected Calibration Error (ECE), Brier score, and AUROC, suffer from a fundamental limitation: they conflate two distinct cognitive capacities:

Type-1 Sensitivity: The model's ability to discriminate between correct and incorrect answers (accuracy).
Type-2 Metacognitive Sensitivity: The model's ability to monitor its own performance (i.e., knowing when it is right or wrong).

The Core Issue: A model can be perfectly calibrated (low ECE) simply by having a high bias (e.g., always reporting 90% confidence) without actually possessing any ability to distinguish correct from incorrect answers. Conversely, a model might have high accuracy but poor confidence monitoring. Standard metrics reward the former or fail to distinguish the two, leading to suboptimal model selection for applications requiring reliable uncertainty estimation (e.g., selective prediction, human-AI collaboration).

2. Methodology

The paper introduces an evaluation framework based on Type-2 Signal Detection Theory (SDT), a psychophysical framework used to separate sensitivity from decision criteria.

Data & Models:
- Models: Four LLMs across three families: Llama-3-8B-Instruct, Llama-3-8B-Base, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-Instruct.
- Dataset: 224,000 factual QA trials generated from TriviaQA and Natural Questions.
- Conditions: Evaluated across 7 temperature settings ( $T \in \{0.1, \dots, 2.0\}$ ) and four knowledge domains (History, Arts, Geography, Science).
Confidence Variable: Instead of using prompted verbal ratings (which suffer from discretization artifacts), the study uses Normalized Log-Probabilities (NLP) of the generated tokens as a continuous confidence signal.
Key Metrics:
- $d'$ (Type-1 Sensitivity): Measures how well NLP separates correct from incorrect answers.
- Meta- $d'$ : The theoretical $d'$ an ideal observer would need to produce the observed confidence patterns. It measures the information content of the confidence signal itself.
- M-ratio (Metacognitive Efficiency): Defined as $M = \text{meta-}d' / d'$ $M = meta- d^{'} / d^{'}$ .
  - $M=1$ : Optimal metacognition (confidence captures all available evidence).
  - $M<1$ : Metacognitive loss (confidence is noisier than the underlying evidence).
  - $M>1$ : Confidence accesses information beyond binary correctness.
Analysis Pipeline:
- Constructed confidence $\times$ accuracy contingency tables.
- Applied Maximum Likelihood Estimation (MLE) to derive Meta- $d'$ .
- Used bootstrap resampling (10,000 iterations) for confidence intervals.
- Pre-registered hypotheses on the Open Science Framework (OSF).

3. Key Contributions

Decoupling Sensitivity from Monitoring: The paper demonstrates that Type-1 performance (accuracy) and Type-2 performance (metacognitive efficiency) are dissociable. A model can be the best discriminator of facts but the worst monitor of its own uncertainty.
Domain-Specific Efficiency: Metacognitive efficiency is not uniform; it varies significantly across knowledge domains (e.g., a model may be efficient in Geography but blind in Science). Aggregate metrics hide these critical weaknesses.
Dissociation of Policy and Capacity: Temperature manipulation affects the Type-2 criterion (confidence bias) but leaves Meta- $d'$ (metacognitive capacity) stable for certain models. This proves that confidence policy (how "confident" a model sounds) is an adjustable parameter independent of its ability to know what it knows.
Inverted Rankings: The study shows that standard metrics (like AUROC2) and the proposed M-ratio produce fully inverted model rankings, answering fundamentally different questions.

4. Key Results

Dissociation of Accuracy and Efficiency:
- Mistral-7B achieved the highest Type-1 sensitivity ( $d' = 1.597$ ) but the lowest M-ratio ($0.852$). It is the best factual discriminator but the worst metacognitive monitor.
- Gemma-2-9B had the lowest Type-1 sensitivity ( $d' = 0.946$ ) but a near-optimal M-ratio ($1.048$).
- Implication: Standard metrics rank Mistral #1; M-ratio ranks it #4. For selective prediction tasks, Gemma-2 would outperform Mistral despite lower raw accuracy.
Domain Variability:
- Different models have different "weakest" domains. For example, Llama-3-Instruct and Gemma-2 were least efficient in Science & Technology, while Mistral was weakest in Arts & Literature.
Temperature Effects:
- For Mistral and Gemma, changing temperature shifted the confidence criterion (bias) but did not significantly alter Meta- $d'$ . This indicates that tuning temperature improves calibration (bias) but does not fix a lack of metacognitive capacity.
- Instruction tuning (Llama-3 Instruct vs. Base) was found to shift the criterion and, in some domains (Science), degrade metacognitive efficiency.
Robustness: The M-ratio metric remained stable across different binning granularities ( $K=3, 4, 6$ ) and quantization levels, confirming its reliability as an ordinal measure.

5. Significance and Implications

Model Selection: For systems relying on confidence (e.g., "abstain if confidence < threshold"), selecting a model based on ECE or AUROC can lead to catastrophic failures. The M-ratio is the superior metric for selecting models that can reliably identify their own errors.
Deployment Strategy: The findings suggest that "calibration" (fixing bias via temperature) is insufficient if the underlying metacognitive signal is weak. Models with low M-ratios require architectural or training interventions, not just post-hoc calibration.
Theoretical Shift: The paper moves LLM evaluation from a monolithic view of "confidence" to a decomposed view of sensitivity (what the model knows) and monitoring (how well it knows what it knows).
Human-AI Collaboration: Understanding that a model's confidence signal may be "uninformative" despite high accuracy is crucial for building trust and effective human oversight loops.

In conclusion, the paper argues that Meta- $d'$ and the M-ratio are essential additions to the LLM evaluation toolkit, revealing structural inefficiencies in confidence signals that standard calibration metrics completely obscure.