💬 NLP

Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction

This study validates a three-tier classification screen for LLM confidence signals by demonstrating that models labeled "Valid" significantly outperform "Invalid" models in selective prediction tasks, with the screen accounting for 47% of the variance in performance.

Original authors: Jon-Paul Cacioli

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jon-Paul Cacioli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of fortune tellers (Large Language Models) to help you make important decisions. You ask them to predict the future, but you also ask them to rate their own confidence: "How sure are you that this prediction is right?"

Some fortune tellers are honest and accurate. When they say, "I'm 90% sure," they are usually right. Others are chaotic liars. When they say, "I'm 90% sure," they are actually wrong.

The problem is: How do you know which fortune teller is lying before you let them make decisions for you?

This paper introduces a "Lie Detector Test" for AI confidence. It's a checklist that sorts these AI models into three groups:

Valid: The honest ones. Their confidence scores actually mean something.
Invalid: The liars. Their confidence scores are random or backwards.
Indeterminate: The ones who are too confusing to tell yet.

Here is how the paper proves this test works, explained through simple analogies.

1. The Big Idea: The "Selective Prediction" Game

To test if an AI's confidence is real, the researchers played a game called Selective Prediction.

Imagine you have a bag of 100 riddles. The AI solves them all. Then, the AI says, "I'm only going to show you the answers I'm most confident about."

If the AI is Valid: When it shows you its top 10% most confident answers, those answers should be almost perfect. It successfully filtered out the mistakes.
If the AI is Invalid: When it shows you its top 10% most confident answers, it might actually be showing you its worst mistakes. It's like a chef who says, "I'm going to serve you my best dishes," but then serves you burnt toast.

2. The "Lie Detector" (The Screen)

Before playing the game, the researchers ran the AI models through a Validity Screen. This screen looks at specific patterns in how the AI answers, similar to how a psychologist looks for signs of faking a test.

The "Valid" Group: These models passed the screen. When they played the "Selective Prediction" game, they did great. They successfully filtered out their errors.
The "Invalid" Group: These models failed the screen. When they played the game, they did terribly. In one shocking case (a model named DeepSeek-R1), the more confident the model was, the more wrong it was. It was a "catastrophic inversion." If you trusted its confidence, you would be led straight into a trap.

3. The Results: Does the Test Work?

The researchers found a massive difference between the groups:

The "Valid" models were like skilled archers. When they picked their best shots, they hit the bullseye.
The "Invalid" models were like drunk archers. When they picked their "best" shots, they missed the target entirely, sometimes even shooting backwards.

The test was so good that it could predict how well the AI would perform in this game 47% of the time just by looking at its confidence patterns. That is a huge amount of predictive power in the world of AI.

4. The "Indeterminate" Zone

Some models were weird. They weren't clearly lying, but they weren't clearly honest either. Their confidence was like a radio with static—you could hear a signal, but it was too fuzzy to trust. The researchers put these in the "Indeterminate" bucket.

The Lesson: If an AI falls here, don't use it for life-or-death decisions yet. It's a "maybe," not a "yes."

5. Why This Matters for You

Imagine you are building a self-driving car or a medical diagnosis tool. You want the AI to say, "I'm not sure about this, so I'll ask a human for help" (this is called abstention).

If you use an Invalid AI, it will say "I'm not sure" about the easy cases and "I'm 100% sure" about the dangerous mistakes. Your car might drive off a cliff because the AI was confidently wrong.
If you use the Validity Screen first, you can filter out the liars. You only deploy the models that have proven they can tell the difference between a guess and a fact.

The Bottom Line

This paper is a warning label and a quality control check. It says: "Don't just trust an AI because it sounds confident. Run it through this simple test first."

If the AI passes the test, you can use its confidence to make smart decisions (like ignoring its low-confidence answers). If it fails, its confidence is a lie, and using it could be dangerous.

In short: The paper gives us a way to separate the honest experts from the confident fools before we let them drive the car.

1. Problem Statement

Large Language Models (LLMs) often provide confidence scores (e.g., "KEEP" vs. "WITHDRAW" or "BET" vs. "NO BET") intended to guide downstream deployment strategies like selective prediction (abstaining from low-confidence answers to improve accuracy). However, these confidence signals are frequently uninformative, miscalibrated, or even inverted (where high confidence correlates with incorrect answers).

Previous work (Cacioli, 2026d, 2026e) introduced a psychometric validity screening protocol (based on clinical scales like PAI/MMPI-3) to classify LLM confidence data into three tiers: Invalid, Indeterminate, and Valid. A critical gap remained: the screening thresholds had not been calibrated against actual deployment outcomes. It was unknown whether a model classified as "Valid" by the screen would actually perform better in selective prediction tasks, or if "Invalid" models would fail catastrophically.

2. Methodology

Dataset and Models

Models: 20 frontier LLMs from 7 families (Anthropic, OpenAI, Google, Qwen, DeepSeek, Zhipu).
Data: 524 items per model across 6 cognitive tracks (Learning, Metacognition, Social Cognition, Attention, Executive Function, Prospective Regulation), totaling 10,480 observations.
Inputs: Binary correctness labels and binary confidence probes (KEEP/WITHDRAW and BET/NO BET).

Validity Screening Protocol

The study applied the validity_screen.py protocol to aggregate data, classifying models based on three core indices:

L (Lie Scale): Proportion of items kept.
Fp (Infrequency): Frequency of rare response patterns.
RBS (Response Bias Scale): Bias in withdrawing items.
Structural/Diagnostic: TRIN and the correlation $r(\text{confidence}, \text{correct})$ .

Classifications:

Invalid (n=3): DeepSeek-R1, Gemini 3.1 Pro, Qwen 80B Think.
Indeterminate (n=3): GPT-5.4 nano, Gemma 3 12B, Gemma 3 1B.
Valid (n=14): All others showing significant positive correlation between confidence and correctness.

Criterion: Selective Prediction

The study used Type 2 AUROC (Area Under the Receiver Operating Characteristic curve) as the primary criterion. This measures the model's ability to discriminate between correct and incorrect items based on its confidence score.

Secondary Metrics: Selective gain (accuracy improvement at specific coverage thresholds, e.g., 70%, 50%), and Risk-coverage curves.
Validation Strategy:
- Concurrent Validation: Comparing tier classifications against metrics computed on the same dataset.
- Split-Half Cross-Validation: 1,000 random 50/50 splits where the screen was trained on one half and AUROC tested on the other to assess generalizability within the sample.

3. Key Results

Tier Performance Differences

The validity screen successfully predicted selective prediction performance with a large effect size:

Valid Models: Mean Type 2 AUROC = 0.624 (95% CI [0.604, 0.647]). All 14 models exceeded chance (0.5).
Invalid Models: Mean Type 2 AUROC = 0.357 (95% CI [0.031, 0.522]).
Indeterminate Models: Mean Type 2 AUROC = 0.554.
Statistical Significance: The difference between Valid and Invalid groups is massive ( $d = 2.81$ , $p = .002$ ). The tiers order monotonically: Invalid < Indeterminate < Valid.
Variance Explained: The three-tier classification accounts for 47.0% of the variance in AUROC ( $\eta^2 = .470$ ).

Specific Failure Modes

DeepSeek-R1 (Invalid): Exhibited catastrophic inversion. It kept only 18.1% of items, but those kept were 25.3% correct, while withdrawn items were 98.6% correct. At 10% coverage, its accuracy dropped to 11.3% (from a baseline of 85.3%).
Gemini 3.1 Pro & Qwen 80B Think (Invalid): Exhibited blanket confidence. They kept >96% of items regardless of correctness, resulting in near-zero selective gain and AUROC near 0.5.
Gemma 3 1B (Indeterminate): Showed ambiguous signals (AUROC 0.483, below chance but not statistically inverted), correctly flagged as Indeterminate rather than Invalid due to insufficient evidence for inversion.

Cross-Validation and Stability

Split-Half CV: Across 1,000 splits, the median effect size remained robust ( $d = 1.77$ ), with $P(d > 0) = 1.0$ . The effect never reversed, though it attenuated due to reduced sample size.
Domain Specificity: While Valid models generally performed well, some showed domain-specific failures (e.g., Sonnet 4.6 had AUROC < 0.55 in Prospective Regulation). However, the aggregate screen remained predictive.
Within-Tier Analysis: Among Valid models, the "L" score (proportion kept) negatively correlated with AUROC ( $\rho = -0.801$ ). Models that kept nearly everything (High L) had less variance to exploit for selective prediction, suggesting a "measurement ceiling" effect.

4. Key Contributions

First Concurrent Criterion Validation: This is the first study to empirically link psychometric validity indices for LLM confidence to downstream operational metrics (selective prediction).
Construct Validity Bridge: It demonstrates that clinical psychometric methodologies (derived from PAI/MMPI-3) are transferable to ML deployment, validating that the "Validity" construct is not just face-valid but operationally predictive.
Mechanistic Diagnosis: The screen does not just flag "bad" models; it identifies why they fail (e.g., catastrophic inversion vs. blanket confidence), allowing for targeted debugging.
Operational Guidance: It establishes a "screen before you deploy" protocol. Using confidence for abstention without this screen risks selecting errors (as seen with DeepSeek-R1) or gaining no benefit (as seen with blanket confidence models).

5. Significance and Limitations

Significance

Safety Critical: In safety-critical applications (e.g., medical diagnosis, legal advice), relying on unvalidated confidence scores can lead to systems that abstain on easy questions and answer hard ones incorrectly. This screen provides a low-cost, interpretable filter to prevent such failures.
Interpretability: It moves beyond black-box calibration metrics, offering a structured framework to understand model behavior through established psychometric lenses.

Limitations

Concurrent Nature: The validation uses the same dataset for screening and testing. While split-half cross-validation mitigates this, it does not prove generalization to entirely new benchmarks (e.g., MMLU, GSM8K) or model generations.
Sample Size: The study covers 20 models. While diverse, the small sample limits formal mixed-effects modeling for family-level clustering.
Elicitation Sensitivity: Results are specific to binary KEEP/WITHDRAW and BET/NO BET probes. Different elicitation formats might yield different validity profiles.
High-Accuracy Ceiling: For models with >95% accuracy, the "Invalid" flag (due to low cell counts in the "incorrect + WITHDRAW" category) may be a data artifact rather than a true signal of invalidity, requiring larger item sets for stable classification.

Conclusion

The paper confirms that a psychometric validity screen can effectively predict whether an LLM's confidence signal is usable for selective prediction. Valid models show positive selective gains, while Invalid models show flat or inverted performance. The screen serves as a necessary prerequisite for any deployment strategy relying on model confidence to make decisions about abstention or routing.