Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals

1. The Problem: Unscreened Confidence Signals

Current Large Language Model (LLM) evaluation practices for metacognition, calibration, and selective prediction (e.g., abstention, routing) operate on a critical, unverified assumption: that the model's confidence signal carries item-level information regarding correctness.

• The Gap: Researchers compute metrics like AUROC, Expected Calibration Error (ECE), or selective prediction gains without first verifying if the confidence signal is dominated by response sets (e.g., blanket confidence, blanket withdrawal, or fixed responding) rather than actual monitoring of correctness.
• The Consequence: If a signal is invalid (e.g., a model is blindly confident on all errors), downstream metrics become misleading. For instance, an invalid model might yield an AUROC near 0.50 (chance) or even negative selective prediction gains, yet these results are often reported as "findings" rather than artifacts of uninterpretable data.
• Analogy: The paper draws a parallel to clinical psychology (MMPI-3, PAI), where substantive interpretation is strictly forbidden if validity scales indicate the patient is not responding honestly or is using a response set. LLM evaluation lacks this mandatory "validity screening" step.

2. Methodology: A Two-Stage Protocol

The paper proposes a portable, benchmark-agnostic protocol grounded in classical psychometrics and clinical assessment traditions. It separates evaluation into two distinct stages:

Stage A: Validity Screening (The Gatekeeper)

This stage determines if the confidence signal is interpretable. It relies on a single $2 \times 2$ contingency table derived from item-level correctness (Correct/Incorrect) and binarized confidence (High/Low).

• Core Indices:
  1. $L$ (Under-reporting): $P(\text{High Confidence} | \text{Incorrect})$. Measures "blanket confidence" on errors.
  2. $F_p$ (Over-reporting): $P(\text{Low Confidence} | \text{Correct})$. Measures "blanket withdrawal" on correct items.
  3. $RBS$ (Response Bias Scale): $F_p - (1 - L)$. Detects inverted monitoring (low confidence on correct items > low confidence on incorrect items).
• Structural Indicator:
  • $TRIN$: Proportion of responses matching the modal decision (fixed responding). High values indicate low variance but do not automatically invalidate the signal.
• Diagnostic Statistic:
  • $r(\text{confidence, correct})$: Point-biserial correlation between confidence and correctness. This is not algebraically redundant with the indices and provides a direct measure of item-level dependence.
• Three-Tier Classification System:
  1. Invalid: Clear threshold violation (e.g., $L \ge 0.95$, $F_p \ge 0.50$, or $RBS > 0$ with CI excluding zero). Action: Do not interpret substantive metrics.
  2. Indeterminate: Point estimate violates a threshold, but the Confidence Interval (CI) includes valid values, or the signal is borderline (e.g., $RBS$ near 0). Action: Metrics may be computed but must be flagged as potentially attenuated.
  3. Valid: No threshold violations. Action: Proceed to Stage B.

Stage B: Substantive Analysis

Only performed if Stage A classifies the signal as Valid. This stage computes standard metacognitive metrics (AUROC, meta-$d'$, calibration curves, selective prediction gains).

3. Key Contributions

1. Portable Protocol Architecture: A standardized, two-stage workflow that decouples "is the data interpretable?" from "what does the data say?"
2. Minimal Index Set: A computationally efficient set of indices ($L, F_p, RBS, TRIN, r$) derived from a single contingency table, avoiding the need for complex internal states or logits.
3. Three-Tier Classification: Moves beyond binary pass/fail to include an "Indeterminate" tier, borrowing from the Jacobson-Truax Reliable Change Index to handle measurement error and borderline cases honestly.
4. VRS Table (Validity Report for Screening): A mandatory reporting template ensuring transparency in how validity was assessed for any study reporting confidence metrics.
5. Failure Mode Demonstration: Empirical proof that unscreened invalid models produce misleading downstream metrics (e.g., AUROC $\approx 0.50$, negative selective gains), creating "false assurance" in safety-critical systems.

4. Results & Validation

The protocol was validated on 20 frontier LLMs across 524 items (6 cognitive domains) and cross-validated on external datasets.

• Classification Outcomes (Derivation Sample):
  • Invalid (4 models): Included DeepSeek-R1 (massively inverted monitoring, $r = -0.798$) and Gemini 3.1 Pro (blanket confidence on errors, $L=0.967$). These models showed AUROC near chance or worse.
  • Indeterminate (2 models): Models with borderline indices (e.g., GPT-5.4 nano, Gemma 3 12B).
  • Valid (14 models): Showed mean $r(\text{confidence, correct}) = 0.18$ (15/16 significant).
• Statistical Significance: The difference in item-sensitivity ($r$) between Valid and Invalid profiles was large ($d = 2.48, p = .001$).
• Stability Analysis:
  • Clear Invalid/Valid cases stabilize at 100–150 items.
  • High-accuracy models with few errors (<26 errors) show inherent instability in $L$ estimation due to wide Wilson CIs.
• Cross-Benchmark Validation:
  • Yang et al. (2024) Dataset: Successfully identified 1 invalid model out of 11 across 10 benchmarks.
  • Probe Format Sensitivity: Models classified as Invalid on binary probes (KEEP/WITHDRAW) often shifted to Valid when using verbalized confidence (0–100). This confirms that validity is a property of the model-probe-task interaction, not an intrinsic model property.
  • Failure Mode: Unscreened invalid models produced AUROC values of 0.499–0.509 and zero selective prediction gain, demonstrating that without screening, researchers cannot distinguish genuine metacognition from noise.

5. Significance and Implications

• Methodological Necessity: The paper argues that reporting confidence-based metrics without validity screening is methodologically incomplete. It prevents the field from building safety-critical systems (e.g., autonomous agents, medical triage) on "hallucinated" confidence signals.
• Clinical Parallel: It successfully transfers decades of clinical psychometric rigor (MMPI-3/PAI) to AI evaluation, treating LLM confidence as a behavioral signal subject to response bias (e.g., RLHF-induced over-confidence).
• Practical Impact:
  • Deployment Safety: Prevents systems from relying on models that are "confidently wrong" or "inverted."
  • Research Reproducibility: The VRS Table ensures that validity conditions are documented, allowing for fair comparison across studies.
  • Training Feedback: Identifies specific failure modes (e.g., blanket confidence vs. blanket withdrawal) that can guide better prompt engineering or RLHF tuning.

In conclusion, "Screen Before You Interpret" establishes that validity screening is not an optional diagnostic but a prerequisite for any meaningful analysis of LLM confidence. It provides the tools to distinguish between a model that truly knows what it knows and one that is merely exhibiting a response artifact.