LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy

1. Problem Statement

Current evaluation metrics for Large Language Models (LLMs), such as Expected Calibration Error (ECE) and the Brier score, conflate two distinct performance components:

1. Sensitivity (Discrimination): The model's ability to distinguish between correct and incorrect answers.
2. Bias (Criterion): The model's tendency to be overconfident or underconfident (cautious vs. liberal responding).

Standard metrics treat these as a composite score. Consequently, a model that is uniformly wrong but perfectly calibrated to its own (incorrect) confidence cannot be distinguished from a model that discriminates well but is miscalibrated. This lack of decomposition hinders targeted interventions (e.g., knowing whether to retrain a model for better knowledge or simply recalibrate its confidence).

The paper proposes applying Signal Detection Theory (SDT), a framework from psychophysics, to LLMs. While SDT metrics like AUROC are emerging, the full parametric framework (unequal-variance modeling, criterion estimation, and z-ROC analysis) has not been rigorously applied to LLMs. Furthermore, the authors investigate a specific analogy: whether temperature scaling in LLMs functions as a criterion shift (analogous to payoff manipulations in human experiments) without altering sensitivity.

2. Methodology

The study is a pre-registered experiment involving three LLMs: Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3-8B-Base.

• Data: 168,000 confidence-rating trials across 7 temperatures ($T \in \{0.1, 0.3, 0.5, 0.7, 1.0, 1.5, 2.0\}$) and 6,000 forced-choice trials. Datasets included TriviaQA (5,000 questions) and Natural Questions (3,000 questions).
• SDT Mapping:
  • Observer: The LLM.
  • Signal/Noise: Correct vs. Incorrect answers.
  • Evidence Variable: Normalized Log-Probability (NLP) of the generated answer sequence.
  • Criterion Manipulation: Temperature ($T$).
• Analysis Framework:
  • Construction of multi-point Receiver Operating Characteristic (ROC) curves using NLP confidence bins.
  • Fitting of Unequal-Variance Signal Detection (UVSD) models to estimate sensitivity ($d_a$), criterion ($c$), and the variance ratio ($s$).
  • Hypotheses:
    • H1: Temperature acts as a pure criterion shift (AUC remains constant, $c$ changes).
    • H2: SDT reveals hidden structure not visible in ECE.
    • H3: Convergence between confidence-rating $d_a$ and 4-Alternative Forced Choice ($d'_{4AFC}$).
    • H4: Evidence distributions are unequal-variance (z-ROC slope < 1.0).

3. Key Contributions

1. Refutation of the Temperature–Criterion Analogy: The study demonstrates that temperature is not a pure criterion manipulation. Unlike human experiments where the stimulus is fixed, changing temperature in LLMs alters the generative path, thereby changing the answer itself (the stimulus). This results in a simultaneous shift in both sensitivity and criterion.
2. Unequal-Variance Evidence Distributions: LLMs exhibit unequal-variance distributions where the signal (correct answers) has higher variance than the noise (incorrect answers). This is more extreme in instruction-tuned models than in base models or human recognition memory.
3. Diagnostic Power of SDT Decomposition: The study proves that models with identical ECE scores can occupy vastly different positions in sensitivity-bias space, necessitating different interventions (recalibration vs. capability improvement).

4. Key Results

A. Temperature Effects (Rejection of H1)

Contrary to the hypothesis that temperature only shifts the criterion, results showed:

• Sensitivity ($d_a$ and AUC) Increased: As temperature rose from 0.1 to 2.0, AUC increased monotonically (e.g., Llama-3-Base AUC rose from 0.751 to 0.857).
• Criterion Shifted: The criterion ($c$) moved from highly liberal to less liberal.
• Accuracy Decreased: Despite higher sensitivity metrics, actual accuracy dropped significantly (e.g., Llama-3-Base accuracy fell from 59.2% to 33.6%).
• Interpretation: The increase in AUC is a distributional effect of temperature on the NLP evidence variable (spreading the distribution of log-probabilities), not an improvement in factual knowledge. The "stimulus" (the generated answer) changed with the "criterion" (temperature), breaking the analogy to human payoff manipulation.

B. Hidden Structure (Support for H2)

• Models with similar ECE scores (e.g., Llama-3-Instruct and Mistral) occupied distinct positions in $(d_a, c)$ space.
• Example: Mistral had higher sensitivity ($d_a \approx 1.97$) but a more liberal bias, whereas Llama-3-Instruct had lower sensitivity ($d_a \approx 1.39$). ECE alone would mislead evaluators into thinking Llama-3 was better calibrated, while SDT revealed Mistral was the superior discriminator with a policy issue.

C. Unequal Variance (Support for H4)

• z-ROC Slopes: All models exhibited slopes significantly below 1.0, confirming unequal variance ($\sigma_{signal} > \sigma_{noise}$).
  • Instruction-tuned models: Slopes ranged from 0.52 to 0.63 (highly unequal variance).
  • Base model: Slopes ranged from 0.77 to 0.87.
  • Comparison: This is more extreme than human recognition memory (typically $s \approx 0.80$).
• Implication: Instruction tuning appears to amplify the variance difference between well-known and edge-of-knowledge items.

D. Domain Specificity (Support for H5)

• Sensitivity varied significantly across knowledge domains. "Science & Technology" was consistently the hardest domain for all models, a nuance invisible to aggregate ECE metrics.

E. Paradigm Convergence (Partial Rejection of H3)

• While correlations between confidence-rating $d_a$ and forced-choice $d'_{4AFC}$ were positive, the forced-choice paradigm suffered from ceiling effects (97.5% accuracy for instruct models), preventing a precise quantitative test of convergence.

5. Significance and Implications

• Diagnostic Precision: SDT decomposition allows researchers to distinguish between "can't discriminate" (low sensitivity, requiring better training data/architecture) and "discriminates but responds poorly" (wrong criterion, requiring recalibration or temperature tuning).
• Limitations of Current Metrics: Standard calibration metrics (ECE) are insufficient for high-stakes AI deployment because they mask the underlying trade-offs between sensitivity and bias.
• Fluency vs. Truth: The study highlights a "fluency-accuracy dissociation." The NLP evidence variable conflates the correctness of an answer with its fluency. The model's confident incorrect answers are often more fluent than correct answers generated by external sources, masking the true truth signal.
• Theoretical Bridge: This work successfully bridges classical psychophysical theory with modern LLM evaluation, providing a principled framework for understanding model confidence that goes beyond descriptive statistics.

Conclusion

The paper establishes that treating LLMs as signal detectors via the full parametric SDT framework reveals critical structural properties (unequal variance, sensitivity-bias trade-offs) that are invisible to standard calibration metrics. Crucially, it demonstrates that temperature scaling is a complex intervention that alters the evidence distribution itself, not just the decision threshold, challenging the direct application of human psychophysics analogies to generative AI.