What Does a Factual-Error Detector Really Measure? Question Difficulty and Model Uncertainty Before the Answer Begins

Technical Summary: What Does a Factual-Error Detector Really Measure?

Problem Statement
Factual-error detectors aim to predict whether a Large Language Model's (LLM) answer is incorrect based on the model and its output. However, high detection accuracy alone does not reveal the source of the predictive signal. A detector might simply identify questions that are difficult for many models, capture properties of the generated text, or reflect output uncertainty, rather than measuring the model's internal assessment of its own knowledge. Existing baselines often fail to distinguish between model-specific predictive information and general "question difficulty." Furthermore, claims regarding internal signals (e.g., layer-wise activations) often lack rigorous cross-model validation or fail to account for the fact that entropy measures dispersion, not truth.

Methodology
The authors propose a rigorous evaluation framework centered on two main components: a "Leave-One-Model-Out" (LOMO) baseline and a prospective audit of scalar readouts across diverse model lineages.

1. LOMO Baseline: Instead of relying on internal states, the baseline calculates the fraction of other available models that answered a specific question incorrectly, excluding the target model. This score depends solely on question difficulty and panel composition, not the target model's answer or internal states.
2. Main Panel Analysis (TriviaQA): The study audits short factual answers from 18 open-weight models (17 retained for analysis). From 42,636 possible model–question pairs, 18,850 survived extraction. The authors compare a favorably selected detector against the LOMO baseline using AUROC. They also employ continuous cross-fitting (polynomial LOMO) and exploratory controls using regenerated answer text to check for overlap between detector signals and answer content.
3. Prospective Audit (Natural Questions): A separate, consistency-filtered audit was conducted on 18 models (12 lineages). To ensure robustness, only pairs where 10 sampled responses received the same correctness verdict were retained. The study tested 24 scalar measures (including entropy and probability margins at 12 relative depths) against a prespecified empirical rule: a Cohen's $d$ effect size $\ge 0.5$ in every model, consistent direction, and statistical significance.
4. Scope Controls:
   • Decoding: Comparisons between sampling and greedy decoding to determine if associations rely on random token selection.
   • Error Types: Analysis of "shared errors" (occurring in $\ge 3$ model families) versus "idiosyncratic errors" to test if error clustering correlates with entropy.
   • Task Dependence: Evaluation across Natural Questions (NQ), TriviaQA, and BioASQ.

Key Contributions

1. Operational Evaluation against Panel Difficulty: The paper establishes LOMO as a strong, detector-independent baseline. It argues that claims of model-specific predictive information must demonstrate gains beyond this baseline.
2. Technically Validated Audit: The study provides a strict, pre-registered audit of 24 scalar readouts across 18 models, requiring exact native-output reconstruction and a minimum effect size in every model.
3. Scope Controls: The inclusion of greedy decoding controls, shared-error analysis, and task-dependent comparisons to delimit the conditions under which findings hold.

Results

• LOMO vs. Detectors: In the main panel, the LOMO baseline significantly outperformed the selected standalone detector. The median AUROC for LOMO was 0.855 versus 0.784 for the detector. The median paired difference was +0.064, positive in all 17 models ($p = 2.9 \times 10^{-4}$).
• Combined Predictors: While LOMO alone was superior, adding detector features (specifically entropy and wider features) to LOMO provided positive incremental gains, suggesting complementary predictive value.
• Entropy and Depth: Among 24 measures in the NQ audit, only native-output entropy (at the final layer, $u_{11}$) met the prespecified empirical criterion ($|d| \ge 0.5$ in all 18 models). No earlier-layer measure passed.
  • The mean effect size for native-output entropy was $d = 1.0515$, with a minimum of 0.6512.
  • The closest pre-output measure (entropy at $u_{10}$) had a minimum effect of 0.1495, failing the threshold.
• Difficulty Sensitivity: When controlling for question difficulty via stratification, the effect of entropy weakened. Only 16/18 models exceeded the $d=0.5$ threshold (e.g., Mistral dropped to 0.102), indicating the signal is sensitive to the sample composition.
• Greedy Decoding: The association between entropy and error persisted under greedy decoding, though the magnitude varied. This suggests the signal is not solely an artifact of sampling from the distribution's tail.
• Shared vs. Idiosyncratic Errors: Shared errors (groupable by meaning) exhibited lower first-content-token entropy than idiosyncratic errors in 17/17 models. This implies that when models converge on the same wrong answer, uncertainty is low, challenging the notion that high entropy is the sole indicator of hallucination.
• Text Controls: Exploratory controls using regenerated text suggested that much of the apparent gain from activation-based detectors overlaps with information contained in the answer text itself.

Significance and Claims
The paper concludes that factual-error detectors should be evaluated against panel question difficulty (LOMO) rather than in isolation. The findings support entropy as a warning signal for potential errors in selected short-answer populations, but not as a measure of truth.

• Limitations of Entropy: Entropy measures dispersion over next tokens, not correctness. A dominant wrong candidate can keep entropy low, while competing surface forms can raise it.
• No Universal Mechanism: The study does not establish a shared causal mechanism or a universal threshold. The success of native-output entropy is specific to the selected, consistency-filtered population.
• Modest Claims: The authors explicitly state that their findings concern the selected populations and do not generalize to long-form generation, retrieval-augmented systems, or production traffic. The evidence supports careful measurement of uncertainty as a trigger for verification or abstention, but low entropy alone cannot justify automatic trust.

In summary, the paper demonstrates that while simple internal signals like output entropy can correlate with errors, they are heavily confounded by question difficulty and text properties. The most robust predictor of factual error in this study was the collective performance of other models on the same question, not the internal state of the target model.