💬 NLP

Diagnosing Correctness Probes under Self-Judgement Confounding

This paper demonstrates that hidden-state readouts in language models often capture the model's self-judgement rather than objective correctness, as evidenced by the superior cross-domain transferability of self-judgement-associated directions compared to those associated with actual correctness.

Original authors: Yi-Long Lu

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi-Long Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a robot is telling the truth. You ask it a question, it gives an answer, and then you have to decide: is the answer actually correct, or is the robot just confident that it's correct? This is the heart of a new field of science called "mechanistic interpretability," where researchers try to peek inside the "brain" of large language models (LLMs) to see what they are really thinking. These models are like giant digital oracles that can write poems, solve math problems, and chat about history, but they sometimes make up facts (a problem known as "hallucination"). Scientists have discovered that if you look at the model's internal electrical signals (called "hidden states") at the exact moment it finishes an answer, you can often predict whether that answer is right or wrong. It's like having a lie detector built into the robot's mind. But here's the tricky part: usually, when a robot gives a wrong answer, it also thinks it's wrong. And when it gives a right answer, it usually thinks it's right. Because the robot's internal "truth signal" and its "self-confidence signal" usually agree, it's been hard to tell if the robot is actually detecting the truth, or if it's just detecting its own opinion.

This paper, titled "Diagnosing Correctness Probes under Self-Judgement Confounding," tackles that exact confusion. The researchers, led by Yi-Long Lu, decided to build a special test where the robot's "truth" and its "self-confidence" disagree. They created scenarios where the model gives a mathematically correct answer but then says, "No, that's wrong," and scenarios where it gives a wrong answer but says, "Yes, that's right." By forcing these two signals to fight each other, they could finally see which signal was actually being picked up by the "lie detector" inside the model.

The team tested this on four different large language models, ranging from 7 billion to 14 billion parameters (think of these as the number of connections in the robot's brain). They used a clever method called "factorial contrasts," which is like separating a mixed smoothie back into its original fruit and milk to see which flavor is which. They found that while the internal signals contained information about both truth and confidence, the most reliably transferable component was the one preserving the robot's self-judgement polarity, not the objective correctness.

Here is the surprising twist: when the researchers looked at the internal signals of a robot that gave a wrong answer but thought it was right, the "lie detector" gave it a high score, just like it would for a correct answer. Conversely, when the robot gave a correct answer but thought it was wrong, the detector gave it a low score. In fact, across all the models and tests they ran—including math problems, movie trivia, and general knowledge quizzes—the internal signal that traveled best from one type of question to another was the one that tracked the robot's confidence, not the actual truth.

The paper explicitly notes that while the "truth" signals we see in these models might be pure objective measures of factual correctness, the evidence shows that transferability alone does not establish objective-correctness semantics. Even when the researchers tried to build a detector using only the "correct/incorrect" labels and ignored the robot's self-judgement entirely, the detector still ended up following the robot's confidence. This suggests that the "truth" we see in these models might be more about how the model feels about its own output than about the actual facts. The authors conclude that just because a signal can be transferred from one task to another doesn't mean it represents the truth; it might just represent the model's own opinion. So, while we can peek inside the robot's brain to see if it's confident, we can't yet be sure if that confidence means it's telling the truth. The study suggests that the most reliable thing we can read from these models is their own self-assessment, not an objective reality check.

Technical Summary: Diagnosing Correctness Probes under Self-Judgement Confounding

Problem Statement

Hidden-state readouts in Large Language Models (LLMs) can predict whether model outputs are factually correct. However, a critical semantic ambiguity exists: objective correctness (OC)—whether an answer aligns with external ground truth—typically correlates strongly with the model's own self-judgement (SJ)—whether the model believes its answer is correct. Because these variables usually agree, conventional correctness-labelled contrasts (e.g., comparing correct vs. incorrect responses) often conflate the neural correlates of external truth with the model's internal evaluative stance.

The central question is whether a transferable "truth direction" extracted from hidden states represents the relation between an answer and external ground truth, or merely the model's own assessment of that answer. Previous work has shown that truth-related directions can transfer across tasks and logical transformations, but reusability alone does not identify the specific variable a readout represents. This ambiguity is particularly problematic when OC and SJ disagree, as standard probes may inadvertently learn to predict the model's confidence or endorsement rather than factual accuracy.

Methodology

Experimental Design: The Conflict Set

To disentangle OC and SJ, the authors construct a "conflict set" by crossing these two binary variables, creating four response types:

A (1,1): Objectively Correct, Self-Endorsed.
B (1,0): Objectively Correct, Self-Rejected (False Negative).
C (0,1): Objectively Incorrect, Self-Endorsed (False Positive).
D (0,0): Objectively Incorrect, Self-Rejected.

The critical diagnostic comparison is between B and C. An OC-associated direction should rank B (correct) above C (incorrect). Conversely, an SJ-associated direction should rank C (endorsed) above B (rejected).

Factorial Decomposition

The authors propose a factorial decomposition of the standard mixed contrast ( $W_{mix} = \mu_A - \mu_D$ ) into two orthogonal main effects:

$W_{meta}$ (SJ-associated): $[(\mu_A - \mu_B) + (\mu_C - \mu_D)] / 2$ . This captures the marginal effect of the model's self-judgement.
$W_{truth}$ (OC-associated): $[(\mu_A - \mu_C) + (\mu_B - \mu_D)] / 2$ . This captures the marginal effect of objective correctness.

Under a linear associative model, $W_{mix}$ is the sum of these two components. The study evaluates which component preserves its semantic polarity (ranking) when transferred across domains.

Data and Models

Models: Four instruction-tuned LLMs up to 14B parameters (Qwen2.5-7B/14B, Llama-3.1-8B, OLMo-3-7B).
Domains:
- Source: Mathematical reasoning (Math) and Factual recall (Movies).
- Target (Out-of-Distribution): MMLU (4-choice) and Binary TruthfulQA.
Procedure:
1. Generate multiple responses per question.
2. Filter for "strict pairs": questions where the model produces at least one correct and one incorrect response, and both are judged with high confidence ( $p_{judge} \ge 0.7$ or $\le 0.3$ ).
3. Extract hidden states at the final answer token.
4. Fit directions on source domains and evaluate on target domains without re-fitting (zero-target-fitting).
5. Measure performance using Area Under the Curve (AUC) for ranking conflict pairs (B vs. C).

Key Results

1. Conventional Probes Follow Self-Judgement

In Experiment 1, the standard mixed direction ( $W_{mix}$ ) was evaluated on B/C conflicts. Contrary to the expectation that it would prioritize objective correctness, $W_{mix}$ consistently ranked C (wrong/self-endorsed) above B (correct/self-rejected). This held true even for an "OC-only" control direction fitted without any SJ labels, suggesting that standard correctness probes are dominated by the model's evaluative stance rather than ground truth.

2. Asymmetric Transferability

In Experiment 2B (cross-domain transfer), the authors tested whether $W_{meta}$ and $W_{truth}$ preserve their expected rankings in target domains:

$W_{meta}$ (SJ-associated): Successfully transferred across all models and directions (Math $\leftrightarrow$ Movies, and to MMLU/TruthfulQA). It consistently ranked C above B with AUCs significantly above chance (0.57–0.93).
$W_{truth}$ (OC-associated): Failed to transfer. In nearly all conditions, the OC-associated direction ranked C above B (AUC < 0.5), effectively reversing the expected ordering. It did not preserve the "correctness" semantics in the target domain.

3. Robustness and Controls

The findings were robust across multiple controls:

Layer Depth: The transfer asymmetry emerges in middle-to-late layers (normalized depth 0.40–0.80).
Null Controls: Random directions and label-shuffled controls did not replicate the pattern.
Confounding Factors: The results persisted after controlling for response token count, response log-probability, and answer-option identity in forced-choice tasks.
OOD Generalization: The SJ-associated signal transferred to MMLU and TruthfulQA without any direction fitting on those specific datasets.

Key Contributions

Semantic Diagnosis of Readouts: The paper provides a method to distinguish whether a transferable readout represents objective truth or the model's self-evaluation by exploiting cases where these variables conflict.
Identification of Confounding: It demonstrates that "truth directions" found in prior literature are often confounded by the model's self-judgement. When OC and SJ disagree, the signal that transfers is the one associated with the model's endorsement, not external correctness.
Factorial Decomposition Framework: The introduction of $W_{meta}$ and $W_{truth}$ as distinct, estimable components offers a rigorous way to separate these semantic signals in hidden state analysis.

Significance and Claims

The paper concludes that transferability alone is insufficient to establish objective-correctness semantics in LLM readouts. While a direction may generalize across tasks, it may be tracking the model's internal confidence or evaluative stance rather than factual truth.

Interpretation: The most reliably transferable component in correctness probes is the one associated with Self-Judgement (SJ). This suggests that models possess a robust internal representation of their own certainty or "endorsement" of an answer, which is distinct from and often overrides the representation of objective ground truth in conflict scenarios.
Implication: Claims that LLMs have "privileged introspective access" or that specific hidden states encode "truth" must be re-evaluated. If a probe predicts correctness but fails when the model is confidently wrong, it is likely measuring the model's belief in its own output rather than the output's factual validity.
Scope: The authors note that their diagnostic sample is limited to high-confidence judgements and specific instruction-tuned models. They do not claim to have solved the problem of truth detection but rather to have diagnosed a fundamental ambiguity in current probing methodologies. The findings suggest that future work on truthfulness must explicitly account for and separate the model's self-evaluative signals from objective correctness.