💬 NLP

Masked by Consensus: Disentangling Privileged Knowledge in LLM Correctness

This paper investigates whether large language models possess privileged internal knowledge about answer correctness, revealing that while self-representations offer no advantage on standard benchmarks due to high inter-model agreement, they do provide a domain-specific edge in factual tasks over peer models when evaluated on disagreement subsets, unlike in math reasoning.

Original authors: Tomer Ashuach, Liat Ein-Dor, Shai Gretz, Yoav Katz, Yonatan Belinkov

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tomer Ashuach, Liat Ein-Dor, Shai Gretz, Yoav Katz, Yonatan Belinkov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess whether a student's answer on a test is correct. You have two ways to do this:

The Outsider: You look at the student's written answer and the question itself.
The Insider: You peek inside the student's brain (their hidden thoughts) while they are thinking.

For a long time, researchers wondered: Does the student have a secret "gut feeling" inside their brain that tells them if they are right, which an outsider can't see? This secret feeling is called "privileged knowledge."

This paper investigates whether Large Language Models (LLMs)—the AI brains behind chatbots—have this kind of secret internal knowledge about their own answers.

The Problem: The "Consensus Mask"

The researchers found a tricky problem in previous studies. Imagine a classroom where 90% of the students agree on the answer to every question. If you are an outsider trying to guess if Student A is right, you can just look at what Student B wrote. Since they usually agree, you can guess Student A's result just by looking at Student B's brain.

Because the models agree so often, the "outsider" view looks just as good as the "insider" view. It was like the secret internal signal was being masked by consensus. The researchers realized that if everyone agrees, you can't tell if the secret signal actually exists.

The Solution: The "Disagreement" Test

To fix this, the researchers created a special test using only the questions where the models disagreed.

Scenario: Model A thinks the answer is "Yes." Model B thinks the answer is "No."
The Test: In these tricky cases, Model B's brain cannot help you guess Model A's result. If you can still guess Model A's correctness better by looking inside Model A's brain than by looking at Model B's brain, then Model A truly has a secret internal signal.

The Big Discovery: It Depends on the Subject

When they ran this "disagreement test," they found a surprising split between two types of tasks:

1. Factual Knowledge (The "Memory" Test)

Examples: "Who was the first president?" or "What is the capital of France?"
Result: Yes, the secret signal exists.
The Analogy: Think of this like a librarian. When a librarian pulls a specific book off a shelf, they have a unique, internal feeling of "I know this fact." Even if another librarian disagrees, the first librarian's internal state holds a special signal that says, "I am retrieving this specific memory." The researchers found that the AI's internal brain waves showed a clear advantage in predicting correctness for these facts, something outsiders couldn't see.

2. Mathematical Reasoning (The "Logic" Test)

Examples: Solving a complex algebra problem or a word problem about savings accounts.
Result: No, the secret signal is missing.
The Analogy: Think of this like a mechanic fixing a car. If two mechanics look at the same broken engine, the difficulty is in the structure of the engine, not in a secret memory. The AI doesn't have a special "I know I'm right" feeling about the math. The clues that tell you if the math is right are all visible on the surface (the question and the logic steps). An outsider looking at the problem can predict the AI's success just as well as the AI can predict it itself.

Where Does the Secret Signal Hide?

The researchers also looked at where in the AI's "brain" (its layers of processing) this secret signal appears.

For Facts: The signal starts small in the early layers (where the AI reads the words) and grows stronger as the information moves deeper into the brain. It's like a memory retrieval process that builds up as the AI thinks about it.
For Math: The signal never really appears. The "difficulty" of the math problem is visible at every single layer, from the start to the finish.

Summary

The paper concludes that AI models do have privileged knowledge, but only for facts they have memorized. They know when they are recalling a memory. However, they do not have this secret knowledge for math or logic; in those cases, their confidence is just as visible to the outside world as it is to themselves.

The reason previous studies missed this was that the models agreed too often, hiding the difference between "knowing a fact" and "solving a problem." By looking only at the moments they disagreed, the researchers finally uncovered the truth.

1. Problem Statement and Context

The central question of this article is whether large language models (LLMs) possess privileged knowledge regarding the correctness of their own responses. In other words, do a model's internal states contain specific signals indicating whether its response will be correct, signals that would be inaccessible to an external observer (another model)?

Theoretical Context: Inspired by the philosophical concept of "epistemic privilege," where an agent has special access to its own internal states.
Contradictory State of the Art:
- Some studies suggest that LLMs can predict their own correctness via linear probes on their hidden states.
- Other recent works (Chi et al., Seo et al., Xiao et al.) challenge this idea, suggesting that probes merely detect retrieval patterns or public features of the question, and that external models can predict correctness as well as the model itself.
Article Hypothesis: Previous conclusions denying the existence of privileged knowledge might be premature due to a critical confounding factor: inter-model consensus. If models often agree on the correctness of a response, an external model can use its own correctness patterns as a proxy to predict the target model's correctness, thereby masking the absence or presence of unique internal signals.

2. Methodology

The authors propose a rigorous experimental framework to isolate unique internal signals of models.

A. Definition of Privileged Knowledge

The hidden state $h$ of a model is decomposed into two latent components:

$z_{public}$ : Features inherent to the question (domain, entity types) accessible to any model.
$z_{private}$ : Model-specific internal states (success of memory retrieval, confidence in reasoning). This component constitutes the "privileged knowledge."

B. Probe Configuration

To test for the existence of $z_{private}$ , the authors train classifiers (probes) to predict the correctness ( $y \in \{0, 1\}$ ) of a response:

Self-Probe: Trained on the internal representations of the target model ( $M_{target}$ ) to predict its own correctness.
External-Probe: Trained on the representations of a different model ( $M_{source}$ $M_{so u r ce}$ ) to predict the correctness of the target model.
- Cross-Model: Uses a comparable-sized LLM.
- Embedding-Model: Uses an embedding model.

C. The Metric: The "Premium Gap"

The Premium Gap is defined as the performance advantage (measured by AUC) of a Self-Probe over the best External-Probe.

If the gap is zero, this suggests that correctness is entirely predictable via public features ( $z_{public}$ ).
If the gap is positive and significant, this indicates the existence of privileged knowledge ( $z_{private}$ ).

D. Key Innovation: Disagreement Subsets

To eliminate the bias of consensus, the authors do not test on the entire random dataset but focus on disagreement subsets.

Definition: A set of questions where the target model and the source model produce opposite correctness labels (one is right, the other is wrong).
Reason: In these cases, the external model cannot use its own correctness patterns as a proxy for the target model. If a Self-Probe still outperforms an External-Probe on this subset, it is direct proof of privileged knowledge.

E. Experimental Configuration

Models: Three similarly sized LLMs (Qwen-2.5-7B, Llama-3.1-8B, Gemma-2-9B) and a larger model (Qwen-3-32B) for scalability.
Data: Five datasets covering two domains:
- Factual Knowledge: Mintaka, TriviaQA, HotPotQA.
- Mathematical Reasoning: GSM1K, MATH.
Method: Linear probes (logistic regression) and non-linear probes (MLP) on hidden states (every 5th layer + last layer).

3. Main Results

A. Absence of Premium Gap on Standard Datasets

On full (random) test sets, the authors confirm that the Premium Gap is zero or negligible.

External probes (especially those based on Gemma) equal or surpass internal probes.
This is explained by a high inter-model agreement rate (~80% for facts, ~75% for math), allowing external probes to exploit shared correctness patterns.

B. Emergence of Domain-Specific Privileged Knowledge

When evaluating on disagreement subsets, results diverge radically depending on the domain:

Factual Knowledge (TriviaQA, Mintaka, HotPotQA):
- A statistically significant Premium Gap (approximately 5%) emerges.
- Self-Probes systematically outperform External-Probes.
- Conclusion: Models possess privileged knowledge regarding the correctness of their factual responses, inaccessible to external observers.
Mathematical Reasoning (MATH, GSM1K):
- No Premium Gap is observed, even on disagreement subsets.
- External probes equal or surpass internal probes.
- Conclusion: Correctness in mathematical reasoning does not rely on privileged internal signals but on structural features of the question accessible to all models.

C. Temporal Localization (Layer-wise Analysis)

The authors analyze at what depth of the network these signals appear:

Factual: The Premium Gap is near zero in early layers and grows progressively from middle layers (around layer 10-15) to deep layers. This suggests that privileged knowledge is linked to memory retrieval that accumulates during the forward pass.
Mathematical: No consistent trend is observed; the gap remains zero or negative at all depths.

D. Analysis of Signal Sources ("Lexical-Only" Experiment)

By removing syntax and keeping only named entities and names:

Correctness prediction for factual tasks and MATH remains strong, suggesting that familiarity with concepts is a key factor.
For GSM1K, performance drops to chance level, indicating that mathematical correctness depends on the computational structure of the problem, not surface tokens.

4. Key Contributions

Identification of a Major Confounding Factor: Demonstrates that inter-model consensus masks the potential existence of privileged knowledge in standard evaluations.
Disagreement Methodology: Introduces evaluation on disagreement subsets as a standard method to isolate unique internal signals of models.
Domain Asymmetry: Establishes that privileged knowledge is not a universal property of LLMs but is domain-specific: it exists for fact retrieval but is absent for complex mathematical reasoning.
Architectural Localization: Maps the progressive emergence of these signals in the middle and deep layers of models for factual tasks.

5. Significance and Implications

Reconciliation of Previous Works: This article resolves the conflict between studies claiming LLMs "know" if they are right and those saying the opposite. The truth is nuanced: they have privileged access to factual correctness, but not to reasoning.
Hallucination Detection: The results suggest that model-specific activations may contain error detection signals (hallucinations) that external "black box" tools cannot see, particularly in knowledge tasks.
Limits of Introspection: A model's ability to self-evaluate depends on the nature of the task. For reasoning tasks, self-evaluation via internal signals offers no advantage over external analysis.
Future: Opens the door to intervention experiments (activation steering) to manipulate these correctness signals and improve model reliability.

In summary, the article demonstrates that LLMs do possess a form of "awareness" of their own correctness, but this capacity is masked by consensus in standard evaluations and limited to factual memory tasks, not extending to complex logical reasoning.