🤖 AI

The Knowing-Saying Gap: When Probes See Errors that Confidence Misses

This paper reveals that while linear probes can accurately detect corrupted context in language models, they fail to reliably predict final answer correctness or guide effective real-time interventions, necessitating model-aware and error-type-aware routing strategies rather than a one-size-fits-all monitoring solution.

Original authors: Jyotin Goel, Ipshita Bandyopadhyay, Justin Shenk

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jyotin Goel, Ipshita Bandyopadhyay, Justin Shenk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician perform a card trick. The magician is so smooth, so confident, and so fast that you never suspect a thing. But what if, deep inside the magician's brain, a tiny alarm was screaming, "Wait! I just swapped the wrong card!"? In the world of Artificial Intelligence, specifically Large Language Models (the super-smart chatbots that write code, plan trips, and solve math problems), this is exactly what happens. These models are like the magician: they can talk with total confidence even when they are making a terrible mistake.

Scientists have long hoped that if a model is confused or wrong, it would "stutter" or say, "I'm not sure about this." This idea is called "verbalized confidence." It's like asking the magician, "Are you sure that's the right card?" and hoping they admit their mistake. But there's another way to check: looking at the model's internal "brain waves" (mathematical signals inside the computer) to see if it knows it's messed up, even if it doesn't say so. This paper investigates a scary but fascinating gap: the difference between what a model knows inside its digital brain and what it says out loud.

The Silent Alarm and the Confident Lie

The researchers set up a digital playground to test this. They created 1,400 math problems that required a model to solve a puzzle step-by-step, like a chain reaction. For example, "If I have 5 apples, and I buy 3 more, then I lose half..." The trick was that they secretly planted a mistake in the very first step of the chain. They wanted to see two things:

The Alarm: Could a special detector (called a "linear probe") spot that the first step was wrong just by looking at the model's internal brain activity?
The Prediction: Could that same detector predict that the final answer would be wrong?

They tested five different models, including some of the smartest ones available today, like Llama-3.1-8B and Qwen3-4B.

The Big Discovery: Knowing but Not Saying

The results were a shock. The "alarm" worked perfectly. When the model made a mistake, the detector could spot the error in the model's internal brain waves with nearly 100% accuracy (specifically, an AUROC score of 0.997 for one model). It was as if the model's brain was screaming, "ERROR! ERROR!"

But here is the twist: The model didn't listen to its own alarm.

Even though the internal alarm was blaring, the model's final answer was just as confident as if it were right. The detector that could see the error perfectly was completely useless at predicting whether the final answer would be wrong. It's like having a smoke detector that screams "FIRE!" when you burn toast, but the fire department (the model's final answer) still shows up and says, "Everything is fine, no fire here."

The paper explicitly rules out a few things that people might hope are true:

Confidence isn't a clue: When they asked the models to say how confident they were, the models just gave a binary "Yes" or "No" with no real nuance. A model that was 99% sure was just as likely to be wrong as a model that was 50% sure.
Thinking harder doesn't help: They tried a "thinking mode" where the model was forced to write out its reasoning step-by-step (Chain-of-Thought). You might think this would help the model catch its own mistakes. Instead, it made the model worse at getting the right answer (dropping accuracy from 27.9% to 1.2%) without changing the fact that the internal alarm was still going off. The model knew it was wrong, but talking about it didn't fix it.

Can We Fix It? The "Branch-and-Pick" Strategy

Since the models won't admit they are wrong, the researchers tried to act as a "safety net" for them. They built a system that listens to the internal alarm and tries to fix the mistake before the model finishes its answer. They tested three different ways to intervene:

Reprompt: Just telling the model, "Hey, check your work!" (This didn't work well).
Replace-Prior: Erasing the wrong step and forcing the model to try again from scratch. This was a mixed bag; it saved some wrong answers but accidentally broke some correct ones.
Branch-and-Pick: This was the winner. When the alarm went off, the system didn't just guess; it asked the model to try the next step four different ways (using different levels of randomness). Then, it used the internal alarm to pick the one path that looked the "cleanest" (least likely to be corrupted).

This "Branch-and-Pick" method was the only one that consistently saved wrong answers without breaking correct ones. On one specific model (Llama-3.1-8B), it rescued 4 wrong answers and broke 0 correct ones. However, the paper notes that this isn't a magic bullet for every situation. The success depended heavily on what kind of mistake was made. For example, fixing a "wrong unit" error worked great, but fixing a "wrong percentage" error actually made things worse.

The Takeaway

The paper concludes that we cannot trust a model's words to tell us if it's right. A model can be internally aware of a disaster while confidently delivering a wrong answer. The "Knowing-Saying Gap" is real and dangerous.

The solution isn't to ask the model to be more honest; it's to build a "model-aware" safety system. We need to listen to the internal alarms (the probes) and use smart strategies like "Branch-and-Pick" to catch errors. But we also need to be careful: there is no single fix that works for every type of mistake or every model. The future of safe AI isn't about trusting the chatbot's confidence; it's about building a guardrail that knows when the chatbot is lying to itself.

Technical Summary: The Knowing-Saying Gap in Language Models

Problem Statement

The paper addresses a critical reliability failure in deployed Large Language Models (LLMs), particularly within agentic pipelines where model outputs feed back into future context (e.g., coding agents, multi-step planning). In these settings, a single error in an early step propagates through subsequent context, causing "cascading context corruption." The core problem is a dissociation between internal state and verbal output: models often continue generating fluent, confident text even after their internal computation has been corrupted by an error. This "fluency without fidelity" renders standard monitoring techniques, such as asking the model for verbalized confidence or self-critique, ineffective because the model's internal knowledge of the error does not translate into its surface behavior.

Methodology

The authors investigate this phenomenon using a controlled, contrastive experimental design:

Dataset: A synthetic dataset of 1,400 multi-hop arithmetic traces generated from 500 base problems. The dataset includes "clean" traces and "corrupted" traces where a specific hop (step) is replaced with a synthetically wrong value. Five distinct error types are used: off_by_one, wrong_operator, wrong_unit, magnitude_error, and wrong_percentage_base.
Models: Evaluation spans five models across two families (Qwen and Llama), including instruction-tuned and reasoning (thinking-mode) variants, ranging from 3B to 8B parameters.
Probe Training: A linear probe (logistic regression) is trained on the residual stream activations of the frozen models. The probe is trained to distinguish between clean and corrupted contexts using two aggregation modes: last-token and mean-over-tokens.
Key Metrics:
- Detection ( $AUROC_{detect}$ ): The ability of the probe to distinguish a corrupted context from a clean one.
- Failure Prediction ( $AUROC_{fail}$ ): The ability of the probe to predict whether the corruption will result in a wrong final answer, given that the error has already occurred.
- Surface Signals: Six scalar uncertainty signals (e.g., entropy, confidence, vocabulary gap) are computed from token logits to serve as baselines.
Interventions: Three probe-triggered strategies are tested to see if detecting the error can improve final accuracy:
1. Reprompt: Adding a recheck instruction.
2. Replace-prior: Regenerating the previous hop from scratch.
3. Branch-and-pick: Sampling multiple continuations and selecting the one with the lowest probe error score.

Key Contributions

Contrastive Dataset: A deterministic, human-free dataset of 1,400 traces with five error types, evaluated across instruction-tuned and reasoning model variants.
Detection vs. Failure Prediction Collapse: A characterization showing that while probes can detect corruption with near-perfect accuracy, they fail to predict whether that corruption leads to a final incorrect answer.
Intervention Analysis: A controlled comparison of three intervention policies, revealing that no single strategy is universally effective; efficacy is highly dependent on the specific model and error type.
Refuted Hypothesis: The authors pre-registered and subsequently refute the hypothesis that "persistence of probe activation across hops" predicts failure.
Robustness Controls: Evidence that the detection signal is not an artifact of data leakage or surface-level token signatures, demonstrated via grouped cross-validation and leave-one-error-type-out evaluations.

Results

1. The Knowing-Saying Dissociation

High Detection Accuracy: Linear probes detect injected errors with $AUROC_{detect} > 0.98$ across all models, peaking at 0.997 on Llama-3.1-8B. The signal emerges primarily from the transition between layer 0 and layer 1.
Failure Prediction Failure: Despite high detection accuracy, the same probes are uninformative about final answer correctness. $AUROC_{fail}$ collapses to near-chance levels (0.47–0.53) across all models. The model "knows" the context is corrupted but does not "say" (or act upon) that knowledge to prevent a wrong final output.
Surface Signals: Verbalized confidence and other surface uncertainty signals perform at or near chance ( $AUROC \approx 0.5$ ) for detection and show no correlation with final correctness. Confidence often collapses to binary extremes (0 or 1) with indistinguishable error rates between the two groups.

2. Impact of Reasoning Modes

Enabling "thinking mode" (extended chain-of-thought) on Qwen3-4B did not close the gap. While it increased token count, it degraded error-condition accuracy from 27.9% to 1.2% without changing the probe's ability to detect the internal error state. This suggests the internal encoding of corruption persists even when the output channel is disrupted.

3. Intervention Effectiveness

Branch-and-pick: This was the only policy that was net-positive across models and uniquely non-breaking on Llama-3.1-8B (4 rescued, 0 broken). However, it increases inference cost significantly.
Replace-prior: This strategy rescued wrong traces but broke correct traces at a comparable rate, resulting in no net gain.
Reprompt: Generally ineffective.
Heterogeneity: Intervention success is highly dependent on error type. For example, replace-prior was highly effective for wrong_unit errors but net-negative for wrong_percentage_base errors.

Significance and Claims

The paper claims that probe-based monitoring is a necessary complement to verbalized confidence, not a replacement. The central finding is that the assumption that a model's internal state is accessible through its verbal outputs is false in the context of cascading errors.

Decodability, not Causation: The authors emphasize that while the error signal is linearly decodable from the residual stream, this does not imply the generation policy consults that signal. Interventions that remove the corrupted value (replace-prior) reduce probe firing rates but do not proportionally improve accuracy, suggesting the probed direction is not the proximal cause of the wrong answer.
Deployment Implications: There is no "one-size-fits-all" intervention. The deployable solution is model-aware, error-type-aware routing. Since no single intervention dominates, systems must route to specific correction strategies based on the predicted model and error type.
Limitations: The authors note that their interventions are not strictly safe (they can break correct traces) and that branch-and-pick incurs significant computational costs. The paper frames the routing strategy as a direction for future work rather than a fully deployable solution, as it currently lacks a robust error-type classifier.

In conclusion, the paper establishes that for agentic systems, relying on the model to "tell" it is making a mistake is insufficient. Internal state monitoring via linear probes offers a high-fidelity detection mechanism, but bridging the gap between detection and successful correction requires nuanced, context-specific intervention strategies.