🤖 machine learning

Architecture Determines Observability in Transformers

This paper demonstrates that a transformer's ability to internally preserve signals about its own decision quality (observability) is not a generic property but an emergent characteristic determined by specific architectural choices and training recipes, often collapsing in models that otherwise achieve low loss.

Original authors: Thomas Carmichael

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Thomas Carmichael

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Confident Liar"

Imagine you are taking a test with an AI student. Sometimes, this student gets an answer wrong, but they are 100% sure they are right. They say, "I am confident this is the capital of France!" while writing "Berlin."

Current safety tools (called "confidence monitors") look at how sure the AI sounds. If the AI sounds confident, the tool assumes it's correct. But as this paper shows, these tools miss a whole class of mistakes where the AI is confidently wrong.

The Solution: Listening to the "Whispers"

The researchers found that even when the AI says it's confident, its internal brain (the "hidden layers") might be whispering a different story. It's like a person saying "I'm fine!" while their hands are shaking.

If you can listen to those internal whispers (the "activations"), you can catch the mistakes the confidence monitor misses. The paper calls this Observability: the ability to read the AI's internal "truth" from its middle layers.

The Twist: It's Not About Intelligence, It's About Blueprints

The most surprising finding is that not all AI models have these "whispers."

Think of AI models like houses.

The "Healthy" House: Some blueprints (architectures) are built with a special, quiet hallway in the middle of the house. Even if the front door (the output) says "Everything is great," you can walk down that hallway and hear the creaking floorboards (the error signal) that tell you something is wrong.
The "Collapsed" House: Other blueprints are built differently. In these houses, the hallway is sealed off or filled with concrete. Even if the house is falling apart, the middle layers are silent. You cannot hear the error signal, no matter how hard you listen.

The paper proves that whether a house has this "whispering hallway" depends entirely on the blueprint (the architecture) and the construction crew (the training recipe), not on how big or smart the house is.

The Evidence: The "Pythia" Experiment

The researchers tested this using a controlled set of models called Pythia. They built several houses using the exact same materials and construction rules, but changed the blueprints slightly.

The Result: They found a specific blueprint (24 layers, 16 heads) that always resulted in a "Collapsed" house. No matter how much they changed the size of the house (from small to huge) or the type of bricks used (different data sets), that specific blueprint always sealed off the hallway.
The Contrast: Other blueprints (like 16 layers or 32 heads) kept the hallway open and "healthy," allowing the error signal to be heard.

This means you can have a tiny model that can be monitored for errors, and a massive model that cannot, simply because of the blueprint.

The "Training" Mystery: When Did the Hallway Get Sealed?

The researchers watched the construction process in real-time (looking at checkpoints during training).

Early on: Both the "Healthy" and "Collapsed" blueprints started with the hallway open. The signal was there.
During Construction: As the training continued, the "Collapsed" blueprint actively erased the signal. The construction crew (the training process) filled in the hallway with concrete.
The Outcome: By the time the house was finished, the signal was gone. The model was still learning to speak perfectly (getting better at its job), but it had lost the ability to "know" when it was making a mistake.

Does This Work in the Real World?

The researchers took a "listener" (a probe) trained on general text (Wikipedia) and tested it on specific tasks like medical questions and reading comprehension.

The Catch: In models with the "Healthy" blueprint, this listener caught about 10–13% of the mistakes that the confidence monitor missed. These were errors where the AI was confidently wrong.
The Limit: In models with the "Collapsed" blueprint, the listener heard nothing. It was like trying to hear a whisper in a soundproof room.

The Bottom Line

Choosing an AI model is a safety decision.

If you want to be able to monitor an AI for confident errors, you can't just pick the biggest or smartest model. You have to pick the model with the right blueprint.

If the blueprint is "collapsed," no amount of better monitoring software will help. The signal isn't there to be found.
If the blueprint is "healthy," you can build monitors that catch mistakes the AI tries to hide.

In short: You can't fix a broken monitor by buying a better microphone if the room itself is soundproof. You have to build the room differently in the first place.

1. Problem Statement

Autoregressive Large Language Models (LLMs) frequently make confident errors—instances where the model outputs a token with high softmax probability (high confidence) that is nevertheless incorrect. Standard monitoring techniques, which rely on output confidence (e.g., max softmax probability) or trained predictors on the final layer, fail to flag these errors.

The paper posits that for activation-based monitoring to succeed, the model must preserve an internal signal regarding "per-token decision quality" that is distinct from and not fully recoverable by the output confidence signal. However, it is unclear whether this signal is a universal property of transformers or if it depends on specific architectural choices and training dynamics. The core problem is that standard probing methods often conflate "decision quality" with "output confidence," leading to inflated estimates of monitorability.

2. Methodology

The authors propose a rigorous framework to isolate and measure Observability, defined as the linear readability of decision quality from frozen mid-layer activations after controlling for output confidence.

Key Definitions and Metrics

Observability ( $\rho_{partial}$ ): The partial Spearman correlation between a linear probe on hidden states ( $h^{(l)}$ $h^{(l)}$ ) and the per-token cross-entropy loss ( $\ell$ $ℓ$ ), controlling for two confounders:
1. Max softmax probability ( $\hat{p}_{max}$ ).
2. Activation norm ( $\|h\|$ ).
  This isolates the "confidence-residualized" signal.
Output-Controlled Residual ( $r_{OC}$ ): A stricter metric that further controls for a trained Multi-Layer Perceptron (MLP) predictor on the final layer representation. If $r_{OC} > 0$ , the mid-layer signal contains information about decision quality that the final output layer discards or fails to utilize.
Observability Collapse: A state where $\rho_{partial}$ drops to a detection threshold (approx. $\le 0.15$ ) and $r_{OC}$ approaches zero, indicating the architecture has failed to preserve the output-independent error signal.

Experimental Protocol

Data: WikiText-103 for training probes; downstream tasks (SQuAD, MedQA, TruthfulQA) for operational validation.
Probing: A simple linear head ( $w^T h + b$ ) trained on frozen activations to predict a binary target: whether the residual loss ( $\ell - \hat{\ell}$ ) is positive (i.e., the model is more wrong than its confidence suggests).
Controls: The study employs extensive controls including nonlinear MLPs, logit entropy controls, and shuffle tests to rule out artifacts.
Datasets: 25 models across 6 families (Pythia, GPT-2, QWEN 2.5, LLAMA, Mistral, Gemma, Phi-3).

3. Key Contributions

A. The "Confidence Contamination" of Probes

The paper demonstrates that standard linear probes are heavily contaminated by output confidence. Across 13 models, controlling for max softmax and activation norm absorbs 57.7% of the raw probe signal. This implies that previous studies claiming to find "reasoning signals" in hidden states may have merely been detecting the model's own confidence levels.

B. Architecture Configuration Determines Observability

The most significant finding is that observability is not a generic property of transformers; it is determined by specific architectural configurations and training recipes.

The Collapse Phenomenon: In the Pythia suite (controlled training), models with the (24 layers, 16 heads) configuration class consistently collapse to $\rho_{partial} \approx 0.10$ , regardless of parameter count (410M to 1.4B) or dataset variant (Pile vs. Deduplicated Pile).
Healthy Band: Other configurations (e.g., 16L/8H, 32L/32H) maintain a "healthy" signal range of $\rho_{partial} \in [0.21, 0.38]$ .
LLAMA vs. QWEN: At matched scales (3B parameters), QWEN 2.5 preserves strong observability ( $\rho_{partial} \approx 0.26$ ), while LLAMA 3.2 collapses ( $\rho_{partial} \approx 0.09$ ). The distributions of probe seeds do not overlap, indicating a fundamental architectural difference.

C. Training-Emergent Collapse

Through checkpoint analysis of Pythia 1B (healthy) vs. Pythia 1.4B (collapsed), the authors show that both configurations initially form the signal early in training. However, during mid-training, the (24L, 16H) architecture erases this signal while predictive loss (perplexity) continues to improve. The healthy configuration recovers and retains the signal. This proves that collapse is a result of training dynamics interacting with architecture, not a static inability of the architecture to represent the signal.

D. Operational Utility (Exclusive Catch)

The study validates the practical utility of these probes on downstream tasks (QA, Medical, Truthfulness).

Exclusive Catch Rate: At a 20% flag rate, the observer catches 10.9%–13.4% of errors that confidence monitoring misses.
Transferability: Probes trained on WikiText transfer to downstream tasks without task-specific retraining, catching "confidently wrong" answers in Medical QA and Reading Comprehension.
Limitation: The observer does not detect "fluent hallucinations" (TruthfulQA) where the model confidently asserts a falsehood; it specifically catches token-level prediction failures where the internal state contradicts the output confidence.

4. Key Results

Metric	Healthy Configurations (e.g., GPT-2, QWEN, Mistral)	Collapsed Configurations (e.g., LLAMA 3B/8B, Pythia 24L/16H)
$\rho_{partial}$	0.21 – 0.38	$\approx$ 0.10 (near detection floor)
$r_{OC}$	Positive (0.09 – 0.17)	Near Zero or Negative
Signal Source	Output-independent, linearly readable	Erased by training; not recoverable by nonlinear probes
Exclusive Catch	~12–15% of errors (at 20% flag rate)	Near random baseline

Nonlinearity: Nonlinear probes (MLPs) do not recover the signal in collapsed configurations, proving the loss is representational, not a limitation of linear readouts.
Scale Independence: Within a healthy family (e.g., QWEN 2.5), observability is stable across a 28x parameter range (0.5B to 14B).
Recipe Dependence: The specific configuration that collapses depends on the training recipe (e.g., Pythia collapses at 24L/16H; LLAMA collapses at 28L/24H).

5. Significance and Implications

Architecture Selection as a Monitoring Decision: The paper argues that choosing a model architecture is not just a performance decision but a monitorability decision. If a specific configuration (like LLAMA 3B or Pythia 24L/16H) is chosen, activation-based monitoring for confident errors becomes impossible, regardless of how sophisticated the monitoring tool is.
Redefining Probing Validity: The study establishes that future probing research targeting decision quality must control for output confidence. Without these controls, results are likely overestimating the "output-independent" signal by a factor of 2–3.
Training Dynamics: The discovery that training actively erases a signal that was initially present suggests that preserving observability is a non-trivial optimization problem. Current training objectives (minimizing loss) may inadvertently destroy the internal signals needed for safety monitoring.
Complementary Monitoring: The observer provides a complementary signal to output confidence. While confidence monitors catch "uncertain" errors, activation monitors catch "confident but wrong" errors. However, this capability is bounded by the architecture's ability to preserve the signal.

Conclusion:
The paper concludes that architecture determines observability. A model's ability to be monitored for confident errors is an upstream property determined by the interaction of its configuration and training recipe. Once a model is trained in a "collapsed" configuration, no post-hoc probe design can recover the lost signal. Therefore, ensuring monitorability requires selecting architectures and training recipes that preserve these internal, confidence-independent signals.