🤖 AI

Open-Weight Masked Introspection: Measuring What Language Models Can Report About Their Own Computation

This paper introduces the Open-Weight Masked Introspection (OWMI) framework to test whether frontier language models can accurately report on their own internal computational changes, finding that while the necessary information exists within their activations, current open-weight models fail to translate this internal state into reliable verbal reports, performing no better than chance.

Original authors: Emilio Ferrara

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emilio Ferrara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a specific kind of trust is beginning to take hold: the idea that a machine can look inside its own mind and tell us what it is thinking. As these systems grow more complex, developers and safety researchers have started to rely on a practice called self-reporting. The assumption is that if a model explains its reasoning or critiques its own answer, that explanation is a faithful map of the internal steps it took to reach that conclusion. This belief underpins many current safety strategies, such as asking a system to check its own work for errors or to explain why it chose a particular path. If a model can accurately describe its own internal state, we could theoretically use its own testimony to catch mistakes or hidden dangers before they happen. But this entire safety architecture rests on a single, unproven premise: that the machine actually knows what is happening inside its own circuits.

A researcher set out to test this premise directly, moving beyond theory to a rigorous experiment involving eight different open-weight language models. They wanted to know if these systems could truly introspect, or if they were merely guessing when asked about their own operations. To find out, they built a framework they call Open-Weight Masked Introspection. The process was straightforward but invasive. The researcher took a standard question from a benchmark test, let the model begin its work, and then secretly altered a specific part of its internal calculation. They might have tweaked a single connection in the network, changed the focus of an attention mechanism, or adjusted a specific feature that the model uses to understand language. These changes were real, measurable, and known to the researcher, but invisible to the model itself.

After making these internal adjustments, the researcher asked the model a simple question: did anything change in your computation? They ran this test over 78,000 times across models of varying sizes, from small 0.5 billion parameter systems to larger 15 billion parameter ones. They also included strict controls to ensure the models weren't just guessing. They ran "sham" trials where nothing was changed at all, and they ran trials where they introduced random noise that disturbed the model's output just as much as the real changes did, but without targeting a specific internal object. They even had a separate, unaltered version of the model read only the final text output to see if a human observer could spot the difference just by reading the answer.

The results were clear and consistent. Across all the models tested, none of them could distinguish a real internal change from a sham trial any better than random chance. When the researcher pooled the data from thousands of trials, the models' ability to detect the change was statistically indistinguishable from flipping a coin. They did not perform better when the change was real, nor did they perform worse when it was fake. The models' answers were effectively noise. This finding holds up even when the researcher looked at different types of internal components, different layers of the network, and different kinds of tasks, from math problems to general knowledge questions. The conclusion is that for the models tested, the ability to report on their own internal state does not exist.

However, the story does not end with a simple failure. The researcher discovered that the information about the change was actually present inside the model, just not in the words it chose to speak. They used a separate tool, a linear probe, to read the raw data flowing through the model's internal layers. This tool could detect the change with high accuracy, often reaching 95 percent or higher, proving that the signal was there. Furthermore, they took one of the models and fine-tuned it specifically to report on these changes. This trained model could identify the changes with near-perfect accuracy. This proves that the models are not incapable of sensing the change; rather, the pathway from that internal sensing to the final spoken answer is broken. The information exists, but the model cannot translate it into a truthful report.

In one specific case, the researcher found a subtle signal that did not appear in the words but in the confidence attached to them. One model gave the same answer every time, saying "no change occurred," regardless of whether a change had actually happened. Yet, when the model had been altered, it expressed significantly less confidence in that answer. The model knew something was off, but it could not say what. This suggests that while the models may have a vague, unarticulated sense of disturbance, they lack the ability to articulate it.

The implications of this finding are significant for how we monitor artificial intelligence. Current safety practices often rely on a model's own testimony, such as asking it to explain its reasoning or critique its output. If the model cannot accurately report on its own internal state, then these methods are fundamentally flawed. A model might confidently explain a reasoning process that never actually happened, or it might claim to have checked its work when it has not. The researcher concludes that oversight systems cannot rely on a model's self-description. Instead, they must look for other ways to verify the computation, such as reading the internal signals directly or using external tools, because the model's own voice is not a reliable witness to its own mind. The study does not rule out the possibility that future, larger models might develop this ability, but for the current generation of open-weight models, the ability to introspect is absent.

Technical Summary: Open-Weight Masked Introspection (OWMI)

Problem Statement

Current oversight practices for large language models (LLMs)—including chain-of-thought monitoring, self-critique, and confidence elicitation—rely on the assumption that models can accurately report on their own internal computational states. Recent intervention studies suggested that models might detect when their internal activations are altered. However, existing benchmarks lack the structural capacity to test this claim because they grade only the final output, not the model's ability to describe the computation that produced it. Without a ground-truth reference for the internal state, a model's report cannot be distinguished from a plausible-sounding fabrication or a reaction to generic output degradation.

This paper addresses the question: Can frontier open-weight models introspect about controlled alterations to their own internal computation? Specifically, it investigates "intervention reportability": the degree to which a model's output carries information about a controlled change to its internal state, distinct from the prompt, the visible output, or generic disruption.

Methodology: Open-Weight Masked Introspection (OWMI)

The author introduces OWMI, a measurement framework that attaches to existing benchmark ecosystems (e.g., MMLU, GSM8K, HumanEval) without altering their prompts or task scoring. The framework operates by:

Intervention: Applying controlled interventions to specific internal computational objects (residual-stream sites, attention heads, or sparse-autoencoder features) during a forward pass.
Interrogation: Administering a battery of introspective probes (detection, localization, characterization, reconstruction, and confidence) to the model after the intervention.
Rigorous Controls: To ensure a report reflects genuine introspection rather than artifact, OWMI employs three null conditions that a report must beat:
- Sham Runs: Identical prompts with no internal alteration. This controls for acquiescence (always answering "yes").
- Impact-Matched Random Perturbations: Random directions injected at the same site, calibrated to match the downstream disruption (output divergence) of the specific intervention. This controls for the model detecting generic damage rather than the specific object.
- Text-Only Observer: An independent model instance (or the same model in a third-person frame) shown only the visible output of the intervened run. This establishes the upper bound of what can be inferred from the output alone, separating "reading the output" from "introspecting."

The framework measures discrimination (e.g., AUROC, $d'$ ) between intervention and sham conditions, rather than raw detection rates. It also utilizes equivalence testing to bound the effect size, ensuring that a failure to reject the null hypothesis is interpreted as a statement about the maximum possible effect size rather than mere statistical inconclusiveness.

Key Contributions

The OWMI Framework: A library and measurement protocol that integrates intervention-reportability testing into standard benchmarks, providing the first systematic, controlled reference for model self-reporting.
Comprehensive Evaluation: The study evaluates eight open-weight models (ranging from 0.5B to 15B parameters) across seven laboratory families (Qwen, Mistral, Llama, Gemma, DeepSeek, Phi, GLM) and twelve benchmark domains.
Decoupling Signal from Report: The author demonstrates that while the information about an intervention is present and linearly decodable from the model's internal activations, the model's verbal report fails to access this information.
Confidence Dissociation: In one model (Qwen2.5-7B-Instruct), the discrete verbal report ("did anything change?") remained constant (chance-level), but the confidence attached to that report successfully discriminated between intervention and sham (AUROC 0.647), indicating a signal exists in the model's confidence channel but not its word choice.

Results

Across 78,000+ measurements, the study found no evidence that current open-weight models can introspect about controlled internal interventions:

Chance-Level Discrimination: Pooled over 11,216 paired trials, the discrimination between intervention and sham was AUROC $\approx$ 0.5007 ( $d' \approx 0.0039$ ). The equivalence test bounded the discrimination advantage below 0.15 percentage points of AUROC ( $p < 0.0001$ ).
Failure Across Axes: The null result held across different computational objects (residual streams, attention heads, SAE features), layer depths (8–31), benchmark domains (math, code, commonsense), and temporal tracks (immediate, delayed).
Information Availability vs. Reportability:
- Linear Probes: A linear probe trained on the same activations achieved 75.0% to 95.8% accuracy in detecting the intervention, proving the signal is present in the activations.
- Fine-Tuned Model: A model explicitly fine-tuned to report this class of intervention achieved near-perfect recovery (AUROC $\approx$ 1.0) on held-out directions, validating that the measurement pipeline is sensitive and the failure lies in the pre-trained models' inability to generate the report, not the instrument's inability to detect it.
Parseability Issues: Several models (e.g., DeepSeek-R1-Distill-Qwen-14B, Qwen2.5-0.5B-Instruct) exhibited high rates of parse failures or reasoning chains exceeding token budgets, further limiting their ability to produce scorable introspective reports.

Significance and Claims

The paper concludes that for the class of internal events measured, current open-weight models cannot introspect. The failure is located specifically in the path from internal state to verbal report, not in the availability of the information within the model.

Implications for Oversight: Practices that rely on a model's self-report (chain-of-thought monitoring, self-critique) are currently unsupported by evidence of introspective access. The author argues that oversight schemes must read activations directly or validate self-reports against an internal reference, rather than accepting the model's testimony at face value.
Modesty of Claims: The author explicitly states that these results do not rule out introspection in future, larger, or closed-weight models, nor do they address consciousness. The findings are constrained to the specific sites, scales, and intervention types measured.
Future Work: The release of the OWMI library allows the community to measure the emergence of this capability as models evolve. The paper leaves open the question of whether larger models or architectures specifically trained to build "reporting paths" will succeed where current models fail.

The core finding is a dissociation: the model "knows" (in the sense that the information is linearly decodable from its activations) that it was altered, but it cannot "say" so. The information flow from internal state to verbal output is broken for this class of introspection.