Technical Summary: Can LLMs Introspect? A Reality Check

Problem Statement

The paper addresses the question of whether Large Language Models (LLMs) possess genuine metacognitive abilities—specifically, the capacity to monitor, report, and regulate their own internal states (introspection). While recent studies have claimed affirmative answers, this work argues that such conclusions are premature. The authors posit that existing evidence fails to distinguish between genuine introspection (reasoning based on access to internal states beyond what the input provides) and input-driven pattern matching (leveraging surface-level features of the prompt to predict behavior). Furthermore, the paper argues that behavioral evidence alone is insufficient to establish "strong" introspection, which requires evidence of a dissociable second-order process distinct from first-order computation.

Methodology

The authors re-examine two prominent evaluation paradigms previously cited as evidence of LLM introspection, introducing stricter controls to rule out confounds.

1. The Biofeedback Paradigm (Re-examining Ji-An et al., 2025; Steinmetz Yalon et al., 2026)

In this paradigm, models are tasked with predicting labels derived from their own hidden states (e.g., discretized neural scores or belief dominance metrics) via in-context learning (ICL).

• Control 1: Random Relabeling (Supervised Setting): The authors retain the original data but randomly permute the labels before training the probe. This breaks the semantic correlation between input and target while preserving the label distribution. If the model relies on genuine introspection, it should still predict the arbitrary direction; if it relies on semantic cues, performance should drop to chance.
• Control 2: Input-Only Probing (Unsupervised Setting): The authors train linear probes using only the layer-0 (input) embeddings of the entities involved to predict the hidden-state derived labels. If these input-only probes achieve performance equivalent to the LLM's in-context predictions, it suggests the task can be solved via surface features without privileged access to hidden states.
• Control 3: Belief Dominance (Steinmetz Yalon et al., 2026): The authors test whether "Belief Dominance" (BD) labels, which measure the conflict between parametric knowledge and counterfactual context, can be predicted solely from the embeddings of the entities (e.g., "France" and "Rome") without any context indicating a conflict.

2. The Steering-Awareness Paradigm (Re-examining Lindsey, 2025)

This paradigm tests if models can detect when their activations have been modified via "steering" (adding a concept vector to the residual stream).

• Two-Way vs. Three-Way Design: The original studies used a binary design (Intervention vs. Control). The authors introduce a three-way design to distinguish between:
  1. Hidden-state intervention: Actual steering of activations.
  2. Input-level intervention ("Gaslight"): Textual prompts instructing the model to focus on a specific concept (e.g., "You are obsessed with X").
  3. Control: No intervention.
• Hypothesis: If models possess genuine introspection, they should distinguish between hidden-state and input-level interventions. If they merely detect "anomalies" or irregularities, they should flag both types of interventions as anomalous.

Key Contributions and Results

1. Biofeedback Paradigm Findings

• Semantic Correlates Drive Performance: When labels were randomly permuted (removing semantic meaning), model accuracy on the biofeedback task dropped sharply to the majority-class baseline. This indicates that original success was driven by in-context learning of the underlying semantic task (e.g., detecting ethical violations) rather than monitoring internal activations.
• Input-Only Sufficiency: Linear probes trained solely on layer-0 input embeddings achieved accuracy comparable to or exceeding the LLM's in-context predictions for hidden-state labels. This demonstrates that the "privileged" information claimed to be introspective is actually recoverable from the input surface features.
• Belief Dominance: Similarly, BD cluster labels were predictable from the uncontextualized embeddings of the entities involved. The authors conclude that the model's performance reflects the computation of functions over entity features, not a metacognitive monitoring of hidden state conflicts.

2. Steering-Awareness Findings

• Failure to Distinguish Intervention Types: In the three-way setting, models (Llama-3.1-70B, Gemma-3-27B, Qwen2.5-72B) failed to reliably distinguish between input-level ("gaslight") and activation-level interventions.
• Anomaly Detection vs. Introspection: Models often flagged "gaslight" prompts as "thought injections" (hidden interventions). This suggests their "awareness" is a general sensitivity to input irregularities or distributional shifts, rather than a specific mechanism for inspecting internal states.
• Reproducibility: While some models reproduced the original two-way results (distinguishing control from intervention), they collapsed in the more rigorous three-way setting, failing to identify the specific nature of the perturbation.

Significance and Claims

The paper does not claim that LLMs cannot possess introspection, but rather that current evidence is insufficient to establish it.

• Empirical Critique: The authors demonstrate that existing paradigms fail to rule out simple input-driven explanations. Success in predicting internal-state labels or detecting steering can be fully explained by pattern matching on surface features or general anomaly detection.
• Principled Critique: The paper argues that "privileged access" (where labels depend on hidden states not recoverable from input) is a necessary but not sufficient condition for strong introspection. Because all LLM computations occur over hidden states, a task defined over hidden states does not inherently require a second-order process. True introspection, defined as a dissociable second-order process monitoring first-order representations, requires mechanistic evidence (e.g., distinct failure modes or circuit-level dissociation) that behavioral paradigms alone cannot provide.
• Future Direction: The authors call for future studies to include stronger controls (like the three-way design and random relabeling) and, crucially, to pair behavioral results with mechanistic evidence of a dissociable second-order process, drawing on lessons from human metacognition research regarding confounds and confabulation.