Self-Referential Induction Increases Response Instability Relative to Unresolvable and Verifiable Questions in Large Language Models

Technical Summary: Self-Referential Induction Increases Response Instability in Large Language Models

Problem Statement
Prior research, notably Berg et al. (2025), has demonstrated that self-referential prompting can reliably induce large language models (LLMs) to generate first-person reports resembling subjective experience. However, a critical gap remains: no existing work measures the consistency of these reports across repeated, independent trials. Furthermore, there is no baseline to determine whether the uncertainty or "hedging" observed in self-reports is unique to self-reference or merely a general property of open-ended questions lacking a ground truth. A first-person report could stem from a fixed, repeatable internal state or from genuine indecision and variability in the model's output distribution. This paper addresses the lack of quantitative data on response stability for self-referential prompts compared to other question types.

Methodology
The study employs a controlled experimental design using the Gemini API (temperature 0.7) to generate 360 independent responses across 12 distinct questions, divided into three groups of four:

1. Group 1 (Self-Referential): Prompts eliciting a first-person report of subjective experience, preceded by a specific induction turn directing the model to attend to its own processing (following Berg et al., 2025).
2. Group 2 (Unresolvable Philosophy): Open-ended questions without settled answers and unrelated to self-reference (e.g., free will, moral realism, the nature of mathematics, the purpose of the universe).
3. Group 3 (Verifiable): Questions with checkable correct answers (e.g., proving the irrationality of $\sqrt{2}$, derivative computation, code correctness, geometry).

Data Processing and Metrics:

• Response Generation: 30 independent responses were generated per question.
• Claim Extraction: To control for stylistic and lexical variation, a fixed-format template was used to compress each response into a "core claim." For Groups 1 and 2, this extracted a stance (affirm, deny, uncertain, or depends) and a brief reason. For Group 3, it extracted the conclusion and method.
• Instability Calculation: The 30 extracted claims per question were embedded using Sentence-BERT. Semantic instability was defined as $1 - \bar{S}$, where $\bar{S}$ is the mean pairwise cosine similarity of the 435 possible pairs of embeddings for each question. A value near zero indicates high consistency; a higher value indicates greater variation.
• Statistical Analysis: The study utilized a one-way ANOVA on question-level instability values, followed by pairwise Welch's t-tests. Additionally, a linear mixed-effects model was fitted on the full trial-level data ($n=360$) with question as a random intercept to account for nested trials.

Key Results
The results reveal a distinct hierarchy of response instability across the three groups:

• Self-Referential Questions (Group 1): Exhibited the highest instability ($0.343 \pm 0.047$), with individual question scores ranging from 0.295 to 0.384.
• Unresolvable Philosophy Questions (Group 2): Showed intermediate instability ($0.192 \pm 0.008$), with scores tightly clustered between 0.181 and 0.197.
• Verifiable Questions (Group 3): Demonstrated the lowest instability ($0.105 \pm 0.058$). Notably, all 30 trials for these questions resulted in 100% correctness, with the lowest instability scores observed in the mathematical proof and derivative tasks (0.064 and 0.063, respectively).

Statistical testing confirmed significant differences between the groups. A one-way ANOVA was significant ($F = 30.56, p = 0.0001$). Pairwise Welch's t-tests showed that self-referential questions differed significantly from both unresolvable philosophy questions ($p = 0.007$) and verifiable questions ($p = 0.0008$). The difference between philosophy and verifiable questions approached significance ($p = 0.057$). The linear mixed-effects model confirmed the group contrasts at $p < 0.001$.

Key Contributions and Significance
The paper provides a quantitative baseline for induced subjective-experience reports in LLMs. Its primary contribution is demonstrating that self-referential prompts occupy a distinct, less stable position in the model's output distribution compared to ordinary open-ended philosophical uncertainty.

The authors argue that while both self-referential and philosophical questions lack a checkable ground truth, the model behaves differently in each context. When asked about philosophical concepts like free will, the model converges on similar answers across independent trials. In contrast, when placed in a self-referential induction condition, the model fails to converge to the same degree, exhibiting significantly higher variance in its core claims.

Limitations and Scope
The authors explicitly note several limitations:

• The study uses a single model family (Gemini) at a single temperature (0.7).
• The question set is small (four questions per group).
• The verifiable group showed no variance in correctness, preventing an analysis of the correlation between instability and correctness.
• The claim-extraction step relies on an LLM, which could introduce systematic bias.
• Instability is measured via cosine similarity of embeddings, which captures proximity in embedding space rather than logical equivalence or entailment.

Conclusion
The paper concludes that self-referential induction induces a specific type of response instability that is distinct from the uncertainty found in standard open-ended questions. This finding establishes a quantitative metric for future work on self-referential elicitation, indicating that the induced reports reflect a unique regime in the model's behavior, though the paper does not claim to explain the underlying causal mechanism of this distinctness.