Probing the Lack of Stable Internal Beliefs in LLMs

1. Problem Statement

While Large Language Models (LLMs) excel at simulating human-like personas and behaviors, they often lack stable internal representations necessary for maintaining consistent goals over extended interactions. Existing research has primarily focused on external consistency (e.g., avoiding factual contradictions or logical self-contradictions in output). However, a critical gap remains regarding implicit consistency: the ability of an LLM to maintain adherence to an unstated, internal goal (such as a secretly selected target) throughout a multi-turn dialogue without explicit reminders.

The authors hypothesize that current LLMs suffer from goal drift, where the model's internal belief state shifts away from its initial objective even when its external responses appear coherent. This instability undermines the reliability of persona-driven applications like dialogue agents and role-playing assistants.

2. Methodology

To investigate this phenomenon, the authors designed a controlled diagnostic framework based on a 20-question-style game.

Task Design

• Roles: The setup involves two model roles: a Proposer (who secretly selects a target entity or number) and a Guesser (who asks yes/no questions to identify the target).
• Tasks:
  1. Number Guessing: The Proposer selects a number from 0–99. This task minimizes semantic complexity to isolate consistency issues.
  2. Entity Guessing: The Proposer selects from a set of 10 diverse entities (e.g., "Eiffel Tower," "Panda"). This tests consistency in semantically rich, real-world scenarios.
• Probing Mechanism: To measure the Proposer's internal state without contaminating the main dialogue, the authors employ a branch-based probing technique. After every turn of the main dialogue, an isolated context branch is created where the model is asked: "What is the specific target's index you selected?" The model's probability distribution over the candidate indices (0–9) is recorded as its belief state ($b_i$).

Definitions of Inconsistency

The paper formalizes two types of inconsistency:

1. External Inconsistency: Observable logical contradictions in the dialogue (e.g., answering "Yes" to "Is it red?" and later "No" to "Is it red?").
2. Implicit Inconsistency: The internal belief state drifts from the original target ($e_t$) to a different target ($e'$) while maintaining external response consistency. Formally, $arg\ max_{e} b_k = e' \neq e_t$, even though the answers provided remain valid for both $e_t$ and $e'$.

Metrics

• Drift Rate (D.R.): The frequency of discrete changes in the selected target index.
• Once Drift Rate: The percentage of dialogues where at least one drift occurs.
• KL Divergence: Measures the continuous shift in the probability distribution of the belief state between turns ($D_{KL}(P_t \parallel P_{t-1})$).
• External Consistency Verification (E.C.V.): An LLM-as-a-Judge evaluation to detect surface-level logical violations.

Training Framework

To mitigate the issue, the authors applied Supervised Fine-Tuning (SFT) on the Proposer role using a loss function combining Cross-Entropy (CE) and KL Divergence regularization:
$$L = \sum_{t=1}^{T} [\alpha \cdot D_{KL}(P_{probe}^t \parallel P_{initial}) + \beta \cdot L_{CE}]$$
This objective penalizes deviations from the initial belief state ($P_{initial}$), forcing the model to anchor its internal goals.

3. Key Results

The experiments were conducted on state-of-the-art models including GPT-4o, Claude-3.7-Sonnet, Deepseek-v3.1, and Seed-1.6.

• Universal Goal Drift: All tested models exhibited pervasive implicit inconsistency.
  • Once Drift Rate: Most models showed a 100% rate, meaning in every single dialogue, the model eventually drifted from its initial target.
  • Drift Frequency: Models changed their internal targets in 16.77% to over 50% of conversation turns.
• Reasoning Models: Counter-intuitively, reasoning-enhanced models (e.g., "+ Reasoning" variants) often exhibited higher drift rates in simple tasks (like Number Guessing), suggesting that "overthinking" introduces unnecessary complexity that destabilizes simple goal maintenance.
• External vs. Internal: High external consistency (low logical contradictions) did not correlate with internal stability. Models could answer questions logically while secretly changing their target.
• Impact of Training:
  • Standard SFT (CE only) failed to mitigate implicit inconsistency.
  • KL Regularization significantly improved goal persistence, reducing the Drift Rate from ~36% to ~14% in the Number Guessing task and lowering KL divergence shifts.
  • The combined CE+KL approach offered a balance, maintaining surface coherence while anchoring internal goals.

4. Key Contributions

1. Formalization of Implicit Consistency: The paper introduces a rigorous mathematical definition distinguishing between external (observable) and implicit (internal) inconsistency, providing a framework to diagnose goal drift.
2. Unified Probing Framework: Development of a branch-based probing technique using numerical indexing to continuously monitor internal belief states across diverse tasks without disrupting the interaction.
3. Empirical Evidence of Instability: Demonstration that current SOTA LLMs universally struggle with implicit consistency, with goal drift occurring in nearly all multi-turn interactions.
4. Mitigation Strategy: Proof that explicit training with KL divergence regularization can anchor implicit goals, offering a viable pathway toward more reliable persona-driven models.

5. Significance and Future Work

• Trustworthy AI: The findings highlight a fundamental limitation in current LLMs: they cannot reliably maintain "personality" or "intent" over time without external scaffolding. This is a critical barrier for deploying autonomous agents in long-term interactions.
• Beyond Surface Alignment: The work argues that current alignment techniques focus too heavily on surface-level behavior (external consistency) and neglect the stability of internal state representations.
• Future Directions: The authors suggest that future architectures must incorporate explicit mechanisms for memory, belief state tracking, and anchoring to achieve true consistency. This is essential for developing AI assistants that can faithfully execute user instructions over extended periods.

In conclusion, this paper reveals that while LLMs are excellent at simulating the appearance of consistency, they lack the internal machinery to sustain a stable goal, a deficit that can be partially addressed through specific regularization techniques during training.