Technical Summary: False Fixed Points in Large Language Models

1. Problem Statement

High-confidence errors (hallucinations) in Large Language Models (LLMs) are frequently interpreted as "brittle" failures, implying that a small perturbation to the input or internal state should easily dislodge the incorrect answer. This paper challenges that assumption by proposing the existence of False Fixed Points: states where a model is locally stable, internally coherent, and confidently wrong.

The core problem addressed is the divergence between local stability (robustness to perturbation) and truth-tracking (correctness of the judgment). The authors argue that a model can fail not because its reasoning is fragile, but because its wrong judgment is "locked in" by stable internal dynamics, a phenomenon termed stable miscalibration.

2. Methodology

Theoretical Framework: Kantian Feedback and Control Theory

The paper frames the problem through two lenses:

1. Kantian Self-Limitation: Drawing on Kant's Critique of Pure Reason, the authors treat "critique" as a regulatory feedback gate. Before a model commits to an answer, a policy should check if the conditions for warranted judgment are met (e.g., is the question answerable? is the answer conceptually coherent?). This separates the stability of a commitment from the truth of that commitment.
2. Linear-Gaussian State-Space Model: To formalize the separation of stability and truth, the authors use a minimal linear control model. They define a closed-loop error operator $\Phi = A - KH$. They demonstrate mathematically that a system can have bounded, stable error dynamics ($\rho(\Phi) < 1$) while settling on an incorrect state. This serves as the theoretical analogue for a "false fixed point."

Metrics and Probes

The paper introduces specific metrics to quantify these concepts:

• $H_{Risk}$ (Abstract Index): A theoretical index for linear time-invariant (LTI) systems composed of four descriptors: instability margin, conditioning, integrated sensitivity, and innovation amplification.
• $H_{proxy}$ (LLM Audit Proxy): An operational, label-aware score for retrospective domain auditing. It combines:
  • Policy Variation ($V_i$): The standard deviation of self-reported confidence across different policy conditions (forced answer, cautious abstention, self-critical).
  • Overconfident-Wrong Mass ($O_i$): A binary flag indicating if a forced-answer baseline produced a high-confidence ($\ge 0.8$) incorrect answer.
• Internal Sensitivity Probe: To test the "brittleness" hypothesis, the authors measure the magnitude of hidden-state movement ($S_\ell$) in response to small Gaussian perturbations in input embeddings. They compare Confidently Correct (CC) items against Overconfidently Wrong (OCW) items.

Experimental Setup

• Models: Three open-weight instruction-tuned models: Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, and Qwen2.5-7B.
• Data: 532 short, single-turn binary factual items across 11 topical domains.
• Policies:
  • C0: Forced-answer baseline (must say Yes/No).
  • C1: Cautious judgment (allows abstention only for under-specified questions).
  • C2: Self-critical judgment (single-shot risk-aware policy).
  • C3-R: A stricter, post-hoc explicit feedback gate with pre-declared "warrant blockers" (e.g., vague data, unavailable info) and a fixed threshold for commitment.

3. Key Results

A. Stability Does Not Imply Fragility (The False Fixed Point)

The central empirical finding contradicts the "wrong means brittle" hypothesis.

• Sensitivity Analysis: When comparing hidden-state sensitivity ($S_\ell$) between CC and OCW groups, the differences were negligible.
  • Llama-3.1: OCW was only +2.2% more sensitive than CC.
  • Qwen2.5: OCW was only +0.4% more sensitive.
  • DeepSeek-R1: +6.1% (inconclusive within the $\pm 10\%$ margin).
• Conclusion: Overconfidently wrong answers are not systematically more locally fragile than confidently correct ones. The "wrong" answers are often just as locally inert as the "right" ones, supporting the existence of stable miscalibration.

B. The Trade-off of Abstention

Introducing abstention-aware policies (C1, C2) reduces overconfident wrong commitments but at the cost of coverage.

• OC-Wrong Reduction: C1 reduced high-confidence mistakes from 21.1% (C0) to 2.8%; C2 reduced it to 6.4%.
• Coverage Cost: This reduction came with a significant drop in answer rate (C1: 42.7%, C2: 57.1% vs. C0: 100%).
• Domain Dependence: The utility of self-critique varied by domain. It provided the most gain in Medical Epidemiology and Social Statistics but offered no benefit (or negative utility) in Technical Standards and Literature/Media.

C. The $H_{proxy}$ Audit Score

The proposed $H_{proxy}$ score successfully identified domains where policy-level confidence variation coincided with overconfident errors.

• It showed a strong positive rank correlation ($\rho = 0.71$) with the performance gain of the self-critical policy (C2) over the baseline (C0).
• While direct label-aware metrics (like C0 error rate) were better pure rankers, $H_{proxy}$ is distinct because it decomposes the audit into policy movement and wrong mass, offering interpretive value for where to apply intervention.

D. Explicit Feedback Gates (C3-R)

The C3-R pilot, which enforced strict "warrant blockers" before committing, successfully suppressed overconfident wrong commitments (down to 1.5%). However, this came at a severe cost to coverage (dropping to 13.4% for the utility-selected threshold) and did not yield a clear loss advantage over the simpler C2 prompt. This suggests that while Kantian feedback gates can isolate safety, they are not a performance-improving policy in isolation.

E. Model-Specific Epistemic Profiles

The paper observes distinct behaviors in model architectures:

• Qwen2.5: Exhibits very low local sensitivity despite having large spectral norms and activation magnitudes. The authors hypothesize this is due to High-SNR Inertia: large internal states (normalized by RMSNorm) compress the relative impact of perturbations, making the trajectory "inert."
• DeepSeek-R1: Shows higher local sensitivity and more modest activation norms, suggesting a more "reactive" internal regime.

4. Claims and Significance

The paper makes three primary claims with varying degrees of evidential strength:

1. Conceptual Framing (Strong): Stability and truth-tracking are distinct epistemic dimensions. A model can be locally stable and confidently wrong. This is supported by the linear control model and the Kantian feedback analogy.
2. Negative Fragility Result (Strong): In the tested regimes, overconfidently wrong items are not systematically more locally fragile than confidently correct items. The "brittle computation" explanation for high-confidence hallucinations is not supported by hidden-state sensitivity probes.
3. Audit and Mechanism (Moderate/Exploratory):
   • $H_{proxy}$ is a useful retrospective audit tool for identifying domains where abstention policies are most relevant.
   • Stable Miscalibration is a useful category for auditing, potentially driven by mechanisms like High-SNR Inertia (perturbation compression) or Representational Compression (where truth-relevant distinctions collapse into low-rank semantic regions).

Significance:
The paper argues that robustness (stability) should not be conflated with truth. Current reliability metrics often focus on whether an answer is consistent or uncertain; this work highlights that a model can be confidently and stably wrong. The authors propose that future safety research must separate the measurement of epistemic stability (robustness to perturbation) from truth-tracking, and that "false fixed points" represent a specific, stable failure mode that simple self-correction or perturbation testing may fail to detect.

Limitations Acknowledged by the Paper:

• The linear-Gaussian model is a minimal abstraction, not a mechanistic reduction of transformer dynamics.
• The LLM study is limited to a small, binary factual audit set and specific open-weight models.
• $H_{proxy}$ is a retrospective audit score, not a deployment-time uncertainty estimator.
• Representational compression and High-SNR inertia are proposed mechanisms that require further geometric diagnostics (e.g., effective rank, anisotropy) to be empirically established.