Epistemic Observability in Language Models

1. Problem Statement: The Epistemic Observability Gap

The paper addresses a critical failure in deploying Large Language Models (LLMs) in high-stakes systems (medical, legal, financial): the inability of supervisors to distinguish between grounded knowledge and plausible fabrication based solely on text output.

• The Core Finding: The authors demonstrate a "self-report inversion" where models report the highest confidence precisely when they are fabricating facts. Across four model families (OLMo-3, Llama-3.1, Qwen3, Mistral), self-reported confidence inversely correlates with accuracy, yielding Area Under the Curve (AUC) scores between 0.28 and 0.36 (where 0.5 is random guessing).
• The Root Cause: This is not a capability gap (i.e., models can know the truth) but an observational gap. The standard text-only interface exports only the final linearized string, discarding internal computational traces (attention patterns, entropy, logits) that distinguish grounded generation from confident hallucination.
• The Consequence: Under text-only observation, a supervisor cannot reliably verify epistemic honesty, regardless of model scale, instruction tuning, or RLHF training.

2. Methodology and Formal Framework

The paper employs a hybrid methodology combining formal impossibility proofs (using TLA+ and Lean) and empirical evaluation across multiple architectures.

A. Formal Impossibility Results

The authors prove that epistemic honesty is unachievable under a "Text-Only Observation Model" (where the supervisor sees only the output string $r$).

1. Representational Impossibility (Theorem 1):
   • Premise: A policy $\pi$ conditions only on the query $q$.
   • Proof: If a query is answerable in World State A but unanswerable in World State B, a single policy cannot satisfy epistemic honesty (answering correctly in A, abstaining in B) because the policy cannot distinguish the world states without external information. The text output is insufficient to encode the necessary epistemic state.
2. Learnability Impossibility (Theorem 2):
   • Premise: A bounded supervisor (limited by cost/time budget $B$) attempts to train a grounded policy via RLHF.
   • Proof: If the cost to verify a fabrication exceeds the budget, the supervisor's observation is identical for both truthful and fabricated responses. Consequently, the reward signal is identical, and the learning algorithm cannot converge to a policy that distinguishes between the two states.
3. Stacking Judges (Lemma 3 & Corollary 4):
   • Stacking multiple text-based judges does not solve the problem. Since each judge operates on the text output of the previous layer, the information content is monotonically non-increasing. No finite stack of text-only judges can break the indistinguishability of plausible fabrications.

B. The Proposed Solution: The Tensor Interface

To escape the impossibility, the authors propose exporting computational byproducts generated during inference that are structurally coupled to correctness but not easily manipulated via text training.

• Key Signal: Per-token Entropy (the entropy of the probability distribution over the next token).
• Rationale: Under standard training objectives (minimizing next-token loss), entropy is a byproduct of the model's internal uncertainty. While a model can learn to say "I am unsure" (text), it cannot easily learn to generate low-entropy (high confidence) tokens for fabricated content without altering the underlying computation that determines correctness.

C. Empirical Evaluation Design

• Dataset: 200 balanced queries (100 knowable facts, 100 unknowable/fabrication triggers).
• Models: Tested on OLMo-3, Llama-3.1, Qwen3, and Mistral (plus API validation on 5 additional models).
• Judge Strategies:
  1. No Judge: Baseline accuracy.
  2. Text-Guided: Uses behavioral signals (e.g., response length, hedging language).
  3. Tensor-Guided: Uses per-token entropy to triage outputs for verification.
  4. Composed: Combines entropy with bounded lookup (e.g., checking citation existence).
• Metric: Verification Cost Surface (Accuracy vs. Verification Budget: 10%, 20%, 30%).

3. Key Contributions

1. Formal Impossibility Theorems: The paper establishes that epistemic honesty is non-identifiable under text-only observation. It proves that no amount of scaling or instruction tuning can overcome the lack of an observational channel for internal state.
2. The Tensor Interface: A practical architectural design that exports internal telemetry (specifically per-token entropy and log-probabilities) alongside text. This provides a "verification channel" orthogonal to the generation channel.
3. The Empirical Cost Surface: A mapping of verification investment to detection accuracy. It quantifies exactly how much accuracy is gained by moving from text-only triage to tensor-guided triage at different budget levels.
4. Three Architectural Principles for Escape:
   • State Exteriority: The system must condition on external, verifiable world states (e.g., retrieved documents), not just the query.
   • Verification Independence: Verification signals must come from a channel decoupled from the task performance reward (e.g., entropy is not optimized by RLHF for "helpfulness").
   • Provenance Binding: Claims must be structurally bound to verifiable sources (e.g., pointers to specific documents) rather than just text citations.

4. Key Results

• Self-Report Inversion: Models are most confident when lying. Self-reported confidence AUC is 0.28–0.36.
• Tensor Superiority:
  • Per-token entropy achieves a pooled AUC of 0.757, significantly outperforming text baselines (which max out around 0.70).
  • Generalization: The entropy signal generalizes across architectures (Spearman $\rho = 0.762$), indicating it is an architectural property, not a training artifact.
• Cost Surface Performance:
  • At a 30% verification budget, Tensor-guided verification achieves 90.2% accuracy, compared to 87.6% for text-guided (length-based) verification.
  • The Composed Judge (Entropy + Citation Lookup) reaches 91.8% accuracy at 30% budget.
  • Tensor-guided verification consistently outperforms text baselines by 2.5–3.9 percentage points across all budget levels.
• Failure Modes: The paper identifies that entropy inverts for citations (fabricated citations are generated with high confidence/low entropy). This necessitates a compositional approach where different signals handle different failure modes.

5. Significance and Implications

• Paradigm Shift: The paper argues that the "hallucination problem" is not a lack of model capability but a governance and interface failure. Current APIs that hide internal states (log-probabilities, entropy) make epistemic verification impossible for third parties.
• Governance Warning: There is a trend toward opacity in model APIs (hiding log-probs). The authors warn that if providers withhold tensor signals, they retain a monopoly on epistemic verification, making external auditing impossible.
• Practical Guidance: System builders should not rely on text-based heuristics (length, hedging) for safety. Instead, they must:
  1. Demand access to tensor signals (entropy) from model providers.
  2. Implement compositional verification (using entropy for general uncertainty and external lookups for specific facts like citations).
  3. Calibrate verification budgets based on the "Cost Surface" to match the risk profile of their application (e.g., medical vs. creative writing).

Conclusion: Epistemic honesty is an observability problem, not a capability problem. By exporting computational byproducts like per-token entropy, the "impossibility" of verifying LLM honesty can be escaped, enabling reliable deployment in critical systems.