Before the First Token: Scale-Dependent Emergence of Hallucination Signals in Autoregressive Language Models

1. Problem Statement

Large Language Models (LLMs) suffer from hallucinations (generating plausible but factually incorrect information), which poses severe risks in high-stakes domains like healthcare, law, and finance. While recent research has established that internal model activations contain latent signals distinguishing factual from fictional outputs, critical gaps remain:

• Temporal Uncertainty: It is unknown when during the generation process these internal signals become most active.
• Scale Dependency: It is unclear if the timing and reliability of these signals depend on the model's parameter count.
• Intervention Feasibility: It is unknown if detecting these signals allows for causal intervention (correcting the hallucination) or if they are merely correlational.

The authors hypothesize that the ability to "know" one is hallucinating before generating a token (pre-commitment encoding) is a scale-dependent phenomenon that emerges only after a certain capacity threshold and is influenced by training methodology.

2. Methodology

The study employs a systematic linear probing approach across a diverse set of models and datasets.

• Models: Seven decoder-only transformers were evaluated, spanning a 60-fold parameter range (117M to 7B):
  • Small/Transition: GPT-2 Small (117M), Pythia-160M, Pythia-410M.
  • Large: Pythia-1.4B, GPT-2 XL (1.5B).
  • Frontier: Pythia-6.9B (Base-only, trained on The Pile) and Qwen2.5-7B (Instruction-tuned, diverse corpus).
• Datasets: Three factual tasks totaling 552 labeled examples:
  • TriviaQA: Difficult, balanced correct/incorrect split.
  • Simple Facts: Easy, near-ceiling performance for large models.
  • Biography: Moderate difficulty, entity-specific retrieval.
• Experimental Design:
  • Activation Extraction: Residual stream activations were extracted at generation positions 0 (pre-token), 1, 2, 3, and 4. Position 0 represents the state after processing the prompt but before any output token is generated.
  • Probing: A linear classifier (L2-regularized logistic regression) was trained on PCA-reduced activations to predict whether the model's subsequent output would be factual or hallucinated.
  • Metrics: Primary metric was AUC-ROC (Area Under the Curve). Statistical significance was assessed via paired t-tests comparing AUC at Position 0 vs. Position 4.
  • Intervention: The authors attempted activation steering (guiding the model away from the hallucination direction) to test if the signal was causal.

3. Key Contributions

1. Scale-Dependent Temporal Transition: The paper provides the first systematic evidence that hallucination detection follows qualitatively different temporal profiles based on model size.
2. The "Null-Signal" Finding: Models below ~400M parameters show no reliable internal signal for hallucinations at any generation stage (AUC near chance).
3. Pre-Commitment Encoding: Models above ~1B parameters (specifically instruction-tuned ones) exhibit a signal peak at Position 0, indicating the model "decides" on the truthfulness of its response before generating the first token.
4. Training Methodology Effect: At the 7B scale, instruction tuning is shown to be a prerequisite for pre-commitment encoding. A base-only model (Pythia-6.9B) showed no temporal structure, while an instruction-tuned model of the same size (Qwen2.5-7B) showed a strong pre-generation signal.
5. Correlational vs. Causal: The study demonstrates that while these signals are detectable, steering activations based on them fails to correct hallucinations, proving the signal is correlational, not causal.

4. Key Results

A. The Three Regimes of Scale

• Regime 1: Small Models (<400M Parameters)

  • Result: AUC values hover around chance (0.48–0.67) at all positions.
  • Finding: No reliable internal signal exists. The model does not "know" it is hallucinating.
  • Implication: Activation-based monitoring is ineffective for small models.

• Regime 2: Transition Models (400M – 1.5B Parameters)

  • Result: A gradual shift occurs. The peak detection signal moves from late generation (Position 4) to early generation.
  • Finding: Pythia-1.4B shows a statistically significant peak at Position 0 (AUC 0.663, p=0.012). GPT-2 XL shows a similar trend (p=0.054).
  • Implication: Pre-commitment encoding begins to emerge in this range.

• Regime 3: Large Models (>1.5B Parameters)

  • Result: Divergence based on training.
  • Instruction-Tuned (Qwen2.5-7B): Strong pre-commitment signal at Position 0 (AUC 0.630, p=0.038). The signal decays as generation proceeds.
  • Base-Only (Pythia-6.9B): Flat temporal profile (AUC 0.603 at Pos 0 vs 0.604 at Pos 4, p=0.989). Despite having 7B parameters and higher factual accuracy than smaller models, it lacks the structured pre-commitment signal.
  • Implication: Scale alone is insufficient; instruction tuning (or equivalent knowledge organization) is required to structure internal representations for pre-commitment.

B. Layer-Wise Analysis

• Optimal detection depth shifts with scale. Small models peak in early layers; large models peak in middle-to-deep layers.
• Qwen2.5-7B (Instruction-tuned) shows the deepest optimal layer (85% depth), suggesting complex late-network processing for factuality.

C. Intervention Failure

• Activation Steering: Attempting to steer the model away from the "hallucination direction" identified by the probe resulted in a 0% correction rate across all models.
• Conclusion: The probe detects a byproduct or correlation of the decision process, not the causal mechanism driving the hallucination. Therefore, probes can flag hallucinations but cannot fix them.

5. Significance and Implications

• Detection Strategy Calibration:

  • <400M: Do not use internal probes; rely on system-level interventions (e.g., RAG, human review).
  • >1B (Instruction-Tuned): Position-0 probing is optimal. It allows for pre-emptive flagging of hallucinations before any user-visible content is generated.
  • >7B (Base-Only): Do not assume pre-commitment exists. Output confidence metrics may be more reliable than internal probing until empirical validation is performed.

• Theoretical Insight:

  • The findings support a Modular Knowledge Circuit hypothesis: Large, instruction-tuned models develop specific circuits that assess knowledge availability before generation begins.
  • It challenges the notion that "more parameters = better internal reliability signals" by showing that base models at 7B lack the structured signals found in instruction-tuned models of the same size.

• AI Safety:

  • The "internal-external disconnect" (models knowing they are hallucinating internally but not expressing it in output probabilities) suggests that relying solely on output confidence is insufficient for safety-critical applications.
  • A "defense-in-depth" strategy combining Position-0 probing (for instruction-tuned models) and output uncertainty is recommended.

Conclusion

The paper establishes that the ability of LLMs to internally signal hallucinations is a scale-dependent emergent property that requires both sufficient capacity (~1B+ parameters) and specific training methodologies (instruction tuning) to manifest as a pre-generation signal. While these signals enable highly effective detection of hallucinations before they occur, they do not currently enable correction, highlighting a critical gap between detection and intervention in AI safety.