Technical Summary: The Geometry of Forgetting

Problem Statement

Large language models (LLMs) frequently produce outdated answers with the same fluency and confidence as correct ones. When asked about facts that have changed since the model's training cutoff (e.g., current government leaders or sports coaches), models may retrieve a stale fact, name a pre-training holder, or fabricate a new one. Existing detection methods, which rely on output-level statistics (e.g., token entropy, semantic entropy) or hidden-state probes trained for general correctness or uncertainty, fail to distinguish between these failure modes. Specifically, they cannot differentiate Stale Recall (retrieving a fact that was correct at training time but is now outdated) from Confabulation (fabricating a fact that never existed). This opacity leaves deployments unable to distinguish recoverable training-time errors from hallucinations.

The authors argue this is not an engineering failure but a structural one: temporal drift (whether a stored fact has changed since training) is encoded in the model's residual stream as a representational axis that is geometrically orthogonal to both correctness and uncertainty. Consequently, any method operating solely on correctness or uncertainty signals is structurally blind to drift by construction.

Methodology

Dataset Construction

The authors constructed a dataset of 3,511 queries spanning 1,342 unique facts across four relations (head_of_government, head_coach, chair_of, owned_by) with day-level tenure precision derived from a frozen Wikidata snapshot (April 2026).

• Temporal Knowledge States: Each (sample, model) pair is classified into one of five states based on the model's output relative to the fact's timeline and the model's training cutoff ($\tau_\theta$):
  1. Stable-Correct: Correct answer, no drift.
  2. Anachronism: Incorrect answer (past holder), no drift (fact never changed).
  3. Stale-Recall: Incorrect answer (past holder), drifted (fact changed post-training).
  4. Obsolete-Current: Incorrect answer (current holder at training, replaced later), drifted.
  5. Confabulation: Incorrect answer (fabricated), drifted.
• Controlled Evaluation: To prevent models from simply detecting the query year (a confound), evaluation is restricted to query years $\ge \tau_\theta + 1$.

Probing and Analysis

The study evaluates six instruction-tuned models (Llama-2, Mistral, Llama-3.1, Qwen-2.5, Gemma-2) across four architecture families.

1. Linear Probes: L1-regularized linear probes are trained on residual-stream activations ($h^{(\ell)}$) to predict the is_drifted label. The authors sweep all layers to identify the peak performance layer.
2. Geometric Independence Tests: To verify that the drift axis is independent of correctness and uncertainty, five complementary measures are employed:
   • Weight Cosines: Measuring the angle between drift, correctness, and uncertainty probe weight vectors.
   • Score Correlations: Pearson correlations between probe scores.
   • Bidirectional Null-Space Projection: Projecting the residual stream onto the subspace orthogonal to nuisance directions (correctness/uncertainty) and retraining the target probe.
   • Iterative Null-Space Projection (INLP): Iteratively removing directions associated with nuisance variables to test for distributed dependence.
   • Mean-Shift Dissociation: Comparing trained probes against untrained difference-of-means (DoM) directions to rule out optimization artifacts.
3. Mechanistic Interventions:
   • Activation Patching: Replacing corrupted hidden states with clean ones to localize temporal processing.
   • Direct Logit Attribution (DLA): Decomposing output logits to compare MLP trajectories for Stale-Recall vs. Confabulation.
   • Causal Steering: Amplifying or suppressing the drift direction to observe changes in holder logits.

Key Results

1. Linear Decodability of Drift

A linear probe trained directly on drift labels achieves AUROC 0.83–0.95 across all six models. In contrast, existing baselines (Token Entropy, Semantic Entropy, SEP, SAPLMA, CCS) remain near chance (0.49–0.57). This confirms that drift is a linearly decodable signal in the residual stream, but it is invisible to methods not explicitly trained on drift.

2. Geometric Orthogonality

Five tests confirm that the drift axis is geometrically independent of the correctness-uncertainty subspace:

• Weight Cosines: $|\cos| \le 0.14$ between drift and both correctness/uncertainty vectors.
• Score Correlations: $|r| \le 0.20$ between drift scores and correctness/uncertainty scores.
• Null-Space Projection: Removing correctness and uncertainty directions results in negligible AUROC degradation ($|\Delta| \le 0.008$), confirming drift does not reside in their span.
• INLP: Removing 10 directions for each nuisance variable still leaves drift detectable.
• Mean-Shift Dissociation: Trained probes are near-orthogonal even when untrained mean-shift directions are highly correlated, indicating the independence is structural, not an artifact of sparsity.

3. Model Specificity and Input Independence

In a cross-cutoff experiment using byte-identical inputs, the drift probe fires on the model whose training cutoff predates the fact's transition and remains silent on the later-cutoff model. The probability $P(A > B)$ ranges from 0.975 to 0.998 across model pairs. This proves the probe detects the model's internal knowledge state, not input properties like entity familiarity or query year.

4. Mechanistic Explanation

• Identical Retrieval Dynamics: Direct Logit Attribution reveals that the MLP retrieval circuit produces near-identical trajectories for Stale-Recall and Confabulation ($r > 0.81$). Both failure modes peak at the same layer, explaining why output confidence (which reflects retrieval strength) cannot separate them.
• Latent but Activatable: Causal steering shows the drift direction is latent under normal inference (suppressing it causes no output change) but, when amplified, it restructures holder logits, selectively engaging the temporal retrieval circuit.

Significance and Claims

The paper claims that temporal drift is a distinct representational property of LLMs, structurally orthogonal to correctness and uncertainty. This geometric independence explains why:

1. Existing hallucination detection methods fail on stale facts.
2. Models can be confidently wrong (high confidence, low uncertainty) on drifted facts because they are performing genuine retrieval of outdated training data.
3. Detecting temporal drift requires probes trained directly on drift labels; repurposing correctness or uncertainty probes is insufficient.

The authors conclude that deployed systems should read this specific drift axis to identify stale knowledge, rather than waiting for surface-level errors or relying on uncertainty estimates. The work provides a mechanistic account of how models encode temporal validity and demonstrates that this signal is accessible via linear probes, offering a path toward more robust temporal fact-checking in LLM applications.