Technical Summary: Right Knowledge, Wrong Answer: Test-Time Steering for Temporal Fact Conflicts in Open-Weight Language Models

1. Problem Definition: Parametric Temporal Conflict (PTC)

Large Language Models (LLMs) trained on corpora spanning multiple years often encode both outdated facts and their newer superseding counterparts within their parametric memory. A distinct failure mode arises when a model possesses the updated fact internally (recoverable via temporal cues) but defaults to the outdated answer under standard prompting. The authors formalize this as Parametric Temporal Conflict (PTC).

Unlike simple knowledge absence (where the model lacks the fact entirely) or generic temporal recall degradation, PTC is defined by two conditions:

1. Outdated Preference: $P_\theta(a_{old} | q) > P_\theta(a_{new} | q)$ under standard prompting.
2. Recoverability: $P_\theta(a_{new} | q, c_{temp}) \gg P_\theta(a_{old} | q, c_{temp})$ when a temporal cue (e.g., "As of 2024...") is provided.

Existing approaches, such as continual learning (weight updates) or retrieval-augmented generation, do not address this specific setting of selecting between coexisting fact versions at inference time without modifying weights or using external retrieval.

2. Methodology: Temporal Attractor Steering (TAS)

The paper introduces Temporal Attractor Steering (TAS), a three-stage, retrieval-free, and training-free inference-time framework designed to detect, localize, and resolve PTC by manipulating residual-stream activations.

Stage 1: Detection

A lightweight linear probe scores whether a query is likely to exhibit PTC. The probe is trained on verified PTC positives and matched non-conflict negatives. If the score exceeds a threshold $\tau$, the query proceeds to the steering stage; otherwise, it is processed normally to preserve behavior on non-conflict queries.

Stage 2: Conflict Localization

Using causal activation patching, the method identifies a conflict-critical layer ($\ell^*$). By patching hidden states from a "temporal-prompt" forward pass into a "standard-prompt" forward pass, the authors determine which layer, when intervened upon, maximizes the Answer-Flip Rate (AFR). Empirical results show that for all tested models, this critical layer resides in the upper half of the network (e.g., layers 23–31 in 28–32 layer models).

Stage 3: Steering

At the identified layer $\ell^*$, the hidden state is perturbed toward the newer-fact representation using a precomputed steering vector $\Delta$:
$$h'_{\ell^*}(q) = h_{\ell^*}(q) + \alpha \cdot c \cdot \Delta$$
where $\alpha$ controls intervention strength and $c$ is the detector score. The steering vector $\Delta$ is constructed by averaging the difference between hidden states under temporal cues and standard prompts ($h_{\ell^*}(q, c_{temp}) - h_{\ell^*}(q)$). The authors propose three variants for averaging granularity: global (V1), per-relation (V2), and per-domain (V3). V2 (per-relation) is found to be the most effective, preventing the dilution of strong signals from specific relations (like "Head of State") by noisier high-volume relations.

3. Key Contributions

1. Formalization and Filtering: The paper defines PTC and introduces a knowledge-recovery filter to distinguish true conflicts from knowledge absence. This ensures that evaluation focuses only on cases where the model can retrieve the new fact but fails to do so by default.
2. Temporal Attractor Steering (TAS): A novel framework that operates on residual-stream activations to selectively override outdated parametric knowledge without retraining or external retrieval. Theoretical analysis provides sufficient conditions (Local Separability, Bounded Steering) under which TAS successfully flips the preference inequality.
3. Verified Multi-Model Evaluation: The authors construct an 8,746-record verified benchmark spanning five Wikidata relations (e.g., CEO, Head of State) and evaluate four open-weight models (Qwen-2.5-1.5B/7B, Mistral-7B-v0.3, Llama-3.1-8B).

4. Experimental Results

The evaluation was conducted on the 8,746-record benchmark across three model families.

• PTC Prevalence: Filtered PTC rates scale monotonically with model capacity, ranging from 4.1% (Qwen-1.5B) to 10.3% (Llama-3.1-8B). The outdated-preference rate (OPR) is relatively family-invariant (~53–54%).
• Localization: Single-layer activation patching achieves Answer-Flip Rates (AFR) of 0.72–0.85 across all models, confirming that conflict resolution circuits are localized to specific upper layers.
• End-to-End Performance: TAS recovers 29–57% of verified PTC cases while maintaining 85–99% accuracy on non-conflict queries (Preservation Accuracy).
  • Best Performance: Qwen-2.5-7B achieved a recovery rate of 56.9% with 94.8% preservation.
  • Comparison to Baseline: TAS outperforms a matched ITI (Inference-Time Intervention) baseline on three of the four models. Specifically, TAS gains +0.142 to +0.152 in recovery on Qwen models compared to ITI, attributed to constructing the steering vector from verified conflict instances rather than general behavioral contrasts.
• Cross-Model Analysis: PTC instances are not perfectly aligned across models. Interestingly, the overlap between Mistral and Llama (different families) is larger than the overlap between the two Qwen models (same family), suggesting training data recency is a stronger predictor of PTC alignment than architectural lineage.

5. Significance and Claims

The paper claims to provide a mechanism-first, retrieval-free account of how outdated parametric knowledge can be selectively overridden at inference time.

• Diagnostic Utility: The knowledge-recovery filter offers a cheap diagnostic to determine if a model holds recoverable new facts before attempting intervention.
• Stability: The conflict-critical layer is family-specific but stable, allowing it to be reused once identified for a given model family.
• Intervention Efficacy: The results demonstrate that temporal conflicts are measurable, localized, and resolvable. The paper modestly claims that while verified-conflict steering (TAS) generally outperforms behavioral-contrast steering (ITI), the optimal construction of the steering vector may vary by model architecture (e.g., ITI performed better on Mistral-7B-v0.3).

The work establishes that PTC is a distinct failure mode separable from knowledge absence, and that targeted activation steering can effectively resolve these conflicts without the computational cost of retraining or the latency of external retrieval.