💬 NLP

MoRFI: Monotonic Sparse Autoencoder Feature Identification

This paper introduces MoRFI, a method using sparse autoencoders to identify latent directions in large language models that monotonically correlate with hallucinations during fine-tuning on new knowledge, demonstrating that these specific features can be intervened upon to recover the model's ability to retrieve stored knowledge.

Original authors: Dimitris Dimakopoulos, Shay B. Cohen, Ioannis Konstas

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dimitris Dimakopoulos, Shay B. Cohen, Ioannis Konstas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a brilliant student who has spent years reading a massive library of books (pre-training). This student has memorized a huge amount of facts. However, when we ask them to learn a new, specific set of facts later (fine-tuning), something strange happens: if the new facts are completely unknown to the student, they start to "hallucinate." They begin to confidently make up answers, even for the old facts they used to know perfectly.

This paper, MoRFI, investigates why this happens inside the student's brain and how to fix it without re-teaching them everything.

The Problem: The "New Knowledge" Glitch

Think of the student's brain as a giant, complex control room filled with thousands of light switches (these are called latent features). When the student learns new things, they flip some switches. The researchers found that when they force the student to learn a lot of new facts they didn't know before, a specific set of switches gets stuck in the "ON" position.

These stuck switches act like a loud noise or a distraction. They drown out the quiet, steady signals the student needs to recall their old, trusted knowledge. The more new, unknown facts they try to cram in, and the longer they study, the louder this noise gets, and the worse they get at answering questions about what they already knew.

The Solution: MoRFI (The Detective)

The researchers built a tool called MoRFI (Monotonic Relationship Feature Identification). Imagine MoRFI as a super-smart detective with a special pair of glasses.

The Investigation: The detective watches the student's control room while they learn. They look for switches that behave in a very specific way: as the student is fed more and more "unknown" facts, these specific switches get brighter and brighter (or dimmer and dimmer) in a steady, predictable line.
The Filter: Most switches just flicker randomly. MoRFI ignores those. It only cares about the switches that change monotonically—meaning they change in one consistent direction as the new knowledge increases.
The Discovery: By tracking these specific switches across different models (like Llama, Gemma, and Mistral), the detective found a small, sparse group of them that are directly responsible for the "hallucination noise."

The Fix: Flipping the Switches Back

Once the detective identifies these "troublemaker" switches, the researchers tried a simple experiment: Steering.

Instead of retraining the student, they physically reached into the control room and flipped those specific switches back to their original state (or flipped them in the opposite direction).

The Result: It worked like magic. By just adjusting these few switches, the student suddenly remembered their old facts again.
The Analogy: It's like a radio that has picked up too much static from a new station. Instead of rebuilding the radio, you just turn the dial slightly to cancel out the static frequency. The music (the old knowledge) becomes clear again.

Key Findings in Plain English

It's Not Erasure, It's Blocking: The study found that the student didn't actually forget the old facts. The facts were still there, but the "noise" from the new learning was blocking the path to retrieve them. Flipping the switches cleared the path.
Turning Off is Better than Turning On: The researchers found that for some switches, it was more effective to turn them off (suppress them) than to turn them up. This suggests that the new learning was over-activating certain parts of the brain, and calming them down was the key.
It Works Everywhere: They tested this on three different types of "students" (different AI models), and the same small group of switches caused the problem in all of them. This suggests a universal mechanism for how these models get confused.
The "Composite" vs. "Single" Switch: If you try to fix the problem by adjusting all the changes at once (a "composite" direction), it helps a little. But if you find the single most important switch and fix just that one, the improvement is massive. This means the problem is very specific and localized, not a general fog over the whole brain.

Summary

The paper argues that when AI models learn new, unknown facts, they don't lose their old knowledge; they just get "distracted" by a few specific internal signals. The authors created a method to find these distracting signals and showed that simply turning them off (or adjusting them) allows the AI to instantly recover its ability to answer questions correctly, effectively curing the hallucination caused by the new training data.

1. Problem Statement

Large Language Models (LLMs) acquire factual knowledge primarily during pre-training. However, subsequent post-training stages, such as Supervised Fine-Tuning (SFT) on new data, often introduce hallucinations. The prevailing hypothesis is that exposing models to facts unknown to their pre-training distribution disrupts their ability to retrieve stored parametric knowledge.

While previous studies have established that fine-tuning on unknown data increases hallucination rates, the internal mechanistic causes remain poorly understood. Specifically, it is unclear:

Which specific latent features in the residual stream are causally linked to the degradation of knowledge retrieval.
Whether these features can be identified and manipulated to recover the model's original knowledge without retraining.
How these mechanisms differ across model architectures and training durations.

2. Methodology: MoRFI

The authors propose MoRFI (Monotonic Relationship Feature Identification), a two-stage pipeline designed to isolate causal latent features using Sparse Autoencoders (SAEs).

A. Controlled Fine-Tuning Experiment

The authors conduct a controlled experiment across three models: Llama 3.1 8B, Gemma 2 9B, and Mistral 7B v03.

Dataset: EntityQuestions (derived from Wikidata).
Variables:
1. Unknown Proportion ( $\uparrow$ Unknown): Training datasets are constructed with varying percentages (0% to 100%) of "Unknown" facts (facts the base model does not know) mixed with "Known" facts.
2. Training Epochs ( $\uparrow$ Epochs): Models are trained for varying durations (Early Stop, 10, 20, 30, 40, 50 epochs).
Observation: As the proportion of unknown facts and training epochs increase, test accuracy on closed-book QA drops significantly, confirming the hallucination phenomenon.

B. The MoRFI Algorithm

MoRFI analyzes SAE activations across a 4D tensor (Samples $\times$ Properties $\times$ Latents $\times$ Time/Epochs) to find features with robust monotonic trends.

Resampling and Aggregation: Uses bootstrapping to generate thousands of simulated test environments, calculating mean activations to create a noise-reduced baseline.
Trend Validation: Applies Spearman rank correlation and Mann-Kendall tests to verify if a feature's activation changes monotonically (consistently increasing or decreasing) as the proportion of unknown data or training epochs increases. This filters out spurious fluctuations.
Directional Sorting: Classifies features into Increasing (activations grow with unknown data) and Decreasing (activations shrink) sets.
Final Ranking: Ranks features based on their frequency of appearing in the top- $K$ altered features across all bootstrap replicates.

C. Activation Steering

Once candidate latents are identified, the authors perform activation steering (intervening at the middle layer of the residual stream) to test causality:

Intervention: Adding or subtracting the SAE decoder vector of a specific latent from the residual stream.
Goal: To see if steering these latents can recover the accuracy lost during fine-tuning on unknown facts.
Control Group: A baseline of "non-trending" latents is used to ensure gains are not artifacts of arbitrary perturbation.

3. Key Contributions

MoRFI Algorithm: A general-purpose, statistically rigorous method for identifying SAE latents that exhibit monotonic trends across controlled fine-tuning conditions, distinguishing them from transient noise.
Causal Link to Hallucination: The paper demonstrates that exposure to unknown facts disrupts knowledge retrieval along specific, identifiable directions in the residual stream.
Knowledge Recovery via Steering: The authors show that single-latent interventions can recover 69–85% of the knowledge lost during fine-tuning on unknown data, effectively reversing hallucinations at inference time.
Asymmetry in Latent Dynamics:
- Negative Steering is Superior: Suppressing (negatively steering) features that increase with unknown data yields larger accuracy gains than amplifying features that decrease.
- Decreasing Latents are Impactful: Features that naturally decrease when exposed to unknown data, when positively steered, also recover significant knowledge, suggesting they encode access pathways to stored knowledge that are being suppressed.

4. Key Results

Performance Degradation: Increasing the percentage of unknown facts in the training mix consistently degrades model performance, with the effect exacerbated by longer training (more epochs).
Steering Efficacy:
- On Llama 3.1 8B, the best negatively steered latent recovered 33.4% relative accuracy (vs. a baseline of 17.8%).
- On Mistral 7B, the best negatively steered latent recovered 39.77% relative accuracy.
- Control Group: Steering with non-trending latents resulted in negligible or negative performance changes, validating that the gains are specific to the identified causal features.
Knowledge Recovery Rate ( $R_K$ ): Of the facts recovered by steering, 69–85% were originally known to the base model (MD0). This confirms the intervention restores access to pre-trained knowledge rather than learning new facts.
Sparse Signal: Single-latent interventions outperformed steering with the composite "difference-in-means" direction ( $\delta_u$ ). This indicates that the knowledge-relevant signal is sparse and concentrated in a few specific latents, while the composite direction contains noise that dilutes the effect.
Semantic Alignment: The recovered facts predominantly relate to geographic predicates (e.g., Country, Capital, Country of Origin), and the top-performing latents show semantic alignment with these concepts in external tools like Neuronpedia.

5. Significance and Implications

Mechanistic Interpretability: The paper moves beyond correlation to establish a causal link between specific SAE latents and the model's ability to access stored knowledge. It suggests that "forgetting" during fine-tuning is not the erasure of knowledge but a disruption of access pathways.
Inference-Time Mitigation: The findings suggest that hallucinations caused by fine-tuning can be mitigated at inference time through targeted activation steering, without the need for retraining or complex retrieval-augmented generation (RAG) systems.
Safety and Alignment: By identifying the specific "misaligned" or "knowledge-disrupting" features, this work provides a pathway for safer fine-tuning practices and targeted interventions to prevent models from hallucinating when exposed to novel or conflicting information.
Generalizability: The MoRFI framework is model-agnostic and applicable to any controlled variable inducing a sequence of model checkpoints, offering a new tool for analyzing model behavior changes.

In conclusion, MoRFI provides a robust method to pinpoint the internal mechanisms of knowledge degradation in LLMs and demonstrates that these degradations are reversible through precise, single-latent interventions, offering a promising avenue for reducing hallucinations in post-trained models.