SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstention

1. Problem Statement

The paper addresses a critical failure mode in Large Language Model (LLM) adaptation: the loss of aligned epistemic abstention during knowledge fine-tuning.

• Epistemic Abstention: The ability of a model to correctly acknowledge when it does not know the answer to a query (e.g., saying "I don't know" or "I am not aware") rather than hallucinating a fabricated answer.
• The Failure: While base models often exhibit robust abstention on unseen or out-of-distribution (OOD) data, standard fine-tuning (even parameter-efficient methods like LoRA) to inject new knowledge often erodes this behavior. The model becomes overconfident and hallucinates on queries outside its new training distribution.
• High-Stakes Impact: In domains like medicine, law, or enterprise search, this failure is dangerous. A model fine-tuned on specific records should not hallucinate details about unseen but related cases.
• Current Limitations: Existing solutions often require post-hoc re-alignment stages, access to proprietary alignment data, or explicit probing of the model's epistemic boundaries, making them impractical for lightweight or privacy-sensitive adaptation scenarios.

2. Methodology: SEAT (Sparse Entity-Aware Tuning)

The authors propose SEAT, a preventive fine-tuning method that preserves abstention behavior without requiring external alignment data. The method is grounded in the hypothesis that knowledge fine-tuning causes excessive representational drift in the model's activation space, displacing the linear directions associated with abstention.

SEAT combines two complementary components:

A. Sparse Updating (Global Stability)

• Mechanism: Instead of updating all parameters, SEAT applies a binary mask to freeze a large fraction of the model's parameters (e.g., 80% frozen, 20% trainable).
• Goal: This constrains the global displacement of residual stream activations. By limiting the degrees of freedom available to the optimizer, it prevents the model from drifting too far from its original representation space, thereby preserving the "abstention manifold" for unseen inputs.
• Implementation: A random mask is used as a simple, model-agnostic instantiation, though magnitude-based or importance-based masks are also explored.

B. Entity-Perturbed Regularization (Local Sharpness)

• Mechanism: SEAT introduces a regularization term based on Kullback-Leibler (KL) divergence.
• Process:
  1. Take the fine-tuning dataset ($D_{ft}$).
  2. Create a perturbed dataset ($\tilde{D}$) by replacing the subject entity in each training example with a fictitious alternative (e.g., changing "John Doe" to "Alice Smith") while keeping the context structure identical.
  3. Compute the KL divergence between the base model's output and the fine-tuned model's output on these perturbed inputs.
• Goal: This prevents "knowledge spillover." It ensures that learning facts about a specific entity does not inadvertently generalize to semantically or token-wise neighboring entities that were not in the training set. It sharpens the local epistemic boundaries.
• Loss Function:
  $$L_{SEAT} = L_{FT} + \gamma L_{KL}$$
  Where $L_{FT}$ is the standard fine-tuning loss, and $L_{KL}$ is the regularization term on entity-perturbed inputs.

3. Key Contributions

1. Identification of a New Failure Mode: The paper formally identifies and characterizes the collapse of aligned epistemic abstention as a distinct form of catastrophic forgetting, separate from forgetting task-specific utility.
2. Theoretical Insight: It provides empirical and theoretical evidence linking the loss of abstention to excessive representational drift in the residual stream, formalizing the bound between activation displacement and abstention score degradation.
3. SEAT Algorithm: A simple, effective training recipe that requires no alignment data, no explicit boundary probing, and no post-hoc re-alignment.
4. Comprehensive Evaluation: Extensive experiments across different models (Llama3-8B, Qwen2.5-7B) and datasets (synthetic PISTOL/TOFU, real-world news RWD).

4. Experimental Results

The authors evaluated SEAT against strong baselines, including Full Fine-Tuning, LoRA, EWC, CLoRA, R-tuning, and Experience Replay.

• Preservation of Abstention (IDK Scores):
  • SEAT achieved near-perfect preservation of abstention behavior on unseen queries.
  • Human Evaluation (IDKHA): SEAT improved abstention scores by 18%–101% over the strongest baselines. For example, on the PISTOL dataset, SEAT achieved an IDKHA score of 0.954, whereas standard Fine-Tuning and LoRA dropped to near 0.000–0.016.
  • String Matching (IDKSM): SEAT consistently maintained high scores, indicating the model correctly refused to answer unknown questions.
• Knowledge Acquisition (FT Score):
  • Despite the constraints, SEAT maintained near-perfect fine-tuning effectiveness (FT scores $\approx$ 1.0), proving that sparse updates do not hinder the ability to learn new target knowledge.
• Ablation Studies:
  • Both Sparse Updating and Entity Perturbation (EP) were found to be essential. Using only sparse tuning resulted in lower abstention preservation (0.806) compared to the full SEAT method (0.954).
  • Different masking strategies (random, magnitude, importance) showed that while the choice of mask matters for sparse tuning alone, the addition of EP consistently boosted performance across all strategies.
• Downstream Utility:
  • SEAT preserved the model's general capabilities (truthfulness, commonsense reasoning, math) comparable to the base model, confirming it does not degrade overall utility.
• Representation Analysis:
  • PCA visualizations showed that SEAT maintains a clear separation between activations for known (fine-tuned) data and unknown (unverifiable) data, whereas standard fine-tuning causes these clusters to collapse into one another.

5. Significance and Impact

• Safety in Adaptation: SEAT establishes that preserving epistemic abstention is a core objective for safe LLM adaptation, particularly in high-stakes environments where hallucination is unacceptable.
• Practicality: By eliminating the need for expensive re-alignment data or complex probing mechanisms, SEAT makes safe knowledge adaptation feasible for practitioners with limited resources or privacy constraints.
• Theoretical Advancement: The work bridges the gap between mechanistic interpretability (linear representations of behaviors) and practical fine-tuning, offering a concrete method to control representational drift.
• Generalizability: The method is model-agnostic and works across different architectures and dataset types (synthetic and real-world), suggesting a robust solution for the broader problem of catastrophic forgetting in LLMs.

In conclusion, SEAT demonstrates that it is possible to update an LLM with new knowledge without sacrificing its ability to admit ignorance, provided the adaptation process is constrained to limit global drift and prevent local knowledge spillover.