Towards Scalable Lifelong Knowledge Editing with Selective Knowledge Suppression

1. Problem Statement

Large Language Models (LLMs) suffer from static knowledge limitations, leading to hallucinations and outdated facts. While Knowledge Editing (KE) aims to update specific facts without full retraining, existing methods face significant challenges in lifelong (sequential) editing scenarios:

• Catastrophic Forgetting & Model Collapse: Parameter-modifying methods (e.g., ROME, MEMIT) often degrade model performance or overwrite unrelated knowledge when edits are applied sequentially.
• High Training Costs & Poor Scalability: Retrieval-augmented methods (e.g., LTE, RECIPE) improve stability but require extensive fine-tuning on specific datasets. This leads to high computational costs and poor generalization when applied to unseen datasets or domains.
• Trade-offs: Existing approaches struggle to simultaneously satisfy Reliability (correctness of edited facts), Generality (robustness to paraphrasing), and Locality (preservation of unedited knowledge).

2. Methodology: LightEdit

The authors propose LightEdit, a lightweight framework designed for scalable lifelong knowledge editing. It operates without modifying the underlying model parameters, relying instead on Edit-Aware Selection and In-Context Decoding.

A. Core Architecture

1. External Knowledge Memory: Edited facts are stored as natural language triples $(s, r, o^*)$ in an external memory $G^*$.
2. Edit-Aware Selector (EAS):
   • Function: A cross-encoder module (based on XLM-RoBERTa) that acts as a filter. It evaluates the relevance between a user query $q$ and retrieved candidates from $G^*$.
   • Mechanism: It frames the selection as a Natural Language Inference (NLI) task, predicting whether a knowledge item entails the query.
   • Goal: To retrieve only relevant knowledge for the current query, preventing irrelevant or noisy retrieved facts from interfering with the model's generation (preserving Locality).
3. In-Context Decoding (ICD):
   • Function: A probability-based inference strategy that suppresses the model's original knowledge probabilities without retraining.
   • Mechanism: When the model generates a response, ICD adjusts the log-probability of the first token. It compares the probability of generating the token based on the new context ($\ell_{new}$) versus the prior knowledge ($\ell_{prior}$).
   • Formula: The adjusted log-probability is defined as:
     $$\log p'(x_1) = \ell_{new} - \alpha \ell_{prior}$$
     Where $\alpha$ is a hyperparameter controlling the suppression strength of the outdated knowledge.
   • Goal: To force the model to prioritize the updated information provided in the context over its pre-trained weights, enhancing Reliability and Generality.

B. Workflow

1. Receive a query $q$.
2. Retrieve top-$k$ candidates from external memory $G^*$.
3. Pass candidates through the Edit-Aware Selector to filter for relevance.
4. Construct a prompt containing the query and selected knowledge.
5. Perform inference using In-Context Decoding to suppress the probability of the original (outdated) answer token.

3. Key Contributions

• Identification of Scalability Bottlenecks: The paper highlights that current lifelong editing methods are hindered by high training costs and dataset-specific overfitting, limiting their real-world applicability.
• LightEdit Framework: A novel approach that combines retrieval-based filtering with probabilistic decoding. It achieves editing effectiveness without parameter updates, significantly reducing computational overhead.
• Selective Knowledge Suppression: The introduction of In-Context Decoding allows for dynamic suppression of outdated knowledge probabilities during inference, effectively resolving knowledge conflicts in a single forward pass.
• Comprehensive Evaluation: Extensive experiments across multiple datasets (ZSRE, Counterfact, RIPE) and models (LLaMA-3, GPT-J) demonstrate superior performance compared to state-of-the-art baselines.

4. Experimental Results

The authors evaluated LightEdit on three benchmarks using LLaMA-3 (8B) and GPT-J (6B).

• Performance Metrics:
  • ZSRE: LightEdit achieved an average score of 0.9664, outperforming RECIPE (0.9380) and LTE (0.7934).
  • Counterfact: Achieved 0.9296 (vs. RECIPE's 0.7664).
  • RIPE: Achieved 0.9818 (vs. RECIPE's 0.5473).
  • Locality: Maintained near-perfect locality scores (>0.99) across all datasets, indicating minimal disruption to unedited knowledge.
• Scalability & Stability:
  • Unlike baselines that degrade rapidly as the number of sequential edits increases, LightEdit maintained consistent performance across 1,000+ edits.
  • Efficiency: LightEdit requires zero edit time (no parameter training) and minimal inference overhead compared to fine-tuning methods (e.g., MEMIT takes ~12s for editing; LightEdit takes ~0s).
• Ablation Studies:
  • Removing the Edit-Aware Selector significantly hurt Locality (irrelevant facts corrupted the output).
  • Removing In-Context Decoding hurt Reliability and Generality (the model reverted to old facts).
  • The optimal suppression coefficient $\alpha$ was found to be 0.2.
• General Capabilities: LightEdit did not degrade the model's performance on general benchmarks (MMLU, CSQA, ANLI), unlike parameter-modifying methods which often caused significant drops in general capability.

5. Significance

• Practical Deployment: LightEdit offers a cost-effective solution for maintaining LLMs in dynamic environments. By eliminating the need for retraining, it enables real-time knowledge updates with minimal computational resources.
• Overcoming Trade-offs: It successfully breaks the traditional trade-off between editing accuracy and model stability, achieving high scores in Reliability, Generality, and Locality simultaneously.
• Future Direction: The work suggests that probabilistic control during inference is a viable and scalable alternative to parameter modification for lifelong learning, opening new avenues for efficient knowledge management in large-scale models.

Limitations Noted:

• The Edit-Aware Selector requires supervised training (though minimal compared to full fine-tuning).
• Performance may vary with different hyperparameters ($\alpha$) across domains.
• Current evaluation is limited to specific benchmarks and may not fully capture complex open-domain reasoning scenarios.