Technical Summary: Edit Knowledge, Not Just Facts via Multi-Step Reasoning over Background Stories

Problem Statement

Large Language Models (LLMs) possess vast world knowledge acquired through pretraining, yet they struggle to update this knowledge dynamically in real-world scenarios where facts change. Existing knowledge editing (KE) approaches primarily focus on atomic facts, often introducing new information via short declarative statements or direct question-answer pairs. While these methods can improve factual recall (Edit Success), they frequently fail to internalize the updated information. Consequently, models may correctly answer direct queries about a new fact but fail to apply that knowledge in downstream reasoning tasks, particularly multi-hop questions that require combining the new fact with pre-existing knowledge. This limitation, termed a lack of knowledge portability, suggests that current methods encourage surface-level memorization rather than integrating updates into the model's coherent internal world representation.

Methodology

The authors propose that effective knowledge update is fundamentally a reasoning problem rather than a memorization problem. To address this, they introduce a reasoning-centric training framework consisting of three core principles:

1. Contextualized Knowledge Representation (Background Stories): Instead of presenting updates as isolated atomic facts, new knowledge is introduced as a coherent background story (e.g., a news article or biography). This narrative contextualizes the novel fact, explaining its relation to existing entities and knowledge, thereby mimicking how information is naturally acquired in the real world.
2. Multi-Hop Question Generation: For each new fact, the framework generates multi-hop questions that explicitly require the new information as an intermediate step in the reasoning chain. These questions are self-generated by prompting the model to combine the new fact with related "anchor" facts from pre-existing knowledge. This ensures the model is trained to use the update as a necessary component of reasoning, not just as a terminal answer.
3. Context Distillation (Teacher-Student Setup): A teacher-student training procedure is employed to internalize the reasoning behavior.
   • The teacher model has access to the background story (context) and answers the generated multi-hop questions.
   • The student model is trained to imitate the teacher's output distribution but is conditioned only on the question, with the background story randomly dropped (context dropout rate of 0.9).
   • This forces the student to internalize both the new knowledge and the associated reasoning behavior without relying on the explicit context at inference time. The training minimizes the Kullback-Leibler (KL) divergence between the teacher and student distributions, utilizing LoRA adapters for efficient parameter updates.

The framework supports two supervision modes: Answer-only (final answer only) and Reasoning-trace (intermediate reasoning steps + final answer).

Key Contributions

• Reframing Knowledge Editing: The paper argues that the failure of existing KE methods to support flexible reasoning stems from a lack of reasoning-based training, not insufficient memorization.
• Reasoning-Centric Framework: A novel training strategy that introduces knowledge as contextualized stories, enforces usage via generated multi-hop questions, and distills reasoning behavior through context distillation.
• New Benchmark (FictBio): The introduction of FictBio, a dataset derived from MQuAKE-T but utilizing fictional individuals and realistic update scenarios (e.g., changes in CEOs or government officials) to avoid contamination from pre-existing associations with real-world figures.
• Empirical Validation: Comprehensive evaluation on FictBio, MQuAKE-CF-3k, and ReCoE benchmarks using state-of-the-art reasoning models (Qwen3-32B) and standard instruction-tuned models (Llama 3.1-70B).

Experimental Results

Experiments demonstrate that the proposed method significantly outperforms existing baselines (including ROME, MEMIT, AlphaEdit, ICE, and SFT) in knowledge portability while maintaining high Edit Success and Locality (preservation of unrelated knowledge).

• Portability Gains: The method substantially improves performance on multi-hop reasoning tasks requiring updated knowledge. On average, it improved portability by 10% on FictBio, 90% on MQuAKE-CF, and 20% on ReCoE compared to the best baselines.
• Reasoning Integration: Unlike baselines that often fail under Chain-of-Thought (CoT) evaluation, the proposed method enables models to successfully leverage updated facts during extended reasoning chains.
• Supervision Modes:
  • For realistic updates (FictBio), Answer-only supervision proved sufficient and often superior, as reasoning-trace supervision sometimes introduced noise from irrelevant intermediate steps.
  • For counterfactual updates (MQuAKE-CF, ReCoE) that strongly conflict with pre-training priors, Reasoning-trace supervision was more effective at overriding conflicting priors during CoT generation.
• Sequential Editing: The method demonstrated stability during sequential batch editing, showing no sharp performance collapse and effectively retaining previously edited knowledge.
• General Capabilities: Evaluation on the MMLU benchmark showed no degradation in general model capabilities, indicating the updates do not harm the model's broader reasoning abilities.
• Ablation Studies: Replacing background stories with atomic facts resulted in significant performance drops, particularly in portability, confirming the importance of contextualized representation.

Significance and Claims

The paper claims that its primary contribution is shifting the paradigm of knowledge editing from memorization to reasoning. By treating knowledge updates as a reasoning problem, the proposed framework successfully densifies the integration of new facts into the LLM's internal world model. This allows models to not only recall updated facts but to flexibly apply them in complex, multi-step reasoning scenarios. The authors assert that this approach addresses a critical gap in current KE literature, enabling LLMs to remain useful and accurate in dynamic environments where knowledge evolves. The work highlights that for reasoning-oriented models, knowledge must be incorporated within the internal reasoning process, a goal achieved through the proposed multi-hop, story-based, distillation strategy.