Technical Summary: Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

1. Problem Statement

Large Language Models (LLMs) rely on static corpora, causing their knowledge to become outdated in fast-changing environments. While Knowledge Editing (KE) aims to update specific facts without retraining, existing methods struggle when transitioning from structured triples to Unstructured Knowledge Editing (UKE). In UKE, new knowledge is provided as a free-form passage containing multiple entangled facts (e.g., an acquisition announcement).

The paper identifies a critical failure mode in current UKE methods: a lack of composability. Even when models successfully memorize an edited passage, they fail to:

1. Decompose: Extract and answer atomic questions about individual facts within the passage without merely regurgitating the entire text.
2. Compose: Combine multiple injected facts to perform multi-hop reasoning.

The authors attribute this failure to the passive reliance of existing editors on the fixed editing passage as the sole learning source. This induces severe overfitting and context reliance, preventing the model from generalizing the new knowledge to unseen query structures. Furthermore, standard On-Policy Self-Distillation (OPSD)—where the model learns from its own generated trajectories guided by a privileged in-context state—fails in UKE because the model's own rollouts often deviate from the new facts (a "coverage failure") before the privileged model can provide corrective supervision.

2. Methodology: Hybrid-Policy Self-Editing (HPSE)

To address the coverage failure and improve composability, the authors propose HPSE, a plug-and-play, data-centric approach that modifies the training signal for gradient-based KE editors without assuming specific parameter update mechanisms.

Core Mechanism

HPSE reframes editing as proactive self-distillation from a privileged in-context state ($\pi^\star$) of the same model (the base model $\pi_0$ reading the new passage $c$). Unlike pure OPSD, which relies solely on the student model's ($\pi_\theta$) own rollouts, HPSE constructs a hybrid rollout ($\pi_\rho$) that dynamically switches between the student and the privileged teacher.

The Hybrid Rollout Process:
At each generation step $t$, given a prefix $y_{<t}$, the next token $y_t$ is sampled from $\pi_\rho$:

• Student Token: Sampled from $\pi_\theta$ (the edited model) if the policies agree or the student is confident.
• Privileged Step-in: Sampled from $\pi^\star$ if a "step-in" condition is triggered.

Step-In Conditions:
The privileged model intervenes when:

1. Gap Condition: The log-probability gap between the privileged model and the student for the privileged's greedy token ($y^\star_t$) exceeds a threshold $\tau$: $\log \pi^\star(y^\star_t) - \log \pi_\theta(y^\star_t) > \tau$.
2. Confidence Condition: The privileged model is confident in its prediction: $\pi^\star(y^\star_t) > \kappa$.

This mechanism ensures that when the student's trajectory deviates from the new facts (off-topic), the privileged model "steps in" to place the missing facts onto the student's trajectory precisely where coverage fails.

Objective Function

The HPSE loss function combines the hybrid distillation term with a standard Negative Log-Likelihood (NLL) anchor:
$$J_{HPSE}(\theta) = \mathbb{E}_{y \sim \pi_\rho} \left[ \sum_{t} D_{KL}(\pi^\star(\cdot|x, y_{<t}) \parallel \pi_\theta(\cdot|x, y_{<t})) \right] - \lambda \log \pi_\theta(c|x)$$

• The hybrid term supervises the new facts along the student's own trajectory (via the step-in mechanism).
• The NLL anchor ($\lambda J_{NLL}$) preserves the standard passage-level likelihood to ground the knowledge under the passage's own prefixes.

Theoretical Advantage

The authors provide a theoretical analysis showing that for a fact span of length $\ell$, the hybrid rollout visits every fact prefix with probability 1, whereas pure OPSD rollouts visit deep prefixes with probability decaying exponentially ($e^{-\tau j}$). Consequently, the supervision signal in HPSE scales linearly with the span length ($\Omega(\ell)$), while OPSD's signal remains constant ($O(1)$). This advantage grows as the injected knowledge becomes more complex and longer.

3. Key Contributions

1. Problem Formulation: The paper introduces an untargeted regime for UKE, where editing prompts are generic (e.g., "Introduce X") rather than specifying which facts are updated. It defines composability (decomposition and composition) as a unified requirement for practical UKE and demonstrates that existing methods fail this requirement.
2. Method Proposal (HPSE): HPSE is the first method to address the coverage failure of on-policy distillation in UKE by using a hybrid rollout strategy. It is plug-and-play, requiring no external data, auxiliary models, or specific parameterization of the edit, making it compatible with various gradient-based editors (e.g., LoRA, FT-M).
3. Theoretical Justification: The paper provides a formal analysis proving that HPSE offers a fact-directed supervision advantage over pure on-policy distillation, with the signal strength growing linearly with the length of the injected knowledge.
4. Empirical Validation: Extensive experiments across four LLM backbones (Qwen2.5, Qwen3, Llama-3.1, Gemma-2) and two KE editors (FT-M, LoRA) show consistent improvements in both decomposition and composition metrics under single and continual editing settings.

4. Experimental Results

The authors evaluated HPSE on two benchmarks adapted for composability: UnKEBench (decomposition) and MQuAKE-uns (composition).

• Single Editing: HPSE significantly improved the performance of both FT-M and LoRA. For instance, on Qwen2.5, HPSE boosted LoRA's composition score (Cmp.) by +21.2% and decomposition score (Dmp.) by +25.6%. It consistently outperformed baselines like MEMIT, AlphaEdit, AnyEdit, and UnKE, particularly in multi-hop reasoning tasks where baselines often hallucinated or reverted to prior knowledge.
• Continual Editing: In settings with sequential edits ($T=10$ to $100$), HPSE maintained robustness. On MQuAKE-uns, LoRA with HPSE improved its average score by +55% to +149% relative to the base version across different sequence lengths, often doubling performance.
• Ablation Studies: Removing the hybrid rollout ("w/o HP") reduced performance to levels below the base editor, confirming the necessity of the step-in mechanism. Removing the NLL anchor ("w/o NLL") degraded joint recall, showing the anchor's role in grounding.
• Case Studies: Visual examples demonstrate that while baselines often repeat the passage verbatim or hallucinate, HPSE successfully answers atomic questions directly and chains multiple facts for multi-hop reasoning.

5. Significance and Claims

The paper claims that composability is a critical, previously overlooked requirement for practical Unstructured Knowledge Editing. It argues that passive injection of knowledge leads to memorization without utility.

The significance of HPSE lies in its ability to transform editing from a passive data-fitting task into an active self-correction process. By proactively constructing training signals through hybrid rollouts, HPSE enables LLMs to internalize new knowledge in a way that is both decomposable (flexibly extractable) and composable (reasoning-capable). The authors emphasize that HPSE offers a "self-contained and affordable path" to practical KE, as it requires no external datasets or powerful teacher models, making it a generalizable solution for updating LLMs in dynamic, real-world scenarios.

The paper concludes by noting that while HPSE currently focuses on text-based editing, future work aims to generalize the approach to lifelong and multimodal editing.