💬 NLP

Benchmarking Knowledge Editing using Logical Rules

This paper introduces a new benchmark for evaluating knowledge editing in Large Language Models that assesses logical consequences via multi-hop questions, revealing that current methods often fail to propagate entailed knowledge despite successfully inserting direct facts.

Original authors: Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Updating a Brain Without Breaking It

Imagine you have a very smart, but slightly outdated, encyclopedia written by a robot (this is a Large Language Model or LLM). One day, you realize a fact in the book is wrong. For example, the book says "Valerie Hobson is a citizen of the UK," but she actually moved to Australia and became an Australian citizen.

In the past, to fix this, you would have to tear out every page of the encyclopedia and rewrite the whole thing from scratch. That takes forever and costs a fortune.

Knowledge Editing is like using a magic pen to just cross out "UK" and write "Australia" without rewriting the whole book. The goal is to update one specific fact quickly.

The Hidden Trap: The "Butterfly Effect" of Facts

The authors of this paper noticed a major flaw in how we test these magic pens. Most tests only check if the robot got the one fact right. They ask: "Is Valerie Hobson a citizen of Australia now?" If the robot says "Yes," the test is a pass.

But the authors argue this is like checking if a house is safe by only looking at the front door. They realized that facts are connected like a web. If you change Valerie's citizenship, other facts might need to change too, even if you didn't ask about them directly.

The Analogy:
Imagine you tell a friend, "I am now living in Australia."

Direct Fact: You are in Australia.
Logical Consequence (The Hidden Trap): If your friend knows that "People living in Australia usually have Australian citizenship," they should automatically assume you are an Australian citizen.

If your friend updates their memory to say you live in Australia, but still thinks you are British, their logic is broken. The paper calls this correlated knowledge. The robot might know the new fact, but it fails to update the logical consequences of that fact.

The New Tool: A "Logic Detective"

To fix this, the authors built a new Benchmark (a test) that acts like a logic detective. Here is how their system works, step-by-step:

The Edit: They take a robot and change one fact (e.g., "Valerie Hobson lives in Australia").
The Rulebook: They use a special tool (called AMIE3) to look at a giant map of facts (a Knowledge Graph) and find the "rules" that connect facts.
- Rule found: "If Person X is married to Person Y, and Person X lives in Country Z, then Person Y usually has the same citizenship as Country Z."
The Trap Question: Instead of just asking about Valerie, the system uses those rules to generate a tricky question about her husband.
- Question: "What is the nationality of Valerie Hobson's husband?"
- Correct Answer: Australian (because of the rule).
- Old Robot Answer: British (because it only updated the direct fact, not the logic).

What They Found: The "Direct vs. Indirect" Gap

The authors tested popular methods (like ROME and FT) using this new logic detective. The results were surprising:

The Direct Win: When asked the simple question ("Where does Valerie live?"), the robots were great. They got the direct edit right almost every time.
The Logical Loss: When asked the "trap" questions about the husband or other connected facts, the robots failed miserably.
- In some cases, the robots were 24% worse at answering the logical questions than the direct ones.

The Metaphor:
It's like teaching a student a new math formula.

Direct Edit: You ask, "What is 2 + 2?" The student says "4." (Perfect!)
Logical Consequence: You ask, "If 2 + 2 is 4, and I have two apples, how many apples do I have?" The student says "I don't know" or "5."
The student memorized the answer but didn't understand the logic behind it.

The Conclusion

The paper concludes that current methods for updating AI brains are too "dumb" to handle the ripple effects of changes. They are good at patching a single hole in a boat but bad at ensuring the whole boat doesn't sink because of the patch.

They found that while some methods (like Knowledge Neurons) did slightly better at handling these logical connections, most popular methods fail to update the "web of truth" that surrounds a single fact. They need a new way to test AI that checks not just if the fact is right, but if the logic holding the facts together is still intact.

Technical Summary: Benchmarking Knowledge Editing using Logical Rules

Problem Statement
Large Language Models (LLMs) are increasingly deployed in applications requiring up-to-date knowledge, yet retraining these models is computationally prohibitive. Knowledge Editing (KE) techniques have emerged to update specific facts without full retraining. However, current benchmarks for KE primarily evaluate the model's ability to recall the directly edited assertion (e.g., "Who is the President?"). They largely neglect the model's ability to maintain logical consistency with correlated or entailed knowledge (e.g., "Who is the spouse of the President?"). Existing methods often successfully inject a direct fact but fail to propagate the logical consequences of that edit through the model's reasoning chains, leading to inconsistencies in multi-hop queries. Furthermore, existing benchmarks that do address multi-hop reasoning (such as MQuAKE) often rely on manually curated questions, limiting their scalability and adaptability across different domains.

Methodology
The authors propose a novel benchmark framework that automatically evaluates KE methods by testing their ability to handle logical consequences derived from a single fact edit. The workflow consists of four main stages:

Knowledge Graph Construction: Starting from a benchmark dataset (questions), the system extracts entities using spaCy and retrieves associated triples from DBpedia via SPARQL to construct a subgraph ( $G$ ).
Logical Rule Mining: The system employs AMIE3, a rule-mining system, to extract logical rules ( $\Phi$ ) from the constructed knowledge graph. These rules represent relationships such as $bornIn(X, Y) \land livedIn(X, Y) \Rightarrow citizen(X, Y)$ .
Test Case Generation: Using the mined rules and the original triples, a Large Language Model (e.g., GPT-4) is prompted to generate multi-hop Question-Answer (QA) pairs. These questions are designed to verify if an edit to a base assertion (e.g., changing a citizenship) correctly propagates to dependent facts (e.g., the nationality of a spouse) via the logical rules.
Evaluation: The benchmark evaluates edited models ( $L^*$ $L^{*}$ ) on two criteria:
- Edit Success: The probability of the model correctly answering the direct query.
- Logical Consistency: The model's ability to answer the generated multi-hop questions correctly, measured by Exact Match (EM) and F1 scores.

The framework was tested on two datasets: MLaKE (multilingual, single-hop and multi-hop) and MQuAKE (temporal knowledge updates). The evaluation focused on parameter-based editing methods: ROME (Rank-One Model Editing), FT (Constrained Fine-tuning), and KN (Knowledge Neurons), using GPT-2 variants as base models.

Key Contributions

A New Benchmark Approach: The paper introduces a scalable, automated benchmark that augments KE evaluation with correlated knowledge derived from logical rules, moving beyond simple fact recall.
Automated Test Case Generation: Unlike manual benchmarks, this approach uses AMIE3 and LLMs to generate domain-adaptable multi-hop questions, ensuring broad coverage without labor-intensive curation.
Comprehensive Evaluation: The authors release an open-source framework and conduct extensive experiments comparing ROME, FT, and KN on both direct and entailed knowledge.

Results
The experiments revealed a significant performance gap between direct editing and logical consistency:

ROME and FT: These methods demonstrated high success in inserting direct assertions but exhibited substantial failures in logical consistency. For instance, ROME showed a performance drop of up to 24% when evaluated on correlated knowledge compared to direct knowledge. FT results were even more pronounced, with near-zero performance on correlated queries, suggesting these methods struggle to propagate edits through reasoning chains.
Knowledge Neurons (KN): In contrast, KN demonstrated relatively better performance in generalizing to correlated knowledge, showing higher F1 and EM scores on multi-hop questions compared to the other methods.
Model Sensitivity: The results varied across model sizes (gpt2-medium, large, xl), but the trend of poor logical propagation for ROME and FT remained consistent.

Significance and Claims
The paper argues that the current state of Knowledge Editing is insufficient for real-world deployment where facts are interdependent. The authors claim that while existing methods can accurately recall edited assertions, they frequently fail to inject entailed knowledge, leading to logical incoherence. The proposed benchmark highlights this critical limitation, demonstrating that semantics-aware evaluation frameworks are necessary to ensure that edits propagate coherently across interconnected facts. The authors modestly conclude that their work exposes weaknesses in current KE methods regarding reasoning chains and provides a tool for future research to improve robustness and consistency. They suggest future work should extend these evaluations to newer LLMs (e.g., LLaMA, Mistral) and larger knowledge graphs like Wikidata.