💬 NLP

CRANE: Knowledge Editing for Reasoning MLLMs

This paper introduces CRANE, a retrieval-augmented framework designed to overcome the unique failure modes of knowledge editing in reasoning multimodal large language models by combining dual-library retrieval with a two-phase training strategy to achieve high grounded success and edit independence without modifying model parameters.

Original authors: Han Huang, Hao Wang, Mengqi Zhang, Shu Wu, Qiang Liu, Liang Wang

Published 2026-06-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Han Huang, Hao Wang, Mengqi Zhang, Shu Wu, Qiang Liu, Liang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Spurious Success" Trap

Imagine you have a very smart, artistic student (a Reasoning Multimodal Large Language Model) who always explains their work step-by-step before giving the final answer. They write their thoughts in a special notebook (the Chain-of-Thought or CoT) before writing the final result.

Researchers tried to "edit" this student's knowledge. For example, they wanted to teach the student that a specific building in a photo is actually "Brasenose College," even though the photo clearly shows a different building (the "National Autonomous University of Mexico").

The Trap:
When researchers tested the student using a traditional method called "Teacher-Forcing," it looked like a perfect success (100% score).

How it worked: The teacher forced the student to write the correct answer ("Brasenose") without letting them think.
The Reality: The student never actually used their reasoning notebook. They just parroted the answer.

The Real Test:
When the researchers let the student think freely, the student looked at the photo, opened their reasoning notebook, and said: "Wait, this picture clearly shows the Mexican university. Even though you told me it's Brasenose, I can see it's not. I'm going to stick with what I see."

The student rejected the new fact because their reasoning process was too strong. Traditional tests missed this failure entirely because they never looked inside the reasoning notebook.

The Three Ways Editing Fails

The paper identifies three specific ways current methods fail when trying to update these "thinking" models:

Structural Collapse (Breaking the Notebook):
- The Analogy: Imagine trying to update a student's knowledge by rewriting their brain chemistry (changing the model's weights).
- The Result: The student gets so confused that they forget how to use their special notebook format. They stop writing "Step 1, Step 2..." and just start babbling or repeating words. The "thinking" structure collapses.
- The Paper's Finding: Methods that change the model's internal weights (like Fine-tuning or LoRA) completely destroy the model's ability to generate valid reasoning chains.
Cognitive Dissonance (The "I See It" Conflict):
- The Analogy: The student has a new fact in their head ("This is a red car"), but the photo shows a blue car.
- The Result: The student's reasoning process is so strong that they look at the photo, realize the new fact contradicts reality, and actively argue against the new fact. They say, "You told me it's red, but my eyes say blue, so I'll go with blue."
- The Paper's Finding: Even if the new fact is stored correctly, the model's visual reasoning overrides it.
Shallow Internalization (The "Rote Memorization" Problem):
- The Analogy: The student memorizes the answer to a specific question: "What is the name of this building?" -> "Brasenose."
- The Result: If you ask the exact same question, they get it right. But if you ask, "What city is this building in?" or show a different photo of the same building, they fail. They haven't truly learned the concept; they just memorized the specific question-answer pair.
- The Paper's Finding: Methods that store facts in an external memory (like a lookup table) fail when the question is rephrased or the image changes.

The Solution: CRANE

The authors propose a new system called CRANE (Counterfactual Reasoning Arbitration for Multi-modal kNowledge Editing). Instead of trying to rewrite the student's brain or force them to memorize, CRANE acts like a smart librarian and a coach.

How CRANE Works:

No Brain Surgery (Retrieval-Augmented):
- CRANE doesn't change the model's internal weights (avoiding "Structural Collapse").
- Instead, it has a library of facts. When a question comes in, it looks up the answer in the library and hands it to the model.
The Coach (Two-Phase Training):
- Phase 1 (Supervised Fine-Tuning): The model is taught the rules of the game. It learns: "Always use your reasoning notebook. If you see a conflict between the photo and the library fact, acknowledge the photo but follow the library fact if instructed."
- Phase 2 (The Reward System - GRPO): The model plays a game where it gets points for doing the right thing.
  - Normal Scenario: If the photo matches the fact, get points for citing the fact.
  - Conflict Scenario: If the photo contradicts the fact, get points for saying, "I see the photo says X, but the library says Y, so I will go with Y."
  - Irrelevant Scenario: If the photo has nothing to do with the library, get points for ignoring the library and using your own knowledge.

The Results

The paper tested CRANE on a new benchmark called ReasonEdit-Bench (a test suite designed specifically to catch these failures).

Success Rate: CRANE achieved a 96.9% success rate in conflict scenarios (where the photo and the new fact disagree). This is huge because other methods dropped to near 0% or single digits.
Reasoning Quality: Unlike other methods that just guessed the answer, CRANE's model actually used the new fact in its reasoning steps (multi-hop reasoning).
Independence: CRANE learned to ignore new facts when they didn't apply (locality), though it was slightly less perfect at this than at solving conflicts.

Summary

The paper argues that you can't just "patch" a smart, reasoning AI like you would a simple calculator. If you try to force a new fact onto it, it might break its thinking process or argue with you based on what it sees.

CRANE solves this by:

Not breaking the model's brain (no weight changes).
Giving the model a "library" of facts to look up.
Training the model to be a good referee that knows when to trust the library and when to trust its own eyes, all while keeping its reasoning steps intact.

The authors conclude that for these advanced "thinking" models, the key to editing knowledge is training the reasoning process, not just injecting the answer.

...` schema required by reasoning models. This results in format rates dropping to near zero (e.g., 0% for FT/LoRA/WISE on Vision-R1) or generating garbled text.
2. Cognitive Dissonance: Even when the edit fact is stored correctly, the model's reasoning chain actively rejects the injected fact in favor of visual priors when the image contradicts the edit. The model acknowledges the contradiction but ultimately sides with the visual evidence, leading to a "Grounded Success" (GS) of 0% in conflict scenarios.
3. Shallow Internalization: Methods that rely on external memory keyed to specific inputs (e.g., GRACE, IKE) succeed on exact queries but fail to generalize to rephrased questions or multi-hop reasoning chains. They fail to propagate the edited knowledge as an intermediate entity, reverting to original parametric knowledge or visual priors.

2. Methodology

A. Diagnostic Framework: ReasonEdit-Bench and CoT-Aware Evaluation

To expose these failures, the authors propose a CoT-aware evaluation protocol and construct ReasonEdit-Bench, an evaluation suite derived from VLKEB but enhanced with:

Conflict Stratification: Separating samples into "Normal" (image matches edit) and "Conflict" (image contradicts edit) categories.
Multi-Level Probes: Expanding edit targets from simple entity names to descriptive facts, testing both entity identification and attribute reasoning.
Multi-Hop Portability: Testing the propagation of edited facts through reasoning chains.

The protocol utilizes an LLM-as-a-Judge to inspect the full reasoning chain, introducing four key metrics:

Schema Integrity: Measures the fraction of outputs maintaining valid CoT format.
Grounded Success (GS): The primary reliability metric. It requires both the surface answer to match the edit target and the reasoning trace to explicitly ground the answer in the edit fact.
Edit Independence (EI): Measures whether the model avoids contaminating unrelated queries with irrelevant edit facts.
Intermediate Entity Used (IU) / Used Visual Instead (UVI): Multi-hop metrics assessing whether the edited entity is correctly propagated or if the model regresses to visual priors.

B. Proposed Solution: CRANE

The authors propose CRANE (Counterfactual Reasoning Arbitration for Multi-modal kNowledge Editing), a retrieval-augmented framework designed to avoid weight modification and thus prevent Structural Collapse. CRANE operates in two phases:

1. Modality-Aware Dual-Library Retrieval System

Dual-Index Architecture: Maintains separate FAISS indices for image embeddings (SigLIP) and text embeddings of edit facts to prevent semantic interference in conflict scenarios.
Contrastive Projection Head: A lightweight linear head trained with symmetric InfoNCE loss to improve entity discrimination in the embedding space, increasing hit@1 from 69.7% to 90.4%.
Confidence-Threshold Defense: Filters out spurious retrievals in locality scenarios (where the query image does not match any edit fact) by setting a similarity threshold, reducing false edit injections.

2. Two-Phase Training Strategy

Phase I: Supervised Fine-Tuning (SFT): The model is fine-tuned on 6,000 high-quality CoT examples to master the structural contract of editing. This phase teaches meta-skills: how to cite edit facts, how to recognize visual-textual conflicts, and how to identify irrelevance, without injecting specific factual knowledge.
Phase II: GRPO with Cognitive Routing Reward: The model is further refined using Group Relative Policy Optimization (GRPO). The reward function ( $R_{total}$ $R_{t o t a l}$ ) combines accuracy, format compliance, and a novel Cognitive Routing Reward ( $R_{reasoning}$ $R_{r e a so nin g}$ ) that adapts based on the scenario:
- Normal: Rewards explicit citation of the edit fact ( $R_{align}$ ).
- Conflict: Rewards a specific three-stage reasoning pattern: acknowledging visual evidence, using a pivot word (e.g., "however"), and committing to the edit fact ( $R_{override}$ ).
- Locality: Rewards recognizing the query as distinct from edit facts and avoiding hallucination ( $R_{isolate}$ ).

3. Key Contributions

Diagnostic Framework: Identification of three failure modes (Structural Collapse, Cognitive Dissonance, Shallow Internalization) and a CoT-aware evaluation protocol backed by an LLM Judge.
ReasonEdit-Bench: A comprehensive evaluation suite featuring conflict stratification, free-form knowledge expansion, and multi-hop portability tests.
CRANE Framework: A retrieval-augmented editing method that requires no per-edit parameter modification, utilizing SFT for structural initialization and GRPO with scenario-specific rewards to arbitrate between visual priors and edit facts.
Empirical Evidence: Comprehensive experiments demonstrating that traditional metrics (TF-Acc) are unreliable for reasoning MLLMs and that CRANE effectively mitigates the identified failure modes.

4. Results

Experiments were conducted on ReasonEdit-Bench and the out-of-distribution MMEVOKE benchmark using Qwen2.5-VL-7B-Instruct as the base model.

Conflict Resolution: CRANE achieves 96.9% Grounded Success on conflict scenarios, significantly outperforming baselines. For instance, while LoRA drops from 92.4% (normal) to 86.9% (conflict) on Qwen2.5-VL, and IKE collapses to 6.1%, CRANE maintains high performance.
Multi-Hop Reasoning: CRANE achieves 96.9% Intermediate Entity Used (IU) in conflict scenarios with only 1.7% UVI (regression to visual priors). In contrast, methods like GRACE and IKE achieve near-zero IU (<0.4%) and >91% UVI, indicating they do not genuinely internalize the knowledge.
Locality: CRANE achieves 97.6% Text Edit Independence and 68.1% Image Edit Independence. While lower than the near-perfect locality of GRACE (99.5%), the authors note that GRACE's locality is a byproduct of its inability to affect the model (failing reliability), whereas CRANE balances high reliability with non-trivial locality.
Out-of-Distribution (MMEVOKE): Under gold retrieval (oracle edit facts), CRANE reaches 87.0%, a 19.2% improvement over the base model. With fusion retrieval, it achieves 69.8%, surpassing the base model with gold retrieval (67.8%), confirming that the reasoning patterns generalize.

5. Significance and Claims

The paper claims that effective knowledge editing for reasoning MLLMs requires a paradigm shift from surface-level answer injection to training the model to reason about edit facts through its explicit CoT process.

Limitations of Current Metrics: The authors emphasize that TF-Acc is unreliable for reasoning models, with the gap between TF-Acc and Grounded Success reaching up to 99 percentage points in conflict scenarios.
Necessity of CoT-Awareness: The study demonstrates that methods failing to preserve the CoT schema or failing to resolve visual-textual conflicts are fundamentally unsuited for reasoning MLLMs.
CRANE's Approach: By decoupling retrieval from weight modification and using reinforcement learning to teach the model how to arbitrate between visual evidence and injected facts, CRANE addresses the root causes of failure without triggering Structural Collapse.
Current Bottleneck: The authors modestly note that while the reasoning component of CRANE generalizes well, the current bottleneck for out-of-distribution deployment is retrieval precision, suggesting future work should focus on domain-adaptive retrieval.