Technical Summary: Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs

Problem Statement

Unified Multimodal Models (UMMs) have emerged as a paradigm for general-purpose multimodal intelligence, utilizing a shared backbone to represent images and text jointly. While knowledge editing (KE) techniques have matured for text-only models—allowing specific facts to be updated without full retraining—it remains unclear whether edits applied solely through the textual modality propagate effectively to visual generation. The core question addressed is: if a UMM is edited to assert a new fact in text (e.g., "an apple is blue"), will it subsequently generate images reflecting this updated knowledge when prompted for visual synthesis? Current methods lack a systematic way to evaluate whether textual parameter updates translate into consistent, visually verifiable changes in generated images.

Methodology

The UNIKE Benchmark

To investigate this, the authors introduce UNIKE, the first benchmark specifically designed for cross-modality knowledge editing in UMMs.

• Scale and Scope: UNIKE comprises 2,971 edit subjects covering two axes of factual knowledge: Attribute edits (intrinsic properties like color, material, shape, size, pattern) and Relation edits (relational facts like occupation, location, affiliation, creator).
• Evaluation Instances: These subjects are expanded into 5,535 evaluation instances using a multi-stage design to test compositional generalization and grounding:
  1. Atomic object: Direct reference to the entity.
  2. Realistic context: Entity in varied lighting/backgrounds.
  3. Complex composition: Multi-entity scenes with interactions.
  4. Derived product: Attributes transferred to derivatives or uses (e.g., lemon juice instead of the lemon).
• Visual Verification: Unlike text-only benchmarks, UNIKE employs VQA-based visual verification. An automated judge (Qwen3-VL) evaluates whether generated images satisfy a visual target description ($t_{vis}$) and answer a specific VQA query ($q_{vqa}$) consistent with the edited fact. Crucially, image prompts are designed to be "answer-neutral," ensuring the model relies on internal edited knowledge rather than prompt leakage.

Evaluation Protocols

The authors evaluate models under two protocols to measure cross-modality transfer:

1. DIRECT: The model generates an image directly from the neutral image prompt ($p_{img}$). This tests if the parameter edit implicitly affects visual synthesis.
2. REASONING-AUGMENTED: Before image generation, the model is prompted to produce an intermediate textual rationale ($r$) that explicitly recalls the edited fact. This rationale is then provided as an additional textual constraint alongside the image prompt. This tests if explicitly activating the edited knowledge in the textual context improves visual transfer.

Experimental Setup

The study evaluates three representative UMMs with distinct architectures: Ovis-U1, BLIP3o-4B, and OmniGen2. Three parameter-editing methods are applied: MEMIT, PMET, and AlphaEdit. The evaluation focuses on sequential editing settings, restricting updates to intermediate MLP layers where factual knowledge is typically localized.

Key Results

The Modality Gap

The primary finding is a striking cross-modality knowledge-editing gap.

• Text-Side Success: Parameter editing methods achieve high efficacy on text-only metrics (up to ~92% in some cases).
• Visual Failure: Under the DIRECT protocol, the best overall VQA accuracy for image generation is only 18.5%. In many cases, VQA accuracy retains only one-eighth to one-quarter of the corresponding text-side efficacy.
• Conclusion: Successfully modifying textual outputs does not guarantee that the model will synthesize images reflecting the updated knowledge.

Impact of Reasoning-Augmented Editing

The Reasoning-augmented Parameter Editing protocol significantly narrows this gap but does not eliminate it.

• Performance Gains: By eliciting an explicit reasoning step, overall VQA accuracy improves for all model-editor pairs, with gains reaching up to 18.6 percentage points.
• Mechanism: The reasoning step forces the model to verbalize the edited fact, converting latent parameter changes into explicit textual constraints that guide the visual generator.
• Limitations: The gap persists, indicating that textual recovery and visual realization remain distinct stages. Performance drops further as prompts move from direct references to complex compositions and derived products.

Mechanistic Analysis: The Conditioning Pathway Bottleneck

The authors provide mechanistic evidence that the gap arises from a conditioning-pathway bottleneck.

• Signal Attenuation: Edit-induced perturbations sufficient to alter textual recall are often attenuated or misaligned before reaching the image generator.
• Architectural Differences:
  • Ovis-U1 uses a frozen, dimensionality-reducing linear projection between the LLM and the Diffusion Transformer (DiT). This projection acts as a filter, preserving only ~35% of the edit-induced perturbation energy, significantly weakening the signal for visual generation.
  • BLIP3o-4B and OmniGen2 lack this specific frozen projection, resulting in larger implicit edit drifts at the LLM output, yet they still struggle with visual transfer, suggesting that signal magnitude alone is insufficient; alignment with the conditioning pathway is critical.
• Reasoning Effect: Reasoning augmentation introduces a stronger, qualitatively different conditioning shift. For Ovis-U1, this increases the effective conditioning distance by a factor of 8.56, effectively bypassing the projection bottleneck by leveraging the full text sequence for conditioning.

Contributions

1. UNIKE Benchmark: Introduction of the first benchmark for cross-modality knowledge editing, featuring 2,971 subjects and 5,535 instances with rigorous visual verification.
2. Empirical Evidence of Modality Gap: Systematic demonstration that text-side editing success does not reliably induce visual changes in UMMs, highlighting a fundamental limitation in current cross-modal transfer.
3. Reasoning-Augmented Protocol: Proposal of a method that explicitly activates edited knowledge via intermediate reasoning, significantly improving visual transfer and providing a baseline for future editing strategies.
4. Mechanistic Insights: Identification of the conditioning-pathway bottleneck as the root cause of the modality gap, showing that edit signals can be filtered or misaligned by architectural interfaces (e.g., frozen projections) before reaching the visual generator.

Significance and Claims

The paper claims that textual knowledge edits alone are insufficient for robust cross-modality transfer in Unified Multimodal Models. The findings motivate the development of modality-aware editing methods that directly target the pathways responsible for visual generation, rather than relying solely on textual parameter updates. The work suggests that for UMMs to effectively simulate and interpret the physical world with updated knowledge, editing strategies must account for the distinct conditioning mechanisms required for visual synthesis. The authors emphasize that without such targeted approaches, models may maintain consistent textual knowledge while generating contradictory visual content.