CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs

Technical Summary: CrossCult-KIBench and MCKI

Problem Statement
Multimodal Large Language Models (MLLMs), predominantly trained on English-centric data, frequently generate culturally misaligned or inappropriate responses when deployed in non-English cultural contexts. While existing adaptation methods like culture-aware retraining or fine-tuning can mitigate broad misalignment, they require substantial culture-specific data and often fail when such evidence is scarce. Conversely, standard knowledge editing techniques focus on correcting factual errors or updating entity-centric associations but lack mechanisms to insert missing culture-conditioned concepts without altering the model's behavior in other cultural settings. This gap necessitates a precise mechanism for cross-cultural knowledge insertion: the ability to supplement MLLMs with specific cultural knowledge for targeted contexts while preserving their original behavior in non-target cultures.

Methodology
To address this, the authors introduce CrossCult-KIBench, a comprehensive benchmark designed to evaluate both the effectiveness of knowledge insertion and its unintended side effects on non-target cultures.

• Benchmark Construction: CrossCult-KIBench covers 49 culturally relevant visual scenarios across three language-culture partitions: English (U.S. context), Chinese (China context), and Arabic (Arab region context). The scenarios are derived from the World Values Survey and categorized into three topic groups: Social Values/Attitudes & Stereotypes, Religious Values, and Ethical Values/Norms. The dataset comprises 9,800 image-grounded cases, where each case pairs a visual input with partition-specific questions and reference answers. Crucially, answers across partitions are not translations but distinct cultural judgments for the same visual situation.
• Evaluation Settings: The benchmark supports two evaluation modes:
  1. Single-Insert: Inserting knowledge for one specific partition (Chinese or Arabic) into a base model.
  2. Sequential-Insert: Sequentially inserting knowledge for all three partitions (English, Chinese, Arabic) into the same model instance to test stability and coexistence.
• Metrics: Performance is measured using ROUGE-L for textual overlap and an LLM-as-Judge (gpt-5.4-mini) for semantic consistency. Key metrics include Reliability (success of target insertion), Generality (transfer to similar scenarios), and Locality (preservation of behavior in non-target languages and unrelated scenarios).
• Proposed Baseline (MCKI): The authors propose Memory-Conditioned Knowledge Insertion (MCKI) as a baseline method. MCKI stores insertion samples in an external memory. For a new query, it uses the frozen MLLM's own representations to generate a router key. A lightweight router matches the query against the memory; if a match exceeds a calibrated threshold, the corresponding cultural knowledge is prepended as a conditional prompt. Otherwise, the model decodes normally using its base state. Only the router is trained, keeping the base MLLM frozen.

Key Contributions

1. Task Formalization: The paper formalizes the novel task of cross-cultural knowledge insertion, distinguishing it from general cultural adaptation or factual knowledge editing.
2. CrossCult-KIBench: The introduction of a rigorous, image-grounded benchmark spanning 49 scenarios and 9,800 multilingual cases, specifically designed to measure insertion success alongside behavioral preservation (locality).
3. MCKI Baseline: The proposal of MCKI, a memory-based approach that demonstrates the feasibility of targeted cultural adaptation without full retraining, establishing a performance benchmark for future work.

Experimental Results
Extensive experiments on InternVL3.5-8B and Qwen3.5-9B reveal significant challenges in balancing cultural adaptation with behavioral preservation:

• Fine-tuning: Achieves high Reliability but severely degrades Cross-Language and Cross-Scenario Locality, indicating that parameter updates overwrite existing cultural behaviors.
• In-Context Learning (IKE): Preserves locality well but shows limited Reliability, as its effectiveness depends heavily on the base model's ability to utilize in-context demonstrations.
• Retrieval-Based Methods (SERAC, MSCKE): Show mixed results; while they improve Reliability, they often suffer from "retrieval confusion," where unrelated queries activate replacement paths, eroding Locality.
• MCKI Performance: MCKI demonstrates a superior balance across metrics. It achieves competitive Reliability and Generality while maintaining high Locality scores, particularly in the Single-Insert setting. In Sequential-Insert tasks, MCKI shows stronger retention of earlier insertions compared to parameter-update methods, though it still faces challenges in transferring inserted knowledge to related examples (Generality) after multiple sequential updates.

Significance and Claims
The paper claims that CrossCult-KIBIBench and MCKI establish a foundational framework for developing more culturally adaptive and responsible MLLMs. The authors emphasize that current approaches struggle to simultaneously achieve effective cultural adaptation and behavioral preservation, highlighting a critical limitation in existing knowledge editing and adaptation strategies. By providing a structured evaluation setting, the work underscores the need for precise mechanisms that can supplement models with missing culture-conditioned knowledge without compromising their performance in other contexts. The authors modestly note that their benchmark uses language-culture partitions as practical proxies for cultural settings and does not claim to capture the full internal diversity of cultures, but rather serves as an operational testbed for insertion mechanisms.