LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs

1. Problem Statement

Evaluating Cross-Lingual Knowledge Transfer (CLKT) in Large Language Models (LLMs) is currently hindered by two major issues:

• Data Leakage and Contamination: Standard benchmarks often fail to distinguish between genuine knowledge transfer and mere memorization. Since LLMs are pretrained on massive multilingual corpora, a model might answer a question correctly in a target language simply because it saw the fact in that language during pretraining, not because it transferred knowledge from a source language.
• Static and Synthetic Limitations: Existing benchmarks often rely on static data (which becomes outdated as models improve) or synthetic/fictitious facts. Synthetic facts can conflict with a model's existing world knowledge, leading to performance degradation that is misinterpreted as a lack of transferability.

The core challenge is to create a benchmark that isolates genuine transfer: measuring whether a model can learn a fact in Language A and successfully reproduce it in Language B, without prior exposure to that fact in Language B.

2. Methodology: The LiveCLKTBench Pipeline

The authors propose LiveCLKTBench, an automated pipeline designed to generate contamination-free, real-world benchmarks. The process consists of four distinct stages:

A. Knowledge Entity Collection & Filtering

• Domains: The system targets rapidly updating, real-world domains: Movies, Music, Sports, and Natural Disasters.
• Temporal Filtering: To ensure the knowledge is unseen, the pipeline selects entities that appear at least six months after the model's pretraining knowledge cutoff. This avoids exposure via trailers, previews, or leaks.
• Leakage Verification: Before inclusion, the target model is prompted to summarize the entity. If the model's summary matches the real-world source document, the entity is discarded as "known." Only entities the model cannot recognize are retained.

B. QA Generation

• Grounding: For each valid entity, a factual multiple-choice question is generated based strictly on the source document (e.g., IMDB, SportsDB).
• Constraints: Questions must target concrete facts (e.g., final scores, release dates) that only become known after the event occurs. Distractors are generated to be plausible but incorrect.
• Verification: A verifier LLM checks that questions adhere to strict criteria (grounded in the document, unambiguous, four options). Human inspection confirms a 95% precision rate.

C. Translation

• Verified questions and their source documents are translated into target evaluation languages (e.g., English, Japanese, Mandarin, French, Spanish) using state-of-the-art LLMs.

D. Evaluation Protocol

Unlike zero-shot benchmarks, LiveCLKTBench uses a Post-Training (Knowledge Injection) + Testing protocol:

1. Injection: The model is post-trained (e.g., via Continual Pre-training or LoRA) on the source-language documents containing the new knowledge.
2. Transfer Test: The model is tested on the target-language questions.
3. Metric: Since the model has never seen the target-language version of the fact, a correct answer proves successful cross-lingual transfer.

Key Metrics:

• Overall Success: The proportion of questions answered correctly in both source and target languages.
• Transfer Success: The conditional probability of answering correctly in the target language given a correct answer in the source language ($A / (A+B)$).

3. Key Contributions

1. Leakage-Free Benchmark: LiveCLKTBench is the first automated pipeline specifically designed to eliminate data leakage by using time-sensitive, real-world entities that are verified as unseen by the model.
2. Real-World Grounding: By using actual events (sports scores, movie releases) rather than synthetic facts, the benchmark reflects practical scenarios where models must integrate new knowledge with existing world knowledge.
3. Scalability and Automation: The pipeline is fully configurable, allowing researchers to easily update benchmarks for new model releases and different language pairs with minimal manual intervention.
4. Comprehensive Evaluation: It supports both Model Comparison (which model transfers better?) and Strategy Comparison (which fine-tuning method works best?).

4. Experimental Results

The authors evaluated several open-source multilingual LLMs (Gemma, Qwen, Mistral, Llama, OLMo) across five languages and three domains.

• Overall Performance: There is a significant performance gap between models. Gemma-2-9B showed the highest overall success, while Qwen2.5-7B demonstrated the best cross-lingual transfer capability. Performance varied by domain, with Sports being the most difficult and Music the easiest.
• Linguistic Distance & Asymmetry:
  • Transferability is strongly influenced by linguistic distance. Models perform significantly worse when transferring between typologically distant languages (e.g., Indo-European to East Asian languages like Japanese and Mandarin).
  • Directional Asymmetry: Transfer is often unbalanced. For example, transferring from English to Japanese yields lower scores than Japanese to English.
• Scaling Laws:
  • Larger models generally exhibit better transfer capabilities.
  • However, diminishing returns are observed; as model size increases, the marginal gain in transferability decreases.
  • Scaling helps general alignment but does not fully solve domain-specific transfer challenges.

5. Significance and Conclusion

LiveCLKTBench addresses a critical gap in the evaluation of multilingual LLMs by providing a reliable, dynamic, and contamination-free testbed.

• Scientific Insight: It reveals that current models struggle with cross-lingual transfer, particularly across distant language families, and that this is not solely a function of model size.
• Future Research: The benchmark serves as a standard for evaluating new post-training strategies (like knowledge editing or continual pretraining) and highlights the need for domain-targeted multilingual adaptation rather than just scaling parameters.
• Sustainability: By automating the generation of "fresh" knowledge, the benchmark remains relevant as models evolve, preventing the "saturation" seen in static datasets.

In summary, the paper argues that to truly understand and improve multilingual LLMs, the community must move beyond static, potentially contaminated benchmarks to dynamic, real-world evaluations like LiveCLKTBench.