Effects of Cross-lingual Evidence in Multilingual Medical Question Answering

1. Problem Statement

Large Language Models (LLMs) have shown promise in medical reasoning but suffer from hallucinations, lack of factual grounding, and performance degradation in low-resource languages. While Retrieval-Augmented Generation (RAG) is a common strategy to mitigate these issues, most existing research focuses on English. There is a significant gap in understanding:

• How external evidence integration strategies perform across high-resource (English, Spanish, French, Italian) vs. low-resource (Basque, Kazakh) languages.
• Whether the source of evidence (curated medical repositories vs. web search vs. LLM parametric knowledge) impacts performance differently based on language and model scale.
• Whether a "one-size-fits-all" retrieval strategy exists for Multilingual Medical Question Answering (MMQA).

2. Methodology

Dataset and Languages

• Source: The study utilizes the CasiMedicos dataset from the MedExpQA benchmark, consisting of 125 multiple-choice clinical case questions.
• Languages: The dataset was evaluated across six languages:
  • High-resource: English (EN), Spanish (ES), French (FR), Italian (IT).
  • Low-resource: Basque (EU), Kazakh (KK).
  • Note: The low-resource languages were translated using Claude-3.5-Sonnet and manually verified by native speakers, with quality ensured via back-translation metrics (BERTScore, COMET).

Models

The authors evaluated a diverse range of LLMs to analyze the impact of parameter scale:

• Small/Mid-sized (<30B): LLaMA-3.1-8B, Qwen-8B/14B, Gemma-12B/27B, MedGemma-27B.
• Large (≥70B): LLaMA-3.1-70B, Qwen-2.5-72B.

External Evidence Sources

Three distinct types of external knowledge were tested:

1. Curated Medical Repositories: Documents retrieved from PubMed, Wikipedia, StatPearls, and medical textbooks (using MedRAG/BM25+MedCPT).
2. Web Search: Documents retrieved via Google (Serper API) and Cohere search tools.
3. Parametric Knowledge: Answers and explanations generated directly by the LLM without external retrieval.

Retrieval Strategies

The study compared three retrieval configurations:

• Monolingual: Retrieval performed entirely in the target language.
• English-only: Retrieval performed entirely in English (regardless of the question language).
• Cross-lingual: A hybrid approach where 50% of retrieved documents are in English and 50% are in the target language.

Experimental Setup

• Document Count: Experiments determined that retrieving 10 documents yielded optimal performance (diminishing returns observed beyond this point).
• Evaluation: Accuracy was measured against the baseline (no retrieval) across all languages and model sizes. Statistical significance was confirmed via chi-square tests.

3. Key Contributions

1. English-Centric Retrieval Dominance: The paper demonstrates that for high-resource languages, retrieving evidence from English web sources consistently outperforms retrieval in the target language. This is attributed to the English-heavy nature of medical content on the web and the English-biased pre-training of current LLMs.
2. Cross-Lingual Necessity for Low-Resource Languages: For low-resource languages (Basque, Kazakh), monolingual retrieval is insufficient due to data scarcity. The optimal strategy is cross-lingual retrieval (combining English and target language evidence), which significantly closes the performance gap with high-resource languages.
3. Model Scale vs. Retrieval Benefit:
   • Small/Mid-sized Models (<30B): Benefit significantly from external evidence, showing accuracy improvements (up to 8.2% over baseline).
   • Large Models (≥70B): Often experience performance degradation when augmented with external evidence. The authors hypothesize that these models have already internalized sufficient medical knowledge during pre-training, and external retrieval introduces noise or conflicts with their parametric knowledge.
4. Source Quality Analysis: Curated repositories (e.g., PubMed) are authoritative but lack multilingual coverage (often <0.03% of retrieved content for non-English languages). Web search provides broader coverage but introduces more noise.

4. Key Results

• Performance Trends:
  • High-Resource Languages: English web search is the best strategy. However, for models ≥70B, adding retrieval often lowers accuracy compared to the baseline.
  • Low-Resource Languages: Cross-lingual retrieval (English + Target) is the only strategy that brings performance to parity with high-resource languages. Monolingual retrieval fails to provide adequate context.
• The "Knowledge Conflict" Phenomenon: Large models (70B+) sometimes perform worse with retrieval because the retrieved text contradicts or distracts from the high-quality medical knowledge already encoded in their parameters.
• Data Scarcity: In English web searches, only ~20% of documents came from authoritative sources (PubMed/Wikipedia). For other languages, this dropped to <0.03%, highlighting a severe lack of expert-curated medical content in non-English digital storage.
• Optimal Strategy:
  • Small Models: Use Web Search (English for high-resource; Cross-lingual for low-resource).
  • Large Models: Often perform best without retrieval (Baseline), or with very careful, high-quality retrieval to avoid noise.

5. Significance and Implications

• Challenging Assumptions: The paper refutes the common assumption that "more external knowledge always equals better performance." It shows that retrieval strategies must be tailored to model scale and language resource availability.
• Practical Guidelines for MMQA:
  • Do not rely solely on monolingual retrieval for low-resource languages; cross-lingual approaches are essential.
  • For very large models, the cost of retrieval (latency, noise) may outweigh the benefits if the model is already well-trained on medical data.
  • Web search is currently a more effective source for multilingual medical QA than curated repositories due to the latter's lack of multilingual coverage.
• Future Directions: The findings suggest a need for better multilingual curation of medical databases and more sophisticated retrieval mechanisms that can filter noise for large-scale models.

Conclusion: The paper establishes that effective Multilingual Medical QA requires a nuanced approach where retrieval strategies are dynamically selected based on the target language's resource level and the LLM's parameter size, rather than applying a uniform RAG pipeline.