Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks

Technical Summary: Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks

Problem Statement
While Large Language Models (LLMs) have demonstrated remarkable capabilities in English-centric medical applications—such as clinical decision support and medical question answering—their reliability in linguistically diverse communities remains unproven. Previous evaluations have noted substantial performance drops for medical LLMs in low-resource languages like Arabic, yet the underlying causes of these discrepancies are poorly understood. Existing literature often attributes failures broadly to limited pretraining data or domain mismatches without systematically disentangling whether the degradation stems from intrinsic linguistic properties (e.g., morphology, tokenization), insufficient medical domain adaptation, architectural choices, or interactions between these factors. This lack of granular understanding hinders the design of effective multilingual training and evaluation strategies.

Methodology
The authors present a systematic, cross-lingual empirical study designed to isolate the specific contributors to performance gaps between Arabic and English medical tasks. The study utilizes MedAraBench, an Arabic medical multiple-choice question (MCQ) dataset containing 19 medical specialties, alongside its English-translated counterpart. By evaluating models on both the original Arabic questions and their English translations, the authors control for medical content to isolate the effect of linguistic representation.

The evaluation framework addresses five specific research questions (RQs) using a diverse set of open-source models, including general-purpose LLMs (e.g., DeepSeek-V3.2, LLaMA 3.3 70B) and medical-domain adapted models (e.g., Meditron 3, Med42, MedGemma). The methodology involves:

1. Cross-lingual Accuracy Comparison: Measuring performance differences between Arabic and English inputs under identical prompting structures.
2. Granular Analysis: Investigating how input length, educational difficulty (years of study), and medical specialty affect performance.
3. Alignment and Output Format: Comparing "soft matching" (letter selection) against "hard matching" (free-form text generation) to assess instruction following and surface-form generation.
4. Tokenization Analysis: Examining tokenization efficiency, fragmentation rates (tokens per word, characters per token), and the prevalence of single-character tokens.
5. Reliability Assessment: Analyzing the correlation between model-reported confidence/explanations and actual correctness.

Key Contributions
The paper makes three primary contributions:

• Diagnostic Framework: The design of a cross-lingual diagnostic evaluation framework that enables controlled analysis across languages, output formats, tokenization behaviors, and reliability signals.
• Isolated Empirical Study: The first systematic study on MedAraBench and its English counterpart to isolate language effects while controlling for medical domain content.
• Disentangled Findings: The identification that Arabic performance degradation is driven by a complex interplay of representational, alignment, and evaluation factors rather than a lack of medical knowledge alone.

Results
The empirical analysis yields several critical findings:

• Persistent Performance Gap: Across nearly all evaluated models, accuracy in Arabic is significantly lower than in English. This gap intensifies with increased task complexity (e.g., later years of medical study) and free-form generation. DeepSeek-V3.2 is the only model showing comparable performance across both languages; notably, larger model size does not guarantee reduced cross-lingual degradation.
• Task Complexity and Length: Performance degradation in Arabic is disproportionately affected by input length and difficulty. While English performance remains relatively stable with longer inputs, Arabic accuracy degrades significantly as question length increases.
• Output Format Sensitivity: The performance gap widens when models are required to generate free-form text answers compared to selecting letter-based options. Arabic outputs exhibit lower surface-form similarity to ground truth, suggesting that instruction following and generation tasks amplify language-specific difficulties.
• Tokenization Fragmentation: Tokenization analysis reveals structural fragmentation in Arabic medical text. Arabic inputs consistently require more tokens per word and exhibit higher rates of single-character tokens compared to English. While this fragmentation correlates with higher token usage and potential performance drops, the effect is not uniform; some models (like DeepSeek-V3.2) maintain stability despite fragmentation, indicating tokenization is a contributing but not sole factor.
• Unreliable Confidence: Model-reported confidence and generated explanations show limited correlation with correctness. In fact, a moderate negative correlation was observed, where higher confidence predictions were often less accurate. This miscalibration suggests that confidence scores are not reliable proxies for correctness in medical settings.

Significance
The paper argues that the observed performance gaps in Arabic medical LLMs are not merely a result of insufficient medical knowledge or data scarcity. Instead, the degradation is driven by interacting factors including linguistic representation issues (tokenization fragmentation), alignment constraints, and evaluation protocols. The study underscores the necessity for language-aware design and evaluation strategies. The authors conclude that without understanding which specific factors dominate model failure, it is difficult to design effective multilingual training strategies or alignment procedures. This work provides a foundational diagnostic approach to move beyond aggregate performance scores and address the specific mechanisms underlying LLM failures in underrepresented languages.