Technical Summary: PersianMedQA

Problem Statement

While Large Language Models (LLMs) have achieved human-level performance on various Natural Language Processing benchmarks, their reliability in high-stakes domains like medicine remains underexplored, particularly for low-resource languages. Existing medical QA benchmarks are predominantly English-centric, creating a significant evaluation gap for non-English contexts. Furthermore, simple translation of medical questions is insufficient; it often strips away critical cultural cues, localized clinical protocols, and region-specific terminology, potentially leading to dangerous misinterpretations in clinical practice. There is a lack of expert-validated, bilingual datasets that can rigorously evaluate LLMs on Persian medical reasoning while accounting for these contextual nuances.

Methodology

Dataset Construction: PersianMedQA

The authors introduced PersianMedQA, a large-scale dataset comprising 20,785 expert-validated multiple-choice questions derived from 14 years of Iranian national medical residency and pre-residency examinations (administered by the Medical Education Assessment Center, Sanjesh).

• Scope: The dataset covers 23 medical specialties, with approximately 70% classified as clinical (patient cases) and 30% as non-clinical (basic sciences).
• Preprocessing: A rigorous cleaning pipeline was employed, including duplicate removal via LaBSE embeddings, exclusion of image-dependent questions to ensure text-only evaluation, and a three-level verification of answer keys by the National Center for Medical Education Assessment.
• Annotation: Questions were annotated for subject matter (specialty), domain type (clinical vs. non-clinical), and patient demographics (age, gender) using a combination of expert review and LLM-assisted extraction.
• Bilingual Translation: To enable cross-lingual evaluation, the dataset was translated into English using Google Translate, GPT-4.1, and Gemini-2.5-Flash. Expert validation confirmed that Gemini-2.5-Flash provided the most accurate preservation of medical terminology and clinical meaning.
• Integrity Checks: The authors implemented safeguards against data contamination, including exact search verification on public web sources and temporal analysis of exam years (2011–2024), confirming that the dataset was not present in the training corpora of most evaluated models.

Experimental Setup

The study benchmarked 41 state-of-the-art models across three categories:

1. General-purpose LLMs (both closed-weight and open-weight).
2. Persian-specific LLMs (models adapted for the Persian language).
3. Medical-specific LLMs (models fine-tuned on biomedical corpora).

Evaluation Protocols:

• Zero-shot: Models were evaluated on both the original Persian questions and the English translations using standardized prompts.
• Chain-of-Thought (CoT): Selected models were evaluated with reasoning steps to assess the impact of explicit reasoning.
• Answer-Only Evaluation: Models were tested on answer choices without question stems to determine if they relied on statistical patterns rather than genuine reasoning.
• Ensembling: Majority-vote ensembles of diverse model families were tested to see if combining models improved accuracy.

Key Contributions

1. PersianMedQA Dataset: The first large-scale, expert-validated bilingual (Persian-English) medical QA benchmark derived from national licensing exams, accompanied by a bilingual medical dictionary.
2. Comprehensive Benchmarking: An evaluation of 41 models, revealing significant performance disparities between closed-weight, open-weight, Persian-adapted, and medical-adapted models.
3. Translation Impact Analysis: A quantitative demonstration that while English translations generally yield higher scores, 3–10% of questions can only be answered correctly in the original Persian due to lost cultural and clinical context (e.g., regional vaccination schedules and drug availability).
4. Domain Adaptation Insights: Evidence that model size alone is insufficient for robust medical performance without strong domain or language adaptation, and that current Persian LLMs struggle significantly with medical reasoning.

Key Results

Performance Hierarchy

• Closed-Weight Dominance: Closed-weight models, particularly GPT-4.1, achieved the highest accuracy, scoring 83.09% in Persian and 80.71% in English.
• Open-Weight Performance: The best open-weight model, DeepSeek-Chat-V3, achieved 68.05% in Persian and 73.30% in English.
• Persian LLMs: Models specifically adapted for Persian (e.g., Dorna, PersianMind) performed poorly, with Dorna scoring only 34.87% in Persian. These models often struggled with instruction following and domain reasoning.
• Medical LLMs: Specialized medical models (e.g., Meditron3) showed only modest improvements over general models and failed to generalize effectively to Persian clinical data (e.g., Meditron3 scored 38.67% in Persian).

Linguistic and Contextual Findings

• Translation Loss: While most models performed better on English translations, a subset of questions (3–10%) was answered correctly only in Persian. Manual analysis attributed this to the loss of regional protocols (e.g., specific tetanus prophylaxis guidelines in Iran vs. Western standards) and semantic drift in medical terminology.
• Subfield Variance: Models performed best on pharmacology and non-clinical questions (factual recall) but struggled significantly in surgery and medical statistics, which require complex reasoning and understanding of language-specific guidelines.
• Reasoning vs. Pattern Matching: In an "answer-only" evaluation, models achieved 35.60% accuracy (vs. 25% random guessing), suggesting they exploit statistical artifacts in answer choices, particularly in medical ethics questions, rather than engaging in genuine ethical reasoning.

Prompting and Ensembling

• Chain-of-Thought: CoT prompting improved accuracy by approximately 2% for large general-purpose models on clinical questions but offered negligible gains for smaller models.
• Ensembling: A majority-vote ensemble of five diverse open-weight models achieved 73.7% accuracy, outperforming the best individual open-weight model in that group by +3.3%.

Significance and Claims

The paper claims that PersianMedQA establishes a foundational benchmark for evaluating bilingual and culturally grounded medical reasoning in LLMs. The authors assert that:

1. Translation is Insufficient: Direct translation of medical guidelines is inefficient and potentially dangerous due to regional variations in healthcare systems, drug availability, and cultural contexts. Native-language evaluation is essential for low-resource languages.
2. Current Gaps: There is a substantial "language gap" where Persian LLMs and medical-specific models currently underperform compared to general-purpose closed-weight models, indicating a need for high-quality, domain-relevant training data in Persian.
3. Future Directions: The work highlights the necessity for developing Persian medical corpora, retrieval-augmented systems that access culturally appropriate guidelines, and further evaluation in other low-resource languages.

The authors emphasize that the tested models are not certified for clinical use and advocate for continued expert-in-the-loop development to ensure responsible deployment in healthcare settings.