PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams -- Dataset Construction and Evaluation

Technical Summary: PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams

Problem Statement
While Large Language Models (LLMs) have demonstrated remarkable proficiency in answering medical examinations, particularly those based on English-language datasets from high-income countries, their performance in Spanish and within the specific epidemiological context of Latin America remains largely unexplored. Countries like Peru present a unique challenge, combining noncommunicable chronic diseases with infectious and tropical diseases, a profile distinct from the data sources used to train most existing medical LLMs. There is a critical need to identify effective LLMs capable of accurately answering medical questions in Spanish from South American contexts to support the growing adoption of AI-based medical applications in the region.

Methodology
The study followed a three-pronged approach: dataset construction, zero-shot evaluation, and parameter-efficient fine-tuning.

1. Dataset Construction (PeruMedQA): The authors curated a multiple-choice question-answering (MCQA) dataset named PeruMedQA, comprising 8,380 questions from medical examinations administered by the Consejo Nacional de Residentado Médico (CONAREME) between 2018 and 2025. The dataset covers 12 specialties and subspecialties. Questions were extracted from PDFs, manually verified for answer accuracy (correcting 16 errors out of 8,380), and standardized to ensure uniformity in the number of options (adding a "None of the above" option where necessary to standardize from four to five choices). The 2025 exam questions were reserved as a test set.
2. Zero-Shot Evaluation: Ten medical LLMs ranging from 3.8 billion to 70 billion parameters were evaluated using zero-shot, task-specific prompts in Spanish. The models included medgemma-4b-it, medgemma-27b-text-it, OctoMed-7B, Llama3-OpenBioLLM-70B, and others. The models were instructed to select a single letter option (A–E) corresponding to the correct answer. Performance was measured by accuracy (correct answers divided by valid answers) and the rate of "hallucinations" (instances where models failed to follow formatting instructions or provided invalid options).
3. Fine-Tuning: To address resource constraints, the authors applied Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA) to the medgemma-4b-it model. The model was trained on 90% of the non-2025 data (6,282 questions) with a 10% validation split, using a learning rate of 5e-5 over 10 epochs. The fine-tuned model was then evaluated against the held-out 2025 test set and compared against the vanilla models.

Key Contributions

• PeruMedQA Dataset: The paper introduces the first harmonized, computer-readable dataset of Peruvian medical residency and subspecialty exam questions. This resource is released as open access to facilitate further research and AI development in Spanish-speaking medical contexts.
• Comprehensive Benchmarking: The study provides a systematic evaluation of ten diverse medical LLMs across various specialties, years, and parameter sizes, specifically within a Latin American epidemiological context.
• Fine-Tuning Strategy: The authors demonstrate a resource-efficient approach to enhancing model performance by fine-tuning a small (4B parameter) model, making high-accuracy medical AI more accessible to regions with limited computational infrastructure.

Results

• Model Performance: medgemma-27b-text-it achieved the highest overall accuracy, reaching 89.29% in Psychiatry. OctoMed-7B showed slight superiority in Neurosurgery and Radiology compared to other models. Conversely, most LLMs with fewer than 10 billion parameters exhibited accuracy below 50% in their vanilla state.
• Hallucinations: Significant variability was observed in instruction adherence. meditron-7b had the highest hallucination rate (66.37%), while Llama3-OpenBioLLM-70B and the fine-tuned medgemma-4b-it had 0.00% invalid answers.
• Fine-Tuning Impact: The fine-tuned version of medgemma-4b-it (medgemma-4b-it-FT) significantly outperformed its vanilla counterpart and all other models with fewer than 10 billion parameters. It rivaled the performance of the 70B parameter model (Llama3-OpenBioLLM-70B) across various examinations and uniquely answered 0.72% (60 questions) of the total dataset that no other model answered correctly.
• Limitations of Current Models: Despite high performance in some areas, 2.27% of all questions (190 questions) were answered incorrectly by every LLM evaluated. This failure rate increased in recent years (21.05% in 2025), particularly in surgical fields like Ophthalmology and General Surgery.

Significance and Claims
The paper claims that while medical LLMs not primarily trained on Spanish or Peruvian epidemiological data can successfully answer advanced medical questions, their performance is highly variable and dependent on model architecture and size. The study concludes that for medical AI applications and research requiring knowledge bases from Spanish-speaking countries with epidemiological profiles similar to Peru's, medgemma-27b-text-it is the most effective current option. However, a fine-tuned version of a smaller model (medgemma-4b-it-FT) or OctoMed-7B offers a viable, resource-efficient alternative that rivals much larger models.

The authors emphasize that deploying these models requires careful evaluation of specific tasks and real-world scenarios. They caution that while fine-tuning improves domain-specific accuracy, it necessitates monitoring for "catastrophic forgetting" of general knowledge. The study positions PeruMedQA as a foundational benchmark for improving LLM capabilities in diverse medical populations, particularly in the Global South.