HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning

1. Problem Statement

The field of biomedical reasoning requires high-quality datasets that capture the linguistic and conceptual complexity of professional medical exams. The original HEAD-QA (v1) dataset, released in 2019, contained 6,765 questions from Spanish medical specialization exams (2013–2017). While influential, v1 faced limitations:

• Temporal Obsolescence: It lacked data from the last decade of medical advancements.
• Scale: The dataset size was insufficient to robustly evaluate modern Large Language Models (LLMs), particularly regarding generalization and knowledge retention.
• Language Gap: The original English translation relied on older machine translation methods, and multilingual variants were limited.
• Evaluation Context: The rise of advanced LLMs (e.g., Llama 3, Mistral) and techniques like Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) necessitated a re-evaluation of how these models perform on specialized medical reasoning tasks compared to simple information retrieval.

2. Methodology

Dataset Construction (HEAD-QA v2)

The authors constructed an expanded dataset based on official exams from the Spanish Ministry of Health (Ministerio de Sanidad de España).

• Scale and Scope: The dataset now contains 12,751 multiple-choice questions, more than doubling the original size. It covers 10 years of exams (2013–2022) across six disciplines: Medicine (MIR), Nursing (EIR), Biology (BIR), Chemistry (QIR), Psychology (PIR), and Pharmacy (FIR).
• Preprocessing Pipeline:
  • Conversion: PDFs were converted to text, preserving two-column layouts.
  • Image Handling: Questions linked to images were identified (334 total), though visual processing was excluded from this specific study.
  • Standardization: Chemical formulas were converted to SMILES notation to facilitate text-based ML processing. Disputed or withdrawn questions were filtered out.
  • Format: Data is stored in Parquet format with a JSON structure containing fields for question ID, text, answer options, correct answer ID, year, and category.
• Multilingual Expansion:
  • Translation Strategy: Unlike v1 (which used Google's seq2seq), v2 uses LLMs (LLaMA-3.1-8B) for translation. Three prompting configurations were tested: Zero-shot, One-shot, and Instruction-tuned (acting as an "expert translator").
  • Selection: The most reliable translation was selected based on error rates and consistency checks.
  • Variants: The dataset includes versions in English, Italian, Galician, and Russian. Quality was assessed via back-translation (Round-Trip Translation) using BLEU and BERTScore, confirming high semantic fidelity, especially for languages closer to Spanish.

Experimental Setup & Inference Strategies

The authors benchmarked four open-source, instruction-tuned LLMs:

• Models: Llama-3.1 (8B, 70B), Mistral-7B, and Mixtral-8x7B (MoE).
• Inference Strategies:
  1. Prompting: Zero-shot, Few-shot (with USMLE-style examples), and Chain-of-Thought (CoT).
  2. Retrieval-Augmented Generation (RAG): Used a corpus of 18 medical textbooks (126k fragments) and the MedCPT retrieval system to fetch relevant context before answering.
  3. Probability-Based Selection: Instead of generating text, the model calculates the log-probability of each answer option sequence and selects the one with the highest likelihood.

Evaluation Metrics

• Accuracy: Proportion of correct answers.
• Normalized Exam Score: Adjusted for the official Spanish penalty scheme (3 wrong answers cancel 1 correct).
• Unanswered Ratio: Fraction of questions where the model failed to provide a valid response.

3. Key Contributions

1. HEAD-QA v2 Dataset: A significantly larger, temporally updated, and multilingual benchmark (12,751 questions) designed to test the limits of modern LLMs in specialized biomedical reasoning.
2. Comprehensive Baselines: Systematic evaluation of state-of-the-art open-source models across different sizes (8B to 70B) and architectures (Dense vs. MoE).
3. Strategy Analysis: A rigorous comparison of prompting, RAG, and probability-based selection, challenging the assumption that complex inference strategies always improve performance in specialized domains.
4. Multilingual Resources: High-quality machine-translated versions in four additional languages, enabling cross-lingual research.

4. Results

Model Performance

• Scale is King: Model size is the primary driver of performance. Llama-3.1-70B consistently achieved the highest accuracy (approx. 83% in English, 83% in Spanish) and normalized exam scores. Smaller models (e.g., Mistral-7B) performed significantly worse.
• Language Effects: Performance was generally higher in English than in Spanish, particularly for smaller models. However, the gap narrowed significantly for the 70B model, suggesting larger models have better multilingual generalization.
• Prompting Strategies:
  • Zero-shot vs. Few-shot: Results were comparable, suggesting that instruction-tuned models do not benefit significantly from few-shot examples in this domain.
  • Chain-of-Thought (CoT): Contrary to expectations, CoT prompting reduced accuracy and increased non-response rates for most models. Explicit reasoning steps appeared to introduce noise or distract the model from its internal knowledge.
• RAG Performance: Retrieval-Augmented Generation did not yield consistent improvements over standard prompting. In many cases, performance slightly degraded. The correlation between retrieval relevance and answer correctness was very low ($r=0.07$), indicating models rely more on internal parametric knowledge than external retrieved text for these specific questions.
• Probability-Based Selection: This method performed the worst across all metrics. Evaluating options independently (without joint reasoning) failed to capture the elimination logic required for multiple-choice questions.

Trends

• Performance remained stable across exam years (2013–2022), with larger models outperforming smaller ones consistently.
• The "positional bias" (preference for middle options) observed in previous studies was noted but did not significantly impact the evaluation as models were not trained on answer positions.

5. Significance and Conclusion

The paper establishes that for highly specialized biomedical question answering:

• Intrinsic Capacity > Inference Complexity: The reasoning ability and knowledge base of the model itself are far more critical than the sophistication of the inference strategy (e.g., RAG or CoT).
• Limitations of External Tools: In domains where the model's internal knowledge is dense and accurate, adding external retrieval (RAG) can introduce noise without adding value.
• Scaling Laws Hold: Increasing model parameters remains the most effective path to improving performance on complex medical reasoning tasks.

Limitations: The study did not include proprietary frontier models (e.g., GPT-4, Claude) due to API costs. Additionally, while translations were validated, large-scale human validation for all languages was not feasible.

Impact: HEAD-QA v2 serves as a robust, reproducible resource for the research community to study model generalization, temporal knowledge retention, and the efficacy of different reasoning strategies in the critical domain of healthcare. The dataset is publicly available on Hugging Face.