💬 NLP

EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering for Enhanced Alignment and Reasoning

This paper introduces EpiQAL, the first diagnostic benchmark designed to evaluate large language models on evidence-grounded epidemiological reasoning across three progressively challenging subsets, revealing that current models struggle with multi-step inference and that performance is not solely determined by model scale.

Original authors: Mingyang Wei, Dehai Min, Zewen Liu, Yuzhang Xie, Guanchen Wu, Ziyang Zhang, Carl Yang, Max S. Y. Lau, Qi He, Lu Cheng, Wei Jin

Published 2026-03-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingyang Wei, Dehai Min, Zewen Liu, Yuzhang Xie, Guanchen Wu, Ziyang Zhang, Carl Yang, Max S. Y. Lau, Qi He, Lu Cheng, Wei Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive, global mystery: How do diseases spread, and what actually stops them?

For a long time, AI (like the chatbots we use today) has been great at answering questions about individual patients. "What medicine cures a headache?" or "What are the symptoms of flu?" But when it comes to the big picture—looking at millions of people, analyzing thousands of research papers, and figuring out complex patterns like "If we close schools in City A, how does that change the virus spread in City B?"—AI has been struggling. It's like a detective who is great at solving crimes in a single house but gets lost when asked to map out a whole city's criminal network.

The paper you shared introduces EpiQAL, a new "exam" designed specifically to test if AI can become a true Epidemiological Detective.

Here is a simple breakdown of how they built this exam and what they found, using some everyday analogies.

1. The Problem: AI is "Cheating" on Exams

Before this paper, AI benchmarks were like high school tests where the answers were often hidden right next to the question. If the question asked, "What is the capital of France?" and the text said, "Paris is the capital of France," the AI could just find the word "Paris" and guess the answer without actually understanding the concept.

In epidemiology, this is dangerous. If an AI just guesses based on word matching, it might tell a government to use a medicine that works for children on adults, or suggest a strategy that works in winter for a disease that spreads in summer.

2. The Solution: The "EpiQAL" Exam

The researchers created a three-part test (like a video game with three levels) to see if AI can actually think, not just search. They built these questions using real scientific papers about diseases like malaria, dengue, and others.

Level 1: The "Fact Finder" (EpiQAL-A)
- The Analogy: Imagine a librarian. You ask, "What year was this book published?" and the librarian pulls the book off the shelf and reads the date.
- The Test: The AI must find a specific fact (like a cure rate) directly in the text.
- The Twist: The researchers made sure the AI couldn't just match words. They changed the question so the AI had to understand what the number meant, not just find the number.
Level 2: The "Puzzle Solver" (EpiQAL-B)
- The Analogy: Imagine you are a chef. You have a recipe that says "Add salt" and another that says "Add pepper." But the recipe never says "The soup needs seasoning." You have to combine those two facts to realize the soup needs seasoning.
- The Test: The AI has to read different parts of a study and connect the dots. "Study A says the virus spreads by mosquitoes. Study B says the mosquitoes are active at night." The AI must conclude: "The virus is likely a night-time threat."
- The Result: This was the hardest level. Most AIs failed here because they couldn't connect the dots; they just looked for the answer in one sentence.
Level 3: The "Mind Reader" (EpiQAL-C)
- The Analogy: Imagine you are reading a mystery novel, but the author cuts out the last chapter where the detective explains the solution. You have to look at the clues (the evidence) and guess what the detective concluded.
- The Test: The researchers hid the "Conclusion" section of the scientific papers. The AI had to look only at the data and methods and guess what the scientists concluded.
- The Twist: This tests if the AI can distinguish between "What the data proves" and "What the scientists hope is true."

3. How They Made Sure the Exam Was Fair

Building a test for AI is hard because AI is smart enough to find loopholes. The researchers used a "Human + Robot" team to build the exam:

The "Trickster" Robots: They used several different AI models to generate "distractors" (wrong answers). These wrong answers were designed to look very convincing, like a fake ID that looks real but has a tiny error.
The "Referee" Humans: When the robots weren't sure if a question was too easy or too hard, a human expert stepped in to check.
The "Difficulty Filter": They ran the questions through a panel of other AIs. If an AI got the answer right too easily, they rewrote the question to make it harder (like removing the obvious clues).

4. The Big Surprise: Bigger Isn't Always Better

When they ran the test on 14 different AI models (from tiny ones to massive ones), they found some shocking results:

Size Doesn't Matter: The biggest, most expensive AI models didn't always win. In fact, a smaller, cleverly designed model (Mistral-7B) actually beat some of the giant models on the hardest levels. It's like a small, nimble detective solving a case faster than a giant, slow-moving robot.
The "Chain of Thought" Effect: When they told the AI to "think step-by-step" (like writing down its reasoning before answering), it got much better at the "Puzzle Solver" level. But on the other levels, it sometimes made things worse, like overthinking a simple question and getting confused.
The "Over-Confidence" Problem: Many AIs were willing to guess. They would pick the right answer and a wrong answer just to be safe. In real life, this is dangerous. If a public health AI suggests a lockdown and suggests doing nothing, it creates chaos. The best models were the ones that knew when to say, "I'm not sure," rather than guessing.

5. Why This Matters

This paper is a wake-up call. It shows that while AI is amazing at writing poems or summarizing emails, it is not yet ready to be the primary brain behind public health decisions.

If we rely on current AI to tell us how to stop a pandemic, it might miss the subtle connections between different studies or misinterpret the data. EpiQAL gives us a way to measure exactly where AI is failing so we can fix it.

In short: We built a very tough, specialized test to see if AI can be a real epidemiologist. The results show that while AI is getting smarter, it still needs a lot of training before we can trust it to save lives on a global scale. It's a work in progress, but now we have a ruler to measure that progress.

1. Problem Statement

Current Large Language Model (LLM) benchmarks in the medical domain primarily focus on clinical knowledge (e.g., diagnosis, treatment guidelines) or patient-level reasoning. However, epidemiological reasoning is distinct: it requires synthesizing heterogeneous study evidence to infer population-level disease burden, transmission dynamics, and intervention effects.

Existing resources fail to address this gap because they:

Are often disease-specific (e.g., COVID-19 only).
Rely on extractive formats or constrained label spaces (e.g., Yes/No).
Lack rigorous grounding to study evidence, making it difficult to distinguish genuine inference from hallucination.
Allow models to exploit superficial cues (e.g., lexical overlap) rather than demonstrating true reasoning.

The authors argue that there is a critical need for a benchmark that systematically evaluates evidence-grounded epidemiological inference across diverse diseases and reasoning depths.

2. Methodology: The EpiQAL Framework

The authors introduce EpiQAL (Epidemiological QA over the Literature), a diagnostic benchmark constructed through a quality-controlled pipeline. The framework addresses four key challenges: Scope (diverse diseases), Grounding (traceable to evidence), Verification (handling multi-answer correctness), and Difficulty (preventing shortcut learning).

A. Dataset Construction Pipeline

The construction process involves four main stages:

Input Processing & Taxonomy:
- A taxonomy of 6 classes and 25 topics (e.g., Surveillance, Outbreak Investigation, Susceptibility) was developed with epidemiology experts to ensure broad coverage.
- Source documents are drawn from PLOS Neglected Tropical Diseases (approx. 10,600 articles).
QA Generation:
- A generation model ( $M_g$ ) creates questions ( $Q$ ), correct options ( $O_c$ ), and distractors ( $O_d$ ) under specific constraints ( $G$ ).
- External Knowledge: For the inference subset, knowledge graph triples are optionally used only during generation to guide reasoning, but withheld during evaluation.
Multi-Model Verification:
- A group of LLMs independently verifies options for factual consistency and reasoning coherence.
- Options are accepted, rejected, or flagged for human review based on a vote ratio ( $v$ ).
- Crucially, checkers verify against the test-time input (which may be masked) rather than the full document to prevent data leakage.
Difficulty Control (Stem Refinement):
- Applied to inference subsets to prevent "shortcut" learning (e.g., matching entity names).
- Salient entities in the question stem are replaced with descriptive phrases retrieved from the web, forcing models to reason about concepts rather than pattern-match.
- A difficulty score ($DiffScore$) based on model performance determines if refinement is needed.

B. The Three Subsets

EpiQAL consists of three subsets designed to isolate specific capabilities:

EpiQAL-A (Text-Grounded Recall):
- Task: Retrieve facts explicitly stated in the document.
- Format: Multi-answer (set-based).
- Distractors: Document-grounded confounders (same entity, wrong context/metric).
EpiQAL-B (Multi-Step Inference):
- Task: Integrate multiple findings to derive a conclusion not explicitly stated in a single sentence.
- Format: Single-answer.
- Distractors: Reasoning-level flaws (causal reversal, entity misattribution, assumption violations).
EpiQAL-C (Conclusion Reconstruction):
- Task: Infer study conclusions using only the Methods and Results sections (the Discussion section is masked).
- Format: Single-answer.
- Distractors: Plausible but unsupported claims (speculation, external dependency, background restatement).

3. Key Contributions

First Epidemiological QA Benchmark: EpiQAL is the first benchmark to systematically evaluate population-level reasoning over diverse diseases with set-based evaluation.
Expert-Curated Taxonomy: A structured framework covering 25 epidemiological topics to ensure broad subdomain coverage.
Automated Construction Framework: A novel pipeline integrating multi-LLM verification, difficulty screening, and targeted human review to ensure high-quality, hallucination-free data.
Comprehensive Benchmarking: Evaluation of 14 models (open-source and proprietary) reveals distinct failure modes in epidemiological reasoning.

4. Experimental Results

The authors evaluated 14 models (including GPT-4/5, DeepSeek, Llama-3, Mistral, Qwen, GLM) across the three subsets.

Key Findings:

Limited Performance: Even state-of-the-art models struggle with epidemiological reasoning. The best-performing model, DeepSeek-Reasoner, achieved an Exact Match (EM) of 0.928 on recall (A) and 0.868 on inference (B), but performance drops significantly on reconstruction (C).
Multi-Step Inference is the Bottleneck: Performance on EpiQAL-B is the lowest across all models. Without Chain-of-Thought (CoT), many models score below 0.50 EM.
Rankings are Subset-Dependent: No single model dominates all three subsets.
- Example: Mistral-7B outperforms Mistral-Large and Llama-3.3-70B on EpiQAL-B and C, despite having fewer parameters. This suggests that architectural choices and instruction tuning matter more than raw scale for this specific domain.
Scale Does Not Guarantee Success: Larger models do not consistently outperform smaller ones. Some smaller models (e.g., Mistral-7B) show superior reasoning capabilities compared to much larger counterparts.
Chain-of-Thought (CoT) Effects:
- EpiQAL-B: CoT provides significant gains (e.g., Llama-3.1-8B improved from 0.318 to 0.665 EM).
- EpiQAL-C: CoT yields mixed results; for some models (e.g., GPT-5-mini), it actually degrades performance, suggesting that explicit reasoning chains can amplify errors when evidence is incomplete.
Distractor Analysis:
- Semantic Near-Misses are the most deceptive distractors in EpiQAL-A.
- Variable Confusion and Causal/Logical Errors are the primary failure points in EpiQAL-B.
- Speculation/Limitation distractors are most deceptive in EpiQAL-C, indicating models struggle to distinguish untested hypotheses from evidence-based conclusions.

5. Significance and Implications

Public Health AI: EpiQAL highlights that current LLMs are not yet reliable for high-stakes public health decision-making, where false positives (e.g., unnecessary interventions) carry significant costs.
Reasoning vs. Retrieval: The benchmark proves that while LLMs are excellent at retrieving facts, they lack the robustness to synthesize complex, population-level causal inferences from research literature.
Future Directions: The results suggest that simply scaling parameters is insufficient. Future improvements require better alignment with epidemiological principles, improved handling of incomplete information, and specialized training on evidence synthesis.
Resource Release: The authors release the dataset, code, and taxonomy to facilitate future research in evidence-grounded reasoning for public health.

In conclusion, EpiQAL serves as a critical diagnostic tool, revealing that the gap between current LLM capabilities and the rigorous demands of epidemiological inference remains wide, particularly in multi-step reasoning and conclusion reconstruction under uncertainty.