IyàwóBench: A Benchmark for Evaluating Large Language Model Clinical Triage Accuracy on Undifferentiated Febrile Illness in Nigerian Primary Health Settings

Technical Summary: IyàwóBench

Problem Statement
Undifferentiated febrile illness is the leading cause of primary care outpatient visits in Nigeria, presenting a significant diagnostic challenge due to overlapping early symptoms of conditions ranging from uncomplicated malaria to life-threatening bacterial meningitis and sepsis. In Nigerian Primary Health Centres (PHCs), where Community Health Extension Workers (CHEWs) often manage high patient volumes with limited resources, the lack of clinical decision support tools leads to dangerous misclassifications, resulting in both under-referral of critical cases and unnecessary over-referral. While Large Language Models (LLMs) show promise for scalable clinical decision support, existing evaluation benchmarks (e.g., MedQA, PubMedQA) are derived from high-income country contexts or standardized licensing exams. These do not reflect the specific disease prevalence, infrastructure constraints, or regional guidelines (such as WHO IMCI and Nigerian Standard Treatment Guidelines) relevant to West African primary care. Consequently, there is no validated framework to evaluate LLM triage reasoning in this specific setting.

Methodology
The authors introduce IyàwóBench v1.0, a benchmark dataset comprising 200 synthetic clinical vignettes designed to evaluate LLM triage accuracy on undifferentiated febrile illness in Nigerian PHCs.

• Data Generation: The vignettes were generated programmatically using statistical distributions derived from 1,200 real patient encounters collected across 19 PHCs in Oyo State, Nigeria, between April 14 and 28, 2026. No individual patient data was used; the dataset is entirely synthetic.
• Vignette Structure: Each vignette includes structured fields mirroring the Iyawo platform interface (age, vitals, symptoms, malaria RDT results, etc.) and free-text clinical notes.
• Ground Truth: Expected triage levels were assigned based on WHO IMCI 2014, WHO Malaria Guidelines 2025, the Surviving Sepsis Campaign 2021, and Nigerian Standard Treatment Guidelines. The three target classes are: REFER NOW (emergency), REFER TODAY (non-emergency), and TREAT HERE (treat locally).
• Models Evaluated: Six LLMs were tested: Claude Sonnet (claude-sonnet-4-5), Llama 4 Scout (17B), Llama 3.3 (70B), Llama 3.1 (8B), Qwen 3 (32B), and GPT OSS (20B).
• Evaluation Protocol: Models received identical prompts requiring a specific structured output format (triage token, diagnosis, reasoning). Performance was measured via two metrics:
  1. Triage Accuracy: The proportion of vignettes where the predicted triage level exactly matched the expected level.
  2. Safety Score: The proportion of REFER NOW cases where the model did not incorrectly downgrade the patient to TREAT HERE.
• Statistical Analysis: Accuracy differences were assessed using the McNemar test for paired proportions.

Key Results

• Safety: All six models achieved a 100% safety score (95% CI: 96.4–100.0%). No model downgraded a critical REFER NOW case to TREAT HERE, regardless of model size or architecture.
• Triage Accuracy: Significant variance was observed among models that successfully parsed the structured output:
  • Claude Sonnet: 67.5% accuracy (135/200).
  • Llama 4 Scout: 59.5% accuracy (119/200).
  • Llama 3.3 70B: 43.0% accuracy (86/200).
  • Llama 3.1 8B: 39.0% accuracy (78/200).
  • Qwen 3 32B & GPT OSS 20B: Near-zero accuracy (1.0% and 0.5%, respectively) due to structured output non-compliance. These models generated detailed clinical reasoning but failed to include the required triage tokens in a parseable format.
• Performance Gaps: The difference between the highest-performing parseable model (Claude Sonnet) and the lowest (Llama 3.1 8B) was 28.5 percentage points. Within the Llama family, accuracy generally correlated with parameter count, though the 70B model underperformed the specialized Claude model by 24.5 points.

Significance and Claims
The paper positions IyàwóBench as the first reproducible evaluation framework for LLM clinical decision support specifically tailored to West African primary care. The authors claim three primary contributions:

1. Validation of Safety: The universal 100% safety score suggests that modern LLMs, even without specific Low- and Middle-Income Country (LMIC) fine-tuning, robustly encode clinical severity signals sufficient to avoid the most dangerous triage error (missing an emergency).
2. Accuracy Variance: The substantial gap in accuracy (up to 28.5 points) between models highlights that parameter count alone does not guarantee clinical performance. The authors suggest that clinically engineered systems with embedded guidelines (like the one used for the ground truth) outperform general-purpose models, implying that domain-specific fine-tuning (e.g., via LoRA) is necessary for deployment in resource-constrained settings.
3. Structured Output as a Bottleneck: The near-zero accuracy of two large models due to formatting failures underscores that for clinical decision support, the ability to generate structured, machine-readable outputs is as critical as clinical reasoning capability.

The authors explicitly note limitations, including the use of a single author for ground-truth labeling without independent physician validation, the focus on triage classification rather than diagnostic or treatment accuracy, and the use of synthetic data derived from a single state (Oyo), which may not generalize to all Nigerian contexts. They conclude that IyàwóBench establishes a baseline for rigorously assessing fine-tuned and domain-adapted models in LMIC primary care.