💻 computer science

IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care

This paper introduces IyawoBench v2.0, a rigorous diagnostic framework and benchmark using Nigerian primary care data to reveal that conventional safety metrics mask critical failure modes in large language models, necessitating scenario-specific model selection for clinical triage in low-resource settings.

Original authors: Anthonio Oladimeji Gabriel, Dimeji Olawuyi

Published 2026-08-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anthonio Oladimeji Gabriel, Dimeji Olawuyi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive ship, but instead of steering through an ocean, you are navigating a stormy sea of medical emergencies. In many parts of the world, there aren't enough trained doctors to check every single patient who walks through the door. So, scientists have started building "digital first responders"—super-smart computer programs called Large Language Models. Think of these models as incredibly well-read librarians who have read every medical textbook ever written. They can look at a patient's symptoms and guess what's wrong, or more importantly, decide how urgent the problem is.

But here is the tricky part: how do you know if a digital librarian is actually safe to let loose in a real hospital? In the past, we checked their safety with simple "pass or fail" tests. It was like asking, "Did the librarian tell the patient to go to the hospital if they were bleeding?" If the answer was "yes," we gave them a gold star and said, "Safe!" But this paper argues that a gold star isn't enough. Just because a model doesn't send a bleeding patient home doesn't mean it's perfect. It might be sending them to the wrong kind of hospital, or it might be sending everyone to the hospital, even people with just a cold, which would clog up the emergency rooms. We need a way to see how the model makes mistakes, not just if it makes them.

This is exactly what the researchers at Iyawo Health in Nigeria set out to do. They created a new, super-detailed test called IyawoBench v2.0. Instead of just asking "Is it safe?", they built a mathematical microscope to look at three specific ways these AI models can trip up. They tested three of the smartest AI models available today (Claude Sonnet 4.6, Llama 3.3 70B, and Llama 3.1 8B) using 200 fake but realistic patient stories based on real data from 19 Nigerian health centers.

The results were a bit of a shocker. The researchers found that every single model they tested had at least one major flaw, even the ones that looked perfect on the old, simple tests.

Here is what they discovered about the three "flaws" they invented to catch these errors:

The "Over-Protective" Flaw (Conservative Escalation Bias): Imagine a bodyguard who is so scared of danger that they tackle everyone who walks through the door, even the mailman. One of the models, Claude Sonnet 4.6, acted like this. It was great at spotting real emergencies, but it also sent too many people with minor problems to the hospital. This is bad because it overwhelms the hospital, making it harder for the truly sick people to get help.
The "Under-Protective" Flaw (Systematic Downgrade Bias): This is the most dangerous one. Imagine a guard who sees a fire but tells the people to "wait and see" instead of calling the fire department. The Llama 3.1 8B model did this. On the old tests, it looked 100% safe because it never sent a critical patient home. But the new test revealed a scary secret: it downgraded 77 out of 100 critical patients to a "come back tomorrow" status. In real life, that delay could be fatal for someone with a severe infection.
The "Confused Middle" Flaw (Middle-Tier Instability): Imagine a traffic cop who can't decide if a car should go fast or slow, so they wave it in both directions randomly. The Llama 3.3 70B model was great at spotting the extremes (super sick or totally fine) but got totally confused about the "medium" cases. It couldn't decide if these patients needed to go to the hospital today or tomorrow, flipping back and forth.

The paper also showed that there is no single "best" AI model for every situation. It depends on what the hospital needs most.

If the hospital is desperate to catch every single emergency and doesn't mind a crowded waiting room, the "over-protective" model (or even a simple rule that says "send everyone to the hospital") might be the best choice.
If the hospital is already full and can't handle more people, the "under-protective" model (which keeps people at home unless they are very sick) might actually be the most efficient, despite its risks.
If you want a balance, the "confused middle" model might be the winner.

The big takeaway is that we can't just pick one AI and say, "This is the winner." The "best" model changes depending on the specific problems the hospital is facing. The authors suggest that instead of just looking for a high score, we need to use their new "cost" math to figure out which mistakes a specific hospital can afford to make and which ones it absolutely cannot. They proved that the old way of testing AI was hiding these dangerous patterns, and we need this new, more detailed map to navigate the future of medical AI safely.

Technical Summary: IyawoBench v2.0

Problem Statement

The deployment of Large Language Models (LLMs) as clinical triage tools in Low- and Middle-Income Countries (LMICs) faces a critical evaluation gap. Existing safety metrics, often relying on binary accuracy or lenient sensitivity scores, produce misleading confidence. A model may score 100% on conventional safety measures (e.g., correctly identifying that an emergency requires some referral) while exhibiting systematic failure modes—such as downgrading life-threatening emergencies to "return tomorrow" rather than "refer now"—that render it unsafe for real-world deployment. Current benchmarks fail to distinguish between clinically equivalent errors and clinically dangerous errors, particularly within ordinal decision spaces where error costs are highly asymmetric.

Methodology

The authors introduce IyawoBench v2.0, an extended diagnostic evaluation framework combining a synthetic dataset with a formal mathematical theory of triage failure.

Dataset Construction

The evaluation utilizes 200 synthetic clinical vignettes derived from statistical distributions of 1,200 real patient encounters across 19 primary health centres in Oyo State, Nigeria.

Scope: Eight febrile illness categories (including Malaria, Typhoid, Pneumonia, Meningitis, and Sepsis) based on WHO and Nigerian Standard Treatment Guidelines.
Labels: Ground-truth triage decisions are ordinal: $y_0$ (TREAT_HERE), $y_1$ (REFER_TODAY), and $y_2$ (REFER_NOW).
Distribution: The dataset is stress-tested with an over-representation of high-acuity cases (50% REFER_NOW, 30% REFER_TODAY, 20% TREAT_HERE).

Formal Mathematical Framework

The paper formalizes triage as an ordinal classification problem with asymmetric error costs, introducing 14 definitions and 2 theorems to decompose safety into three distinct failure modes:

Conservative Escalation Bias (CEB): A model that correctly identifies emergencies but over-escalates low-acuity cases (TREAT_HERE $\to$ $\to$ REFER_NOW/TODAY) at rates that could collapse referral capacity.
- Metric: Escalation Bias Index (EBI), defined as the difference between lenient sensitivity and specificity on low-acuity cases.
Systematic Downgrade Bias (SDB): A model that appears safe under lenient metrics (never sending an emergency home) but systematically downgrades true emergencies by one level (REFER_NOW $\to$ $\to$ REFER_TODAY).
- Metric: Strict Sensitivity vs. Lenient Sensitivity.
Middle-Tier Instability (MTI): High error rates and bidirectional confusion (upshifts and downshifts) specifically within the moderate-urgency (REFER_TODAY) class.
- Metric: Middle-Tier Confusion Rate (MCR) and Direction Asymmetry.

Additionally, the authors propose Expected Deployment Cost (EDC), a metric that translates model errors into asymmetric clinical costs reflecting Nigerian Primary Health Centre (PHC) realities (e.g., missing a life-threatening emergency is weighted 20x higher than over-referring a stable patient).

Experimental Setup

The framework was applied to three frontier models:

Claude Sonnet 4.6 (Proprietary)
Llama 3.3 70B (Open-weight, Large)
Llama 3.1 8B (Open-weight, Small)
Baselines: Five simple strategies (e.g., always refer now, always treat here) were included for comparison.

Evaluation was conducted across three distinct deployment scenarios with varying cost matrices: Emergency-Focused, System-Sustainability, and Balanced-Deployment.

Key Results

1. Universal Presence of Failure Modes

All three evaluated models exhibited at least one formal failure mode, demonstrating that "state-of-the-art" performance does not equate to clinical safety in triage contexts.

Claude Sonnet 4.6: Exhibited CEB. It correctly identified emergencies but over-escalated 85% of moderate cases (REFER_TODAY) to REFER_NOW.
Llama 3.3 70B: Exhibited CEB, MTI, and SDB simultaneously. It showed bidirectional instability in the middle tier and over-escalation of low-acuity cases.
Llama 3.1 8B: Exhibited SDB. While it achieved 100% lenient sensitivity (no emergencies sent home), its Strict Sensitivity collapsed to 23%. It systematically downgraded 77% of true emergencies (REFER_NOW) to REFER_TODAY.

2. The "Hidden" Under-Triage Gap

The most significant finding is the 77 percentage point gap between lenient and strict sensitivity for Llama 3.1 8B. Under conventional v1.0 metrics, the model appeared 100% safe. Under the new strict metric, it failed to identify 77 out of 100 true emergencies as requiring immediate referral. In a real deployment, this would result in patients with conditions like bacterial meningitis being told to return in 24 hours rather than receiving immediate care.

3. Context-Dependent Model Ranking

The paper demonstrates that no single model is optimal across all contexts. The ranking of models changes entirely based on the deployment scenario's cost matrix:

Emergency-Focused: The simple "always_refer_now" baseline was optimal (lowest cost), followed by Claude Sonnet 4.6.
System-Sustainability: Llama 3.1 8B was optimal (lowest cost), followed by the simple "always_refer_today" baseline.
Balanced-Deployment: Llama 3.3 70B was optimal, followed by Claude Sonnet 4.6.

Significance and Claims

The paper claims that single-ranking benchmarks are structurally inadequate for selecting clinical AI in resource-constrained health systems.

Diagnostic over Aggregate: The authors argue that aggregate accuracy and binary safety scores conceal critical failure modes. The proposed framework (EBI, SDB, MTI, EDC) provides a diagnostic instrument to identify how a model fails, not just if it fails.
Architecture-Agnostic Failure: Failure modes are not inherent to specific model families or scales (e.g., proprietary vs. open, large vs. small) but are properties of how models interact with clinical decision structures. Even a simple constant-prediction baseline can exhibit formal failure modes (Theorem 1).
Contextual Selection: The "best" model depends entirely on the health system's specific constraints (e.g., whether the bottleneck is referral capacity or the risk of missing emergencies). The Expected Deployment Cost framework offers a principled mechanism to align model selection with these specific system priorities.
Transferability: While tested on febrile illness triage, the mathematical framework is claimed to be applicable to any clinical AI system operating on ordinal decision spaces with asymmetric error costs.

The authors conclude that IyawoBench v2.0 serves as both a rigorous benchmark and a diagnostic tool, enabling deploying organizations to implement targeted mitigations (e.g., human-in-the-loop review for specific failure modes) rather than discarding AI deployment entirely.