🤖 machine learning

Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care

This paper introduces MANANA, a non-parametric prompt-learning framework that enhances LLM-based anti-seizure medication recommendations in Ugandan pediatric epilepsy care by learning local prescribing patterns from unstructured notes and utilizing Bayesian prompt averaging to generate uncertainty-aware deferral signals, thereby achieving higher accuracy and enabling selective prediction for specialist review.

Original authors: Shreyas Rajesh, Kartik Sharma, Tonmoy Monsoor, Mehmet Yigit Turali, Richard Idro, Juliana Kayaga, Robert Sebunya, Tracy Tushabe Namata, Jessica Nichole Pasqua, Vwani Roychowdhury, Rajarshi Mazumder

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shreyas Rajesh, Kartik Sharma, Tonmoy Monsoor, Mehmet Yigit Turali, Richard Idro, Juliana Kayaga, Robert Sebunya, Tracy Tushabe Namata, Jessica Nichole Pasqua, Vwani Roychowdhury, Rajarshi Mazumder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a general doctor working in a busy clinic in Uganda. You have a patient with epilepsy, and you need to decide which medication to prescribe. The problem is that the "expert" advice you usually rely on (from big medical textbooks or AI trained in wealthy countries) often doesn't fit your reality. Maybe the expensive drug isn't available, or the local way of treating patients is different. You need an AI assistant that understands your specific neighborhood, not just the textbook.

This paper introduces MANANA, a new way to teach Large Language Models (LLMs) to be that helpful local assistant, specifically for epilepsy care in resource-limited settings.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Tourist" Doctor

Standard AI models are like tourists. They have read all the medical books (trained on data from rich countries like the US or UK), but they don't know the local rules.

If you ask a tourist AI for a prescription, it might suggest a drug that is too expensive or simply unavailable in Uganda.
It makes mistakes because it's applying "Western rules" to a "Ugandan reality."
The paper found that while these AIs could guess some things correctly, their errors were systematic—they were confidently wrong because they didn't know the local context.

2. The Solution: MANANA (The "Apprentice" System)

Instead of trying to retrain the AI's brain (which is heavy, expensive, and hard to audit), the authors created MANANA. Think of MANANA as a smart apprentice who learns by keeping a "field journal" of mistakes and lessons.

MANANA works like a three-person team:

The Predictor (The Apprentice): Looks at the patient's notes and suggests three possible medication plans.
The Inspector (The Critic): Checks the Apprentice's suggestions against what the real doctor actually prescribed. If they don't match, the Inspector writes a note explaining why it was wrong (e.g., "Don't suggest Drug X here; it's out of stock").
The Architect (The Teacher): Looks at all the Inspector's notes. If the same mistake happens over and over with different patients, the Architect writes a permanent rule into the "Field Journal" (the memory) to prevent it from happening again.

The Magic: The AI doesn't change its internal brain code. Instead, it just updates its "Field Journal." This makes it easy to audit (doctors can read the journal to see what the AI learned) and easy to adapt to new clinics.

3. Two Ways to Learn

The paper tested two versions of this apprentice:

MANANA-Single: The team writes one big list of rules (e.g., "Always check Drug A first").
MANANA-Multi: The team creates a group of "specialist agents." One agent focuses on drug availability, another on side effects, another on patient history. They work together to solve the puzzle.

4. Knowing When to Say "I Don't Know" (The Safety Valve)

In medicine, it's dangerous for an AI to guess when it's unsure. The paper introduces a feature called Bayesian Prompt Averaging (BPA).

Think of this as the AI checking its own confidence.

As the AI learns, it goes through different "stages" of its Field Journal.
BPA looks at all these stages and asks: "How sure are we that this answer is right?"
High Confidence: If the AI is very sure (e.g., 99% sure), it gives the prescription to the doctor to use.
Low Confidence: If the AI is shaky, it raises a red flag and says, "I'm not sure about this one; please send this patient to a specialist."

This is crucial because in places with few specialists, you don't want to waste their time on easy cases. You want them to only see the hard, uncertain cases.

5. The Results

The team tested this on real patient records from two different hospitals in Uganda.

Better than the Basics: MANANA was much better at guessing the right medication than standard AI prompts or old-school computer models.
The "Multi" Team Won: The version with the group of specialist agents (MANANA-Multi) performed the best.
The Safety Net Worked: When the system was allowed to "defer" (pass on) the cases it wasn't sure about, its accuracy on the cases it did handle skyrocketed.
- It could handle the most confident 50% of cases with 95% accuracy.
- It could handle the most confident 25% of cases with 99% accuracy.

Summary Analogy

Imagine you are trying to navigate a city you've never visited.

Standard AI is like a GPS trained only on New York City. It will tell you to "turn left at the subway," but in your city, there is no subway.
MANANA is like a local guide who starts with a blank notebook. Every time the GPS gives a wrong turn, the guide writes down, "No subway here, turn right instead." After a few days, the guide has a perfect, custom map for your city.
BPA (The Confidence Check) is the guide saying, "I know the route to the market perfectly, but I'm not sure about the route to the mountain. Let's ask a local expert for the mountain part."

The paper concludes that this approach allows AI to learn from limited local data without needing expensive retraining, making it a practical tool for helping doctors in places where specialist care is scarce.

Technical Summary: Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care

Problem Statement
Specialist epilepsy expertise is scarce in low- and middle-income countries (LMICs), where generalist clinicians manage longitudinal treatment with limited diagnostic resources (e.g., lack of EEG or imaging) and constrained medication availability. While Large Language Models (LLMs) offer potential for decision support, current medical AI benchmarks are dominated by high-income settings with structured Electronic Health Records (EHRs). Consequently, standard LLMs fail to adapt to local prescribing practices, medication stock-outs, and follow-up norms in resource-constrained contexts. Furthermore, existing systems lack mechanisms to quantify uncertainty, preventing the strategic deferral of difficult cases to scarce specialists. The core challenge is to develop a system that learns local prescribing logic from unstructured clinical notes without requiring parametric weight updates (which are computationally expensive and difficult to audit) and can provide confidence scores to enable selective prediction.

Methodology: MANANA Framework
The authors introduce MANANA, a non-parametric prompt-learning framework designed to adapt LLMs to local clinical environments using a small patient-level training set (50 patients in the study). MANANA operates without updating model weights, instead learning an explicit, auditable memory state that conditions future predictions.

Multi-Agent Learning Loop:
- Predictor: An LLM agent that, given a patient's history and the current memory state, proposes three candidate anti-seizure medication (ASM) regimens.
- Inspector: An agent that compares the predicted regimens against the actual physician prescription. It generates structured reports and "candidate learnings" (concise lessons grounded in the specific case note).
- Architect: An agent that aggregates evidence from an append-only buffer of candidate learnings. It updates the memory state only when recurring patterns are detected across distinct clinic visits, ensuring robustness against noise.
Memory Variants:
- MANANA-Single: Learns a single shared list of correction rules (e.g., "If drug X is unavailable, switch to drug Y").
- MANANA-Multi: Instantiates a population of "specialist agents," where each agent is responsible for extracting a specific clinical signal. The Architect can spawn, edit, or prune these agents based on recurring evidence, maintaining a bounded, inspectable set of expert prompts.
Bayesian Prompt Averaging (BPA):
To address uncertainty, the authors propose BPA, a Bayesian model averaging procedure over the learned prompt trajectory. Instead of a single final prompt, the system treats the sequence of memory states ( $m_1, \dots, m_T$ ) as an ensemble of estimators.
- Posterior Weighting: Using a held-out validation set, BPA calculates the marginal likelihood of each memory state. States that perform better on validation data are assigned higher weights.
- Deferral Signal: The weighted average of predictions yields a prescription probability distribution. This allows the system to identify high-confidence cases (which can be auto-handled) and low-confidence cases (which are deferred to specialists), effectively routing scarce expert attention to where it is most needed.

Key Contributions

MANANA Framework: A non-parametric, error-grounded prompt-learning system that learns local prescribing decisions from unstructured notes without clinician-written rules or expert review of reasoning traces. It is instantiated in Single-agent and Multi-agent variants.
Performance in Low-Resource Settings: Demonstrated that MANANA, trained on a small dataset, achieves strong performance on held-out data across two independently collected Ugandan pediatric epilepsy cohorts, outperforming classical ML models, direct LLM prompting, and state-of-the-art prompt optimization baselines (e.g., TextGrad, DSPy).
Uncertainty-Aware Deferral: The introduction of BPA converts the learned prompt trajectory into prescription probabilities. This enables clinically meaningful selective prediction, allowing the system to handle the most confident cases with high precision while deferring uncertain cases for specialist review.

Experimental Results
The study utilized two cohorts from Ugandan referral centers: Cohort A (332 patients, 1,040 visits) for training/validation and Cohort B (367 patients, 1,509 visits) for independent transfer testing. The task was to predict ASM regimens (monotherapy or polytherapy) based on longitudinal unstructured notes.

Accuracy: MANANA-Multi achieved the highest Exact Match@3 (EM@3) scores. On Cohort A (Monotherapy, Visit 3), it reached 97.0%, outperforming the best prompt-optimization baseline (TextGrad at 87.3%) and direct prompting (59.3%). On Cohort B (Transfer), MANANA-Multi achieved 93.2% (Monotherapy, Visit 3) and 69.8% (Polytherapy, Visit 3), significantly outperforming classical baselines and demonstrating robustness across different documentation formats and clinicians.
Uncertainty and Deferral: BPA improved Top-1 accuracy and created a larger confidence gap between correct and incorrect predictions compared to simple majority voting.
- On the held-out Cohort B, the system achieved 99% precision on the top-25% most confident cases and 95% precision on the top-50% most confident cases.
- This allows a clinic to defer the bottom 50% of cases to specialists while maintaining 95% accuracy on the auto-handled half, effectively triaging workload.

Significance and Claims
The paper claims that MANANA provides a path toward locally adapted clinical LLM systems that can learn from limited, site-specific data without the burden of retraining model weights. By converting observed prescription errors into an auditable prompt memory, the system captures local prescribing rules (e.g., drug availability constraints) that standard medical knowledge bases miss.

Crucially, the authors emphasize that the system's ability to generate deferral signals via Bayesian prompt averaging addresses a critical deployment need: ensuring that scarce specialist attention is reserved for cases where the model is least certain. The study posits that this approach bridges the gap between high-level medical knowledge and the practical realities of low-resource clinical environments, offering a deployable solution for longitudinal care management where specialist expertise is unavailable.

Limitations
The authors note that the current deferral thresholds are chosen retrospectively on held-out data and would require prospective tuning in a real-world clinic. Additionally, the system optimizes for agreement with the treating physician's regimen (which reflects local resource constraints) rather than proving the regimen is clinically optimal for seizure freedom. Future work is needed to validate the learned memories against clinical outcomes and to integrate direct seizure prediction.