💻 computer science

PhenoAgent: agentic LLM framework for phenotyping electronic health records via structured query decomposition and self-correction

PhenoAgent is a privacy-preserving, agentic LLM framework that utilizes structured query decomposition, self-correction, and multi-model consensus to accurately extract clinical phenotypes from free-text electronic health records, significantly outperforming traditional baselines while matching physician-level performance on Japanese hospital datasets.

Original authors: Yuki Kashiwada, Rieko Sakurai, Yuta Yokokawa, Kenichiro Ando, Chinatsu Gocho, Chiaki Murata, Kosuke Tohda, Maki Tanioka, Shinya Sonobe, Tatsuya Harada, Jun Takayama

Published 2026-06-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuki Kashiwada, Rieko Sakurai, Yuta Yokokawa, Kenichiro Ando, Chinatsu Gocho, Chiaki Murata, Kosuke Tohda, Maki Tanioka, Shinya Sonobe, Tatsuya Harada, Jun Takayama

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a hospital as a massive library where every patient's story is written in a unique, messy diary (the Electronic Health Record). Most of these stories are written in free-flowing paragraphs, not neat checklists. If a doctor wants to find all the patients who had a specific, complex situation—like "a heart attack that required a specific life-support machine"—they can't just search for the words "heart attack" and "machine." They have to read through thousands of messy diaries to find the ones where those two things happened together, often described in different ways or hidden in the middle of long sentences.

PhenoAgent is a new, smart tool designed to act like a super-efficient, privacy-focused librarian who can read these messy diaries and find exactly the right patients without ever leaving the hospital building.

Here is how it works, broken down into simple steps:

1. The "Recipe Breakdown" (Structured Query Decomposition)

Imagine you ask a chef to make a "Spicy Beef Stir-fry with Tofu." Instead of just shouting the order, PhenoAgent first breaks that order down into a strict shopping list:

Must have: Beef
Must have: Tofu
Must have: Spicy sauce
Must NOT have: Chicken

In the paper, this is called structured query decomposition. The system takes a complex medical question and turns it into a clear, structured checklist (a JSON schema) before it even starts reading the patient records. This stops the computer from getting confused by messy language.

2. The "Self-Correction Loop" (Self-Correction)

Sometimes, even smart computers get confused and write their answers in the wrong format (like writing a paragraph instead of a checklist). PhenoAgent has a built-in "spell-checker" for logic.

It asks three different AI "experts" to read the patient diary and fill out the checklist.
If an expert makes a mistake or writes the answer in the wrong format, the system says, "Wait, that doesn't look right. Try again."
The expert tries again (up to two more times) until the answer fits the checklist perfectly. This is the self-correction loop.

3. The "Unanimous Jury" (Unanimous Ensemble Voting)

This is the most important safety rule. PhenoAgent doesn't just ask one AI; it asks three different AI models (GPT-OSS-20B, DeepSeek-32B, and Llama-3.1-Swallow-70B).

Think of these three AIs as a jury.
To decide that a patient does match the criteria, all three jurors must vote "Yes."
If even one juror says "No" or "I'm not sure," the patient is not selected.
This makes the system very careful. It might miss a few patients (lower sensitivity), but it is extremely unlikely to pick someone who doesn't belong (high precision). It's better to be safe than sorry when dealing with medical data.

Why is this special?

Most AI tools today live in the "cloud" (on big servers owned by tech companies). Hospitals often can't send patient data there because of privacy laws.

The "Local" Advantage: PhenoAgent is designed to run entirely on a single computer inside the hospital. It uses "open-weight" models (like free, open-source software) so the hospital keeps all the data on its own walls. No data leaves the building.
The "Non-English" Advantage: It was tested specifically on Japanese hospital records, which are notoriously difficult for computers because they mix different writing systems and abbreviations.

The Results: How well did it work?

The researchers tested PhenoAgent on real patient records from two different Japanese hospitals.

The Gold Standard: They compared the AI's work against two human doctors. A patient was only counted as a "match" if both doctors agreed they fit the criteria.
The Score: PhenoAgent performed almost as well as the two doctors agreeing with each other. It scored a "macro-F1" of about 71.6, which is very close to the human agreement score of 71.4.
Beating the Old Ways: It crushed the old methods (like simple keyword searching or standard database searches) by a huge margin—improving accuracy by 24 to 43 points.

The Catch (Limitations)

The paper notes that because PhenoAgent uses the "Unanimous Jury" rule, it is very strict.

If a patient's condition is described vaguely or indirectly in the notes, the AI might say "No" because the three jurors couldn't all agree.
This is a trade-off: The system prioritizes accuracy (making sure it doesn't pick the wrong people) over finding every single possible match. This is great for clinical research where you need a very clean list of patients, but it might miss some rare cases.

Summary

PhenoAgent is a privacy-safe, locally-run AI librarian that breaks complex medical questions into checklists, double-checks its own work, and requires three different AI experts to agree before selecting a patient. It proved it could find the right patients in messy Japanese hospital records almost as well as human doctors, without ever sending sensitive data to the internet.

Technical Summary: PhenoAgent

Problem Statement

Clinical phenotype information is predominantly embedded in unstructured, free-text electronic health records (EHRs), creating a significant barrier to structured queries for cohort selection and clinical trial recruitment. While computational phenotyping is essential for secondary EHR use (e.g., research, quality improvement), existing methods face limitations:

Standard queries and keyword matching struggle with synonyms, negation, and complex contextual expressions.
Rule-based and dictionary-based methods require extensive manual curation and lack flexibility for ad-hoc composite phenotypes (e.g., "heart failure treated with ECMO").
Machine learning approaches often depend on task-specific fine-tuning and annotated datasets, limiting routine clinical deployment.
Generative Large Language Models (LLMs) offer promise but face practical constraints in clinical settings, specifically regarding patient privacy (limiting cloud-based API use) and the need for local, privacy-preserving deployment within hospital firewalls.
Non-English contexts, particularly Japanese clinical text, present unique challenges due to multiple writing systems, institution-specific abbreviations, and non-segmented phrasing, with fewer available annotated resources compared to English.

Methodology: PhenoAgent Framework

The authors propose PhenoAgent, a locally deployable, three-stage agentic LLM framework designed to retrieve composite clinical phenotypes from free-text Japanese EHRs without relying on cloud services or task-specific annotation.

1. Structured Query Decomposition

Instead of processing a natural language query directly against a record, PhenoAgent first decomposes a clinical query $Q$ into a structured JSON schema $S_k$ . This schema enumerates the constituent medical concepts required to answer the query and assigns a Boolean field to each. This step constrains the downstream extraction task and reduces ambiguity, allowing the system to handle complex combinations of diagnosis, treatment, and comorbidity without per-query fine-tuning.

2. Self-Correcting Semantic Evaluation

The framework evaluates each clinical record $C$ against the schema $S_k$ using open-weight LLMs.

Process: The model generates a JSON object containing binary values and free-text rationales for each field in the schema.
Validation: Outputs are validated against the required JSON schema. If validation fails (e.g., malformed JSON), the system triggers a self-correction loop, prompting the same model again with a format reminder. Up to two retries are allowed (three total attempts).
Fallback: If a valid output is not obtained after retries, unresolved fields are set to negative. This mechanism, inspired by the Reflexion framework, resolves the majority of format failures.

3. Unanimous Ensemble Voting

To ensure stability and reduce model-specific hallucinations, PhenoAgent employs an ensemble of three distinct open-weight LLMs: GPT-OSS-20B, DeepSeek-32B, and Llama-3.1-Swallow-70B.

Decision Rule: A case is classified as positive only when all three constituent models unanimously agree that every required field in the schema is positive.
Rationale: This conservative voting rule prioritizes precision over sensitivity, filtering out unstable predictions and ensuring that only high-confidence retrievals are accepted.

Implementation Details

Deployment: The entire pipeline runs locally on a single workstation (Apple M4 Max with 128GB unified memory) using 8-bit quantized models via the MLX framework. No external API calls are made.
Datasets: Evaluated on two independent Japanese emergency department datasets:
- TUH-DEM: 842 admissions from Tohoku University Hospital (2022–2023).
- OUH-DEM: 196 admissions from Okayama University Hospital (2023–2024).
Reference Labels: Ground truth was established via two-physician concordance. A record was labeled positive only if two independent board-certified emergency physicians both judged it to match the query.

Key Results

Performance Metrics

PhenoAgent was benchmarked against three information-retrieval baselines (Keyword Search, Semantic Search with ICD-10 expansion, BM25) and six single-LLM baselines.

Macro F1 Scores:
- TUH-DEM (Internal): PhenoAgent achieved a macro F1 of 71.6 (95% CI 66.5–75.7). This matched the inter-physician agreement F1 of 71.4 and exceeded the best single-LLM baseline (GPT-OSS-20B at 67.3) and information-retrieval baselines (Semantic Search at 47.9) by 24–34 points.
- OUH-DEM (External): PhenoAgent maintained a macro F1 of 71.1 (95% CI 68.6–73.8), demonstrating generalizability across institutions with different documentation practices. It outperformed information-retrieval baselines by 35–43 points.
Agreement (Cohen's $\kappa$ ):
- PhenoAgent achieved $\kappa$ values of 0.69 (TUH-DEM) and 0.68 (OUH-DEM), significantly higher than keyword search ( $\approx$ 0.38–0.44) and single-LLM baselines (ranging from 0.31 to 0.61).
Precision-Sensitivity Trade-off:
- PhenoAgent maintained high precision (70.5% in TUH-DEM, 78.8% in OUH-DEM) comparable to keyword search but achieved substantially higher sensitivity (76.5% vs. 34.0% for keyword search in TUH-DEM).
- The unanimous voting strategy resulted in a conservative prediction profile, prioritizing high-confidence retrievals. While single LLMs achieved higher sensitivity in some external validation cases, PhenoAgent's precision remained robust.

Specific Observations

Rare Phenotypes: The framework performed well on rare phenotypes (prevalence <5%), though performance varied for clinically ambiguous conditions like sepsis, where evidence is often implicit or inferred from clinical trajectories rather than explicit terms.
Self-Correction: The self-correction loop resolved 84–99% of JSON-format failures, contributing a 1.3-point improvement in macro F1 in the TUH-DEM dataset.
Model Selection: The ensemble of general-purpose models (selected via exhaustive search) performed competitively against specialized medical models (e.g., MedLLM-72B), suggesting that instruction following and structured output generation are critical for this task.

Significance and Claims

The authors claim that PhenoAgent demonstrates that agentic LLM workflows can support flexible, privacy-preserving phenotype retrieval from free-text EHRs in non-English clinical settings.

Key contributions highlighted include:

Local Deployability: The framework operates entirely within hospital-managed environments using open-weight models, addressing critical barriers regarding patient privacy, data governance, and network restrictions.
Flexibility without Fine-Tuning: By using structured query decomposition, the system handles composite phenotypes (diagnosis + treatment/comorbidity) without requiring per-query fine-tuning or extensive ontology engineering.
Performance Parity with Experts: The system's performance approaches the level of inter-rater consistency observed between human physicians, suggesting it can serve as a reliable tool for clinical research cohort selection.
Robustness in Non-English Contexts: The successful application to Japanese EHRs validates the approach for settings where annotated datasets and clinical language resources are limited.

The paper concludes that while the unanimous voting rule creates a precision-first system suitable for research cohort selection (where false positives increase manual review burden), the framework is modular. It can adapt to different precision-sensitivity balances (e.g., via majority voting) and incorporate stronger local models as they become available, serving as a practical bridge between general-purpose LLM advances and reliable clinical research infrastructure.