💬 NLP

An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

The paper introduces RARE-PHENIX, an end-to-end AI framework that integrates large language models for extracting, standardizing, and prioritizing Human Phenotype Ontology terms from clinical notes, demonstrating superior performance over existing baselines in supporting rare disease diagnosis across multi-site datasets.

Original authors: Cathy Shyr, Yan Hu, Rory J. Tinker, Thomas A. Cassini, Kevin W. Byram, Rizwan Hamid, Daniel V. Fabbri, Adam Wright, Josh F. Peterson, Lisa Bastarache, Hua Xu

Published 2026-02-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cathy Shyr, Yan Hu, Rory J. Tinker, Thomas A. Cassini, Kevin W. Byram, Rizwan Hamid, Daniel V. Fabbri, Adam Wright, Josh F. Peterson, Lisa Bastarache, Hua Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a very tricky mystery: Why is this patient sick?

In the world of rare diseases, the clues are often hidden inside thousands of pages of messy, handwritten-style doctor's notes. These notes are full of descriptions like "the child seems small for their age" or "they get tired easily." But to solve the mystery, the detective needs to translate these messy descriptions into a specific, standardized code (like a barcode) that a computer can use to find the answer. This process is called phenotyping.

The problem is that doing this manually is like trying to read a million books to find three specific words. It takes forever, it's exhausting, and it's hard to scale.

Enter RARE-PHENIX, a new AI framework developed by researchers at Vanderbilt and other institutions. Think of RARE-PHENIX not as a single robot, but as a three-person detective team working together to solve the case faster and more accurately than ever before.

Here is how this team works, broken down into simple steps:

1. The Scout (Extraction)

The Job: Read the messy notes and find the clues.
The Old Way: Previous AI tools were like a magnifying glass that just highlighted any word that looked like a symptom. They would find "fever" and "tiredness," but they might also get distracted by irrelevant details.
The RARE-PHENIX Way: This team uses a super-smart Large Language Model (LLM)—think of it as a detective who has read every medical textbook ever written. It reads the doctor's notes and says, "Ah, 'small for age' isn't just a comment; that's a specific clue called 'Failure to Thrive'." It pulls out the relevant symptoms from the chaos.

2. The Translator (Standardization)

The Job: Turn the clues into a universal language.
The Problem: One doctor might write "scoliosis," another might write "curved spine," and a third might write "back is bent." If the computer doesn't know these are the same thing, it gets confused.
The RARE-PHENIX Way: This team member acts as a universal translator. It takes all those different ways of saying "curved spine" and converts them all into one official code: HP:0002650 (Scoliosis). This is based on a giant dictionary called the Human Phenotype Ontology (HPO). Now, the computer speaks the same language as the doctors.

3. The Filter (Prioritization)

The Job: Decide which clues actually matter.
The Problem: Imagine you have a list of 100 clues. "Has a nose," "Has eyes," and "Is breathing" are all true, but they don't help solve a rare disease mystery. They are too common. The real clues are the weird, specific ones like "extra fingers" or "unusual skin spots."
The RARE-PHENIX Way: This is the most important step. The team uses a smart ranking system to sort the list. It pushes the boring, common clues to the bottom and brings the rare, specific, and diagnostic clues to the very top. It's like a search engine that doesn't just show you all results, but shows you the most relevant ones first.

Why is this a big deal?

The researchers tested this system on over 16,000 real patient notes from a major hospital. They compared it to the current "best" AI tool (called PhenoBERT).

The Result: RARE-PHENIX was much better at finding the right clues and putting them in the right order.
The Analogy: If the old AI was like a person shouting a list of 50 random words at you, RARE-PHENIX is like a detective handing you a sticky note with the top 3 most important words that will actually solve the case.

The Bottom Line

Rare disease patients often wait years for a diagnosis because their symptoms are hard to pin down. RARE-PHENIX acts as a force multiplier for doctors. It doesn't replace the doctor; instead, it does the heavy lifting of reading the notes, translating the jargon, and highlighting the most important clues.

By automating this "detective work," the AI helps doctors skip the long, boring parts of the investigation and focus immediately on the most likely answers, potentially saving patients years of uncertainty. It turns a mountain of messy paperwork into a clear, actionable roadmap for diagnosis.

1. Problem Statement

Rare disease diagnosis is hindered by a "diagnostic odyssey" averaging 4–8 years, largely due to the difficulty of translating unstructured clinical notes into structured, standardized phenotypes.

The Gap: While Electronic Health Records (EHRs) contain vast amounts of data, rare disease features are often embedded in unstructured text and poorly captured by standard billing codes.
Limitations of Existing AI: Current AI approaches typically optimize isolated components (e.g., just extraction or just normalization) rather than modeling the full clinical workflow. They often fail to distinguish between common, non-specific symptoms (e.g., "fatigue") and diagnostically informative phenotypes, leading to lists that are accurate but clinically unhelpful for narrowing differential diagnoses.
Goal: To develop an end-to-end AI framework that mimics the clinician's workflow: extracting features, standardizing them to the Human Phenotype Ontology (HPO), and prioritizing the most diagnostically relevant terms.

2. Methodology: The RARE-PHENIX Framework

The authors developed RARE-PHENIX (RARE disease PHENotyping with Intelligent eXtraction), a modular, three-stage pipeline designed to process unstructured clinical notes into a ranked list of HPO terms.

Module 1: Phenotype Extraction

Objective: Identify and extract rare disease features from raw clinical text.
Approaches:
1. Instruction Fine-Tuning: Open-source LLaMA models (ranging from 1B to 70B parameters, including LLaMA-2, 3, 3.1, 3.2, and 3.3) were fine-tuned using Parameter-Efficient Fine-Tuning (PEFT/QLoRA).
  - Training Data: A combination of the RareDis Corpus (832 expert-annotated documents from the National Organization for Rare Disorders) and Synthetic Clinical Narratives generated by ChatGPT-4o based on clinician-curated HPO terms from 2,671 Undiagnosed Diseases Network (UDN) patients.
2. Few-Shot Prompting: A closed-source Azure OpenAI ChatGPT-4o model was used with few-shot prompting to extract features without fine-tuning, serving as a deployable alternative.
Output: Raw text spans of phenotypes wrapped in HTML tags.

Module 2: Standardization to HPO Terms

Objective: Map extracted free-text phenotypes to structured HPO terms (e.g., "near sighted" $\rightarrow$ HP:0000545 Myopia).
Approach: Retrieval-Augmented Generation (RAG).
- A vector database of HPO terms was created using sentence embeddings (text-embedding-3-small).
- For each extracted feature, the top-10 semantically similar HPO terms are retrieved.
- An LLM is prompted to select the most appropriate HPO term based on the retrieved context and ontology definitions. If no match is found, it outputs "none."
Benefit: Grounds the model in external knowledge to prevent hallucinations and ensures interoperability for downstream genomic analysis.

Module 3: Prioritization of Diagnostically Informative Phenotypes

Objective: Rank the standardized HPO terms to surface the most clinically relevant ones, filtering out common/non-specific symptoms.
Approach: Supervised Learning-to-Rank.
- Task: A pairwise ranking model learns to assign higher scores to clinician-curated (positive) HPO terms compared to non-curated (negative) terms.
- Features: Includes patient demographics (age, sex) and knowledgebase-derived metrics (OMIM, Orphanet) such as information content, gene/disease association counts, and Inverse Document Frequency (IDF).
- Model: XGBoost with a pairwise ranking objective was selected as the best performer after hyperparameter tuning.
Output: A ranked list of HPO terms where the top entries are most likely to aid diagnosis.

3. Study Design and Data

Training Cohort: 2,671 patients from 11 UDN clinical sites (excluding Vanderbilt).
External Validation Cohort: 143 patients from Vanderbilt University Medical Center (VUMC), totaling 16,357 real-world clinical notes. This cohort was held out completely during model development.
Gold Standard: Clinician-curated HPO terms from the UDN workflow.
Baseline Comparator: PhenoBERT, a state-of-the-art deep learning system for HPO extraction.

4. Key Results

The framework was evaluated using ontology-based semantic similarity (Lin measure), Precision, Recall, and F1 scores.

End-to-End Performance: RARE-PHENIX consistently outperformed PhenoBERT.
- Ontology-based Similarity: RARE-PHENIX (using LLaMA-2-70b) achieved 0.70 similarity at top-50 cutoffs, compared to 0.58 for PhenoBERT.
- Precision/Recall: RARE-PHENIX showed significantly higher precision and F1 scores. For example, adding the standardization module (Module 2) increased precision from 0.25 to 0.43 for LLaMA-2-70b.
Ablation Analysis:
- Standardization (Module 2): Crucial for improving precision and F1 by reducing noise and mapping to specific terms.
- Prioritization (Module 3): Significantly improved performance, particularly at lower top-k cutoffs (k=10 to k=30). Prioritization increased ontology similarity by ~0.06–0.09 and precision by ~0.11–0.14 compared to random ordering, proving that ranking matters as much as extraction.
Error Analysis:
- RARE-PHENIX reduced False Negatives by 29% and False Positives by 24% compared to PhenoBERT.
- Most errors were attributed to linguistic variations in clinical notes (e.g., "severe growth retardation" vs. "failure to thrive") rather than model extraction failures.

5. Key Contributions

End-to-End Workflow Modeling: First framework to operationalize the entire clinical phenotyping workflow (Extraction $\rightarrow$ Standardization $\rightarrow$ Prioritization) rather than optimizing components in isolation.
Prioritization Mechanism: Introduces a supervised learning-to-rank module that specifically filters for diagnostically informative phenotypes, addressing the "dilution" problem where accurate but irrelevant terms obscure diagnostic signals.
RAG for Standardization: Demonstrates the efficacy of Retrieval-Augmented Generation for grounding LLM outputs in the Human Phenotype Ontology, reducing hallucinations.
External Validation: Validated on a large, independent, real-world dataset (16k+ notes) from a different institution, demonstrating strong generalizability.

6. Significance and Future Impact

Clinical Utility: By aligning AI output with the actual clinician workflow, RARE-PHENIX produces structured, ranked phenotypes that are more concordant with human curation. This can significantly reduce the manual burden of phenotyping.
Scalability: The modular design allows the extraction and standardization components to be deployed upstream of other diagnostic tools (e.g., gene prioritization pipelines).
Resource Flexibility: The study shows that while large models (70B) perform best, in-context learning with ChatGPT-4o achieves comparable recall, offering a viable path for resource-constrained settings where local fine-tuning is not feasible.
Future Work: The authors note the need for prospective studies to measure the impact on diagnostic accuracy, time-to-diagnosis, and clinician workflow efficiency in real-time clinical settings.

In summary, RARE-PHENIX represents a shift from treating phenotyping as a simple text extraction task to modeling it as a complex, multi-step clinical reasoning process, resulting in AI outputs that are significantly more actionable for rare disease diagnosis.