Leveraging Open-Source Large Language Models to Identify Undiagnosed Patients with Rare Genetic Aortopathies

This study demonstrates that an open-source Large Language Model pipeline, enhanced with retrieval-augmented generation on curated genetic corpora, can effectively analyze unstructured clinical notes to identify undiagnosed patients with rare genetic aortopathies who would benefit from genetic testing, achieving high accuracy and offering a promising decision-support tool for earlier disease recognition.

The Gist

Imagine your body is a complex, high-tech house. Most of the time, the lights flicker or the pipes leak because of simple, common issues like a burnt-out bulb or a clogged drain. But sometimes, the problem is a hidden, rare flaw in the house's original blueprints—a "genetic aortopathy." This is a rare condition where the main artery (the aorta) is weak and prone to bursting. Because these blueprints are so unique and the symptoms can look like many different common problems, doctors often miss them. If they miss it, the house could collapse suddenly and fatally.

The Problem: The Needle in the Haystack
Currently, finding these rare blueprint flaws relies on a doctor (the primary care physician) having a "gut feeling" that a patient's symptoms are actually genetic. They have to spot the needle in the haystack and then send the patient to a specialist. But with thousands of patients and millions of symptoms, many needles get lost in the hay. We need a better way to scan the whole haystack automatically.

The Solution: A Super-Smart Librarian
The researchers built a digital tool using an Open-Source Large Language Model (LLM). Think of this LLM as a super-smart, tireless librarian who has read every medical book, research paper, and case study ever written about these rare heart conditions.

However, just having a smart librarian isn't enough; they need to know which books to check for a specific patient. That's where RAG (Retrieval Augmented Generation) comes in.

• The Analogy: Imagine the librarian is taking a test. Instead of memorizing every fact, they are allowed to open a specific, curated "cheat sheet" of relevant medical knowledge right before answering. When the tool looks at a patient's notes, it instantly finds the most relevant medical facts from its "cheat sheet" and uses them to make a smart decision.

How It Works in Real Life
The team fed this digital librarian 22,510 pages of real patient notes (the "narrative details" doctors write down) from 500 people.

• The Test: They gave the tool 250 patients who did have the rare condition and 250 who didn't.
• The Job: The tool had to read the messy, free-flowing notes and decide: "Does this person need a genetic test?"

The Results: A Winning Scorecard
The tool performed incredibly well. Out of 499 patients, it correctly identified 425 of them.

• It was right about 83% of the time.
• It was very good at catching the people who actually needed help (high sensitivity) and very good at not panicking about people who were fine (high specificity).
• Only one patient was so confusing that the tool wisely said, "I'm not sure, a human doctor needs to take a closer look."

Why This Matters
This paper shows that we can use free, open-source AI to act as a safety net for doctors. Instead of relying on a human to remember every rare symptom, this tool scans the notes, cross-references them with the latest medical knowledge, and flags the patients who might be at risk.

The Bottom Line
Think of this as a digital early-warning system. It doesn't replace the doctor; it acts like a helpful assistant who whispers, "Hey, I noticed something in this patient's notes that matches a rare genetic pattern. Maybe we should run a genetic test to be safe." By catching these rare conditions earlier, we can prevent tragic heart failures and save lives.

Technical Summary

1. Problem Statement

Rare genetic aortopathies are often undiagnosed due to phenotypic heterogeneity (the wide variation in how symptoms present). This diagnostic delay poses a significant risk, as it can lead to fatal cardiac outcomes.

• Current Bottleneck: While genetic testing offers a pathway for early intervention, it relies heavily on primary care physicians (PCPs) to recognize the genetic nature of a patient's symptoms and initiate a referral to clinical genetics.
• The Gap: Many cases do not fit obvious diagnostic patterns, leading to missed opportunities for screening. There is a critical need for broad-scale, automated methods to identify these "hidden" cases from unstructured clinical data.

2. Methodology

The authors developed an open-source Large Language Model (LLM) pipeline designed to analyze unstructured clinical notes and recommend genetic testing. The technical architecture includes:

• Data Source: The system was trained and validated using 22,510 patient progress notes from 500 individuals in the Penn Medicine BioBank. The dataset was balanced with 250 confirmed cases (patients with genetic aortopathies) and 250 controls.
• Core Technology: The pipeline utilizes Retrieval-Augmented Generation (RAG).
  • Instead of relying solely on the LLM's pre-trained knowledge, the system retrieves specific, curated information from a genetic aortopathy-related corpus.
  • This ensures the model leverages up-to-date, relevant clinical knowledge when analyzing patient narratives.
• Workflow: The LLM processes free-text clinical notes to extract phenotypic details, compares them against the retrieved genetic knowledge base, and generates a recommendation on whether the patient should undergo genetic testing.

3. Key Contributions

• Open-Source Solution: The study demonstrates the viability of using open-source LLMs (rather than proprietary models) for high-stakes clinical decision support, promoting accessibility and reproducibility in medical AI.
• RAG Integration for Rare Diseases: It successfully adapts the RAG framework to the specific domain of rare genetic diseases, addressing the "knowledge gap" where standard LLMs might lack specific, nuanced clinical guidelines.
• Unstructured Data Utilization: The work proves that free-text clinical notes, often overlooked in structured data analyses, contain sufficient signal to automate the flagging of at-risk patients.

4. Results

The pipeline was rigorously validated on the 500-patient cohort with the following performance metrics:

• Coverage: The system successfully categorized 425 out of 499 patients. One case was excluded from automated categorization due to incomplete information, requiring further clinician evaluation.
• Performance Metrics:
  • Accuracy: 0.834
  • Precision: 0.835
  • Sensitivity (Recall): 0.831
  • Specificity: 0.836
  • F1-Score: 0.833
  • F3-Score: 0.832 (emphasizing recall, crucial for medical screening to avoid missing cases).

The balanced scores indicate the model is effective at both identifying true positive cases and minimizing false alarms.

5. Significance

This research illustrates a feasible pathway for early recognition of rare genetic disease risks in primary care settings.

• Clinical Impact: By acting as a decision-support tool, the system can assist clinicians in identifying patients who would otherwise be missed due to non-specific symptom presentation.
• Public Health: Earlier identification allows for proactive interventions, potentially preventing fatal cardiac events associated with undiagnosed aortopathies.
• Scalability: The use of open-source models and RAG suggests this approach can be scaled to other rare diseases and integrated into existing Electronic Health Record (EHR) systems to facilitate broad-scale screening.