Longitudinal information extraction from clinical notes in rare diseases: an efficient approach with small language models

This study demonstrates that small language models (SLMs) can effectively and efficiently extract longitudinal serum creatinine data from unstructured French clinical notes for rare kidney diseases, offering a privacy-preserving and resource-efficient alternative to traditional methods despite challenges with text duplication and language nuances.

The Gist

Imagine you are a detective trying to solve a mystery about a patient's health over many years. The clues are scattered everywhere, but they aren't in neat, organized files. Instead, they are hidden inside thousands of handwritten (or typed) diary entries—doctors' notes. These notes are messy, full of different ways of writing dates, and sometimes the clues are buried in paragraphs about other things.

This paper is about a new, clever way to find those hidden clues automatically, specifically for patients with rare kidney diseases.

Here is the story of what they did, explained simply:

The Problem: The "Needle in a Haystack"

Rare diseases are tricky because there are very few patients. To understand how the disease progresses, doctors need to track a specific number called serum creatinine (a measure of how well the kidneys are working) over time.

• The Haystack: The hospital's computer system has millions of pages of unstructured text (doctors' notes).
• The Needles: The specific numbers, dates, and units (like "145 µmol/L on March 15th") hidden inside those notes.
• The Old Way: Before this, humans had to read every single note and manually write down these numbers. It was slow, expensive, and prone to human error.
• The "Big Robot" Problem: Recently, people tried using giant Artificial Intelligence (AI) models (called Large Language Models) to read the notes. But these "Big Robots" are like supercomputers: they cost a fortune to run, require massive amounts of electricity, and often need to send private patient data to the cloud, which breaks privacy rules.

The Solution: The "Pocket-Sized Detective"

The researchers asked: Can we use a smaller, lighter, and cheaper AI (a "Small Language Model" or SLM) that can run right on a hospital's own computer without sending data anywhere?

Think of the Small Language Models as a team of smart, portable detectives. They aren't as powerful as the super-giant robots, but they are fast, cheap, and can stay inside the hospital walls to keep patient secrets safe.

How They Tested It

The team took 81 real patient notes from a French hospital and tried to teach four different "detectives" (four different AI models) to find the kidney numbers.

They tried different teaching methods (called "prompts"):

1. Zero-Shot: Just asking the detective, "Find the kidney numbers."
2. With Rules: Giving the detective a checklist: "Only look for kidney numbers, ignore family members' results, and fix the date formats."
3. Few-Shot: Showing the detective two examples of a perfect answer before asking it to work.

They also tested if the detectives worked better if spoken to in French (the language of the notes) or English.

The Results: The "Pocket Detective" Wins!

The results were surprisingly good.

• The Old Way (Rule-based): It was like using a metal detector that only beeps for perfect shapes. It missed a lot of clues (low "recall") because real notes are messy.
• The "Big Robot" (LLMs): Great at finding clues, but too heavy and expensive to use in a real hospital.
• The "Pocket Detective" (SLMs): The best detective (a model called Qwen3-8B) found 93% of the correct clues! It was much better than the old metal detector and almost as good as the giant robots, but it ran on a standard computer.

Key Findings:

• Bigger is slightly better: The slightly larger "detectives" (8 billion parameters) were better than the tiny ones (3 billion), but even the small ones did a great job.
• Instructions matter: Giving the detective a clear checklist (rules) helped them avoid mistakes, like confusing a patient's kidney numbers with their parent's numbers.
• Language doesn't matter much: Whether the detective was asked in French or English, they performed similarly well.

Why This Matters

This is a game-changer for rare diseases.

1. Privacy: Because these small models run locally, patient data never leaves the hospital.
2. Speed & Cost: Hospitals don't need to buy supercomputers. They can run this on regular servers.
3. Better Care: By automatically turning messy notes into clean data, doctors can finally see the full "movie" of a patient's kidney health, not just a few snapshots. This helps them predict the future of the disease and design better treatments.

The Bottom Line

The researchers proved that you don't need a massive, expensive AI to solve complex medical puzzles. With the right "small" AI and a little bit of smart instruction, you can unlock the hidden history of rare diseases from messy doctor's notes, making research faster, cheaper, and safer for everyone.

Technical Summary

1. Problem Statement

Rare diseases, such as renal ciliopathies, require rigorous longitudinal monitoring to track disease progression and design clinical trials. However, critical biomarker data (e.g., serum creatinine) are often locked in unstructured electronic health record (EHR) notes rather than structured databases.

• Data Scarcity: Annotated data is extremely limited in rare disease cohorts, making traditional supervised deep learning approaches (which require large training sets) infeasible.
• Deployment Barriers: While Large Language Models (LLMs) show promise in clinical NLP, their deployment in healthcare is hindered by high computational costs, privacy concerns regarding cloud-based processing, and implementation complexity.
• The Gap: There is a lack of evaluation regarding Small Language Models (SLMs) (typically <10B parameters) for extracting structured longitudinal data from real-world, multilingual (French) clinical notes without extensive fine-tuning.

2. Methodology

The authors developed and evaluated a pipeline to extract serum creatinine measurements (defined as a triplet: Date, Value, Unit) from French clinical notes.

Data Source

• Cohort: 11 patients with renal ciliopathies from Necker Hospital (a French national reference center).
• Dataset: 81 clinical notes containing 200 annotated ground-truth triplets.
• Preprocessing: Notes were preselected based on the co-occurrence of "creatinine," a numeric value, and a unit (µmol/L or mg/dL). Notes post-renal failure were excluded.

Model Selection

Four open-source SLMs (<10B parameters) were tested locally to ensure privacy and efficiency:

1. Mistral-7B-Instruct-v0.3
2. Llama-3.2-3B-Instruct
3. Qwen3-4B-Instruct
4. Qwen3-8B-Instruct

Experimental Design

• Prompting Strategies:
  • Zero-Shot (ZS): Direct instruction.
  • ZS + Rules: Explicit constraints (e.g., "exclude family members," "handle relative dates").
  • Few-Shot (FS): 2-shot examples with rules.
• Languages: Prompts were tested in both French (native note language) and English.
• Constraints: Models used schema-constrained generation (via the DSPy framework) to output JSON. Temperature was fixed at 0.
• Baseline: A regular expression (regex) based extractor was implemented for comparison.

Post-Processing Pipeline

To handle real-world inconsistencies, a post-processing step was applied:

• Normalization: Standardizing date formats (handling partial dates like "November 2020" by assigning the 1st), harmonizing decimal commas, and mapping units to a controlled vocabulary.
• Filtering: Discarding extractions that did not match serum creatinine (e.g., eGFR or urine creatinine) or belonged to family members.

Evaluation Metrics

• Ground Truth: Manually annotated by two experts.
• Metrics: Precision, Recall, and F1-score at the document level.
• Tolerance: A 30-day window was allowed for date matching to account for relative time expressions (e.g., "today").

3. Key Results

Performance Comparison

• Regex Baseline: Achieved high precision (0.820) but very low recall (0.254), resulting in an F1-score of 0.387. It failed to capture complex narrative structures.
• SLM Performance: All SLMs significantly outperformed the baseline.
  • Best Configuration: Qwen3-8B using Zero-Shot + Rules in English achieved the highest performance:
    • Precision: 0.958
    • Recall: 0.914
    • F1-Score: 0.936 (Note: The abstract mentions 0.928, likely referring to a specific subset or post-processing nuance, but the results section highlights 0.936).
  • Worst Configuration: Llama-3.2-3B with Zero-Shot + Rules in French (F1: 0.520).

Key Findings

1. Model Size Matters: Performance generally correlated with parameter size. Qwen3-8B > Qwen3-4B > Mistral-7B > Llama-3.2-3B.
2. Prompting Strategy:
   • Qwen models benefited most from explicit rules.
   • Llama-3.2-3B performed best with Few-Shot prompting.
   • Mistral-7B performed best with simple Zero-Shot (English).
3. Language: English prompts generally yielded slightly better results than French prompts, though Qwen3-4B performed well with French rules.
4. Robustness: Only the best model (Qwen3-8B) demonstrated perfect consistency when processing duplicated text across multiple notes, a common issue in clinical documentation.

Error Analysis

• False Positives: Primarily caused by misattributing dates when multiple dates appeared in a paragraph, or extracting family members' results despite exclusion rules.
• False Negatives: Often due to the model failing to link a value to a specific date in complex narratives.
• Post-Processing Impact: The pipeline successfully filtered out non-serum creatinine extractions (e.g., eGFR) that the models initially captured.

4. Key Contributions

1. First Real-World SLM Evaluation: This is the first study to evaluate SLMs for extracting longitudinal biomarker trajectories from real-world, unstructured French clinical notes in rare diseases.
2. Efficient Pipeline: Demonstrated that a combination of preselection, targeted prompting, and lightweight post-processing can achieve high accuracy (F1 > 0.9) without fine-tuning or cloud-based LLMs.
3. Privacy-Preserving Solution: Validated that models can be run locally on hospital infrastructure, addressing data sovereignty and privacy concerns.
4. Multilingual Capability: Showed that SLMs can effectively handle French clinical text, even when prompted in English.

5. Significance and Implications

• Rare Disease Research: The approach addresses the "data scarcity" bottleneck by unlocking longitudinal data hidden in free-text notes, enabling more accurate prognosis modeling and clinical trial design for rare conditions.
• Clinical Utility: By increasing the density of biomarker data points, the method allows for better reconstruction of disease trajectories (e.g., kidney function decline) which is critical for nephrology.
• Scalability: The use of SLMs makes this solution feasible for resource-constrained hospital environments, offering a cost-effective alternative to large cloud-based models.
• Future Directions: The authors propose extending this framework to other biomarkers (proteinuria, electrolytes) and other chronic diseases, as well as improving subject attribution (distinguishing patient vs. family data) through advanced prompting or fine-tuning.

Conclusion: The study confirms that Small Language Models, when coupled with structured prompting and post-processing, are a viable, privacy-preserving, and highly effective tool for transforming unstructured clinical narratives into structured longitudinal data for rare disease research.