💬 NLP

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

This paper demonstrates that locally deployable, open-source large language models can accurately and efficiently retrieve patient-specific information from Finnish electronic health records using natural language queries, achieving high accuracy and consistency while highlighting the necessity for human oversight to mitigate clinically significant errors.

Original authors: Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten, Kimmo Kaski, Otso Arponen

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten, Kimmo Kaski, Otso Arponen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a doctor's office as a massive, chaotic library. Inside this library, every patient has a "life story" written in thousands of pages of notes, lab reports, and scan results. These notes are written in a mix of languages (mostly Finnish, with some medical Latin and English), and they span decades.

When a doctor needs to answer a simple question like, "When was this patient's breast cancer first found?", they usually have to act like a detective, manually flipping through hundreds of pages, searching for a needle in a haystack. It's slow, tiring, and easy to miss a crucial detail.

This paper is about building a super-smart, privacy-focused robot librarian to help with this task.

Here is the breakdown of their experiment in simple terms:

1. The Problem: The "Needle in a Haystack"

Doctors are drowning in data. Electronic Health Records (EHRs) are huge. Finding a specific fact (like a diagnosis date) is like trying to find a specific sentence in a book that is 500 pages long, written in a foreign language, and buried inside a stack of 20 other books. Doing this manually is slow and prone to human error.

2. The Solution: The "Local Brain"

The researchers built a system using Large Language Models (LLMs). Think of these models as super-intelligent brains that have read the entire internet and can understand human language.

The Twist: Usually, these AI brains live in the "cloud" (on big servers far away). But for medical records, you can't send private patient data to the cloud. It's a privacy nightmare.
The Fix: They tested running these AI brains locally on the hospital's own computers. This means the patient's data never leaves the building. It stays in the "house."

3. The Test: The "Exam"

They took the medical records of 183 breast cancer patients and created a test. They asked the AI 1,664 specific questions, such as:

"What year was the cancer first detected?"
"Was the tumor on the left or right side?"
"Did the patient get chemotherapy before surgery?"

They tested different "sizes" of AI brains, ranging from small (4 billion parameters) to massive (70 billion parameters).

4. The Results: The "Star Students"

The results were surprisingly good!

The Big Brains Win (Mostly): The largest AI models (like Llama-3.1-70B) got about 95% accuracy. They were like top-tier students who could read the whole book and instantly find the answer.
The Underdog: A medium-sized model (Qwen3-30B) performed just as well as the giants but was much lighter and faster. It's like a compact sports car that goes just as fast as a massive truck but uses less fuel.
The "Specialized" Trap: They tried models that were specifically trained on medical texts or the Finnish language. Surprisingly, these didn't do much better than the general-purpose models. It turns out, a general smart brain can learn the medical jargon just fine if you give it the whole book to read.

5. The Catch: "Hallucinations" and "Confidence"

Even the best students make mistakes.

The 5% Error: About 5% of the time, the AI got it wrong. In a medical setting, even a small error can be dangerous. The researchers found that about 3% of the errors were "clinically significant" (meaning they could actually hurt a patient's treatment plan).
The Confidence Trap: Sometimes, the AI would say an answer with 100% confidence, even if it was wrong. This is like a student who is absolutely sure they spelled a word right, but they actually spelled it wrong. The researchers found that the AI's "confidence meter" isn't always trustworthy.
The "Wording" Issue: If you asked the same question in two slightly different ways (e.g., "When was cancer found?" vs. "What year did the cancer appear?"), the AI sometimes gave two different answers. This is a bit like a human getting confused by how a question is phrased.

6. The "Quantization" Trick: Shrinking the Elephant

Big AI models are heavy; they require massive computer memory (like trying to fit an elephant in a sedan).

The researchers tried quantization, which is like compressing a high-resolution photo into a smaller file.
They found they could shrink the models down to 4-bit precision (making them much smaller and faster) without losing much accuracy. This is like taking a giant elephant and fitting it into a compact car without breaking the car. This makes it possible to run these powerful tools on standard hospital computers.

The Bottom Line

This paper proves that we can build a secure, local AI assistant that helps doctors find critical information in patient records quickly and accurately.

However, it's not ready to replace the doctor yet.
Think of this AI as a super-fast research assistant. It can scan the library in seconds and hand the doctor a draft answer. But the doctor must still read that draft, check the facts, and make the final decision. The AI is a powerful tool to reduce the doctor's workload, but it needs human oversight to ensure patient safety.

In short: We have a robot librarian that is incredibly fast and smart, but we still need a human librarian to double-check its work before we hand the book to the patient.

1. Problem Statement

Modern clinical practice faces an "information paradox": Electronic Health Records (EHRs) contain vast amounts of longitudinal patient data, but the volume, fragmentation, and heterogeneity of this data create a significant cognitive burden for clinicians.

The Challenge: Clinicians must manually review years of documentation (radiology, pathology, surgical summaries, medication records) to answer specific patient questions. This process is time-consuming, prone to error, and often relies on inefficient keyword searches that fail to capture temporal or contextual relationships.
The Gap: While Large Language Models (LLMs) have shown promise in medical QA, most prior work relies on synthetic data, general-domain benchmarks, or English-only datasets. There is a lack of rigorous evaluation on real-world, multilingual (Finnish), longitudinal EHRs using locally deployable, retrieval-free models. Furthermore, the impact of model scale, quantization, and calibration on clinical safety in offline hospital environments remains under-explored.

2. Methodology

The authors developed a Clinical Contextual Question Answering (CCQA) framework designed to operate entirely offline within institutional boundaries to ensure data privacy.

Dataset:
- Source: 183 pseudonymized patient records from the Wellbeing Services County of Pirkanmaa, Finland.
- Content: Longitudinal records (median 14 years) containing heterogeneous data (coded entries and free-text narratives) primarily in Finnish, with embedded English and Latin medical terminology.
- Annotation: 1,664 expert-annotated question-answer pairs covering 21 clinical variables (e.g., tumor laterality, receptor status, treatment history, diagnosis year).
- Variants: Each question was formulated in two semantically equivalent versions (V1 and V2) to test robustness.
Models Evaluated:
- 16 instruction-tuned open-source LLMs ranging from 4B to 70B parameters.
- Families: Google (Gemma, MedGemma), Meta (Llama-3.1/3.3), DeepSeek (R1-Distill), Qwen, and LumiOpen (Finnish-specialized Poro-2).
- Constraints: Experiments were conducted on hardware with a 160 GB GPU memory limit. Two models (Llama-4-Scout, GPT-OSS) were excluded due to memory constraints.
Experimental Setup:
- Retrieval-Free: Models processed the entire patient EHR as a single context window without prior retrieval or filtering.
- Tasks: Evaluated in two modes: Free-text generation and Multiple-choice selection (to enable calibration analysis).
- Strategies: Tested low-precision quantization (4-bit, 8-bit), prompt conditioning (one-shot, context), and consistency across question paraphrasing.
- Safety Evaluation: A medical oncologist reviewed all incorrect responses and inconsistent paraphrase pairs to classify errors as "clinically significant" or "insignificant."

3. Key Contributions

First Large-Scale Benchmark on Finnish EHRs: Provides a rigorous evaluation of LLMs on real-world, non-English (Finnish) longitudinal oncology data, addressing a major gap in multilingual clinical NLP.
Retrieval-Free Long-Context Processing: Demonstrates that processing full patient histories end-to-end is feasible and effective, avoiding the pitfalls of retrieval-augmented generation (RAG) where relevant historical context might be missed.
Holistic Deployment Analysis: Moves beyond simple accuracy to evaluate calibration (confidence vs. accuracy), consistency (robustness to phrasing), latency, memory footprint, and quantization effects.
Clinical Safety Assessment: Quantifies the rate of clinically significant errors and highlights the risks of semantic inconsistency (where paraphrasing a question changes the answer from correct to dangerous).

4. Key Results

Accuracy & Model Scale:
- Top-performing models achieved ~95% accuracy.
- Llama-3.1-70B achieved the highest accuracy (95.3% on V1) with 97.3% consistency across question variants.
- Qwen3-30B-A3B-2507 (30B parameters) matched the performance of the 70B models (95.1% accuracy) while requiring significantly less memory and inference time, suggesting architectural efficiency matters more than raw parameter count.
- Reasoning Models: DeepSeek R1-Distill models showed comparable accuracy at large scales but suffered from much higher latency (up to 153s per query) due to intermediate reasoning traces.
Specialization vs. Generalization:
- No Advantage for Specialization: Models specialized for medical text (MedGemma) or the Finnish language (Llama-Poro-2) did not outperform general-purpose instruction-tuned models. Generalist models handled the mixed Finnish/English/Latin terminology effectively.
- Context Window Limits: Models with limited context windows (e.g., Poro-2 with 8k tokens) saw accuracy plummet to near zero on longer records, whereas long-context models maintained high performance.
Quantization & Efficiency:
- 4-bit Quantization: Successfully reduced GPU memory usage by ~60-70% (e.g., Llama-3.1-70B dropped from ~148 GB to ~53 GB) with only a modest accuracy drop (95.3% $\to$ 92.1%).
- 8-bit Quantization: Caused severe accuracy degradation for some architectures (e.g., Llama-3.1-70B dropped to 65.1%), indicating that quantization effects are architecture-dependent.
Calibration & Consistency:
- Calibration Variability: High accuracy did not guarantee well-calibrated confidence. Some models were overconfident in incorrect answers.
- Inconsistency: While top models were consistent (>97%), a small fraction of cases (0.96% of all responses) showed that semantically equivalent questions produced discordant answers, with one being correct and the other a clinically significant error.
Clinical Safety:
- Expert review of the best model (Llama-3.1-70B) revealed that 2.9% of all outputs contained clinically significant errors.
- This highlights that high aggregate accuracy masks the severity of specific failures in a clinical setting.

5. Significance and Conclusion

This study demonstrates that locally deployable, open-source LLMs can accurately retrieve patient-specific information from complex, multilingual, longitudinal EHRs without transmitting data outside the hospital.

Feasibility: It is possible to run these models on-site using quantized 4-bit configurations on standard hospital hardware (160 GB VRAM), ensuring data privacy.
Deployment Guidelines:
- Model Selection: Mid-sized, efficient models (like Qwen3-30B) may offer the best trade-off between performance, speed, and resource usage.
- Safety First: Accuracy alone is insufficient. Systems must be validated for consistency (robustness to phrasing) and calibration.
- Human-in-the-Loop: Due to the presence of clinically significant errors (even in high-performing models), these systems should serve as decision support tools for clinicians, not autonomous decision-makers.
Future Work: The authors call for prospective, multi-site validation and workflow-integrated studies to assess real-world impact on time savings and clinical outcomes.

In summary, the paper provides a critical roadmap for integrating LLMs into clinical workflows, emphasizing that while the technology is ready for retrieval-free information extraction, rigorous safety validation and human oversight remain non-negotiable.