Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset

This study demonstrates that fine-tuning the LLaMA 3.1-8B model on a small, validated corpus of simulated Finnish clinical conversations effectively supports medical transcription by achieving high semantic similarity despite low n-gram overlap, thereby validating a privacy-oriented approach for reducing physician burnout in low-resource languages.

The Gist

Imagine a busy doctor's office. The doctor is great at diagnosing patients and explaining treatments, but they are drowning in paperwork. Every time they talk to a patient, they have to stop, turn to a computer, and type out a long, detailed report. This takes time away from the patient, causes stress for the doctor, and can lead to mistakes.

This paper is about building a digital assistant (an AI) that can listen to the doctor and patient, understand what they are saying, and automatically write the medical report for them. But there's a catch: this assistant needs to speak Finnish, a language that is notoriously difficult for computers because its words change shape constantly (like a chameleon changing colors), and there aren't many examples of Finnish medical conversations available to teach the AI.

Here is the story of how the researchers built and tested this assistant, explained simply:

1. The Problem: The "Paperwork Mountain"

Doctors are burning out because they spend more time typing than talking to patients. In English-speaking countries, there are many smart AI tools to help with this. But in Finland, the tools are scarce. Finnish is a "low-resource" language for AI, meaning there isn't a massive library of Finnish medical data to teach the computer. Plus, medical talk is full of jargon, and Finnish grammar is complex.

2. The Solution: Teaching a Smart Robot (Fine-Tuning)

The researchers didn't build a robot from scratch. Instead, they took a very smart, general-purpose AI brain called LLaMA 3.1 (think of it as a brilliant student who knows everything about the world but has never studied medicine or Finnish specifically).

They decided to give this student a "specialized crash course" in Finnish medical transcription. This process is called Fine-Tuning.

• The Classroom: They created a small, high-quality "textbook" for the AI. Since they couldn't find enough real medical recordings, they had students from nursing and medical schools act out fake doctor-patient conversations.
• The Recording: They recorded these role-plays as audio (MP3) and wrote down what was supposed to be said (the "correct" text).
• The Lesson: They fed this small dataset to the AI, teaching it: "When you hear a patient say X in Finnish, you should write a medical report that looks like Y."

3. The Challenge: The "Tiny Library"

Usually, to teach an AI well, you need a library with millions of books. Here, the researchers only had seven short conversations. It's like trying to teach someone to be a master chef by only letting them cook seven meals.

To make sure the AI actually learned and didn't just memorize those seven meals, they used a clever trick called 7-Fold Cross-Validation.

• The Analogy: Imagine you have 7 puzzle pieces. You give the AI 6 pieces to study and ask it to guess the 7th. Then you swap them: study 6 different ones, guess a different one. You do this seven times. This ensures the AI truly understands the rules of Finnish medical writing, not just the specific words in the examples.

4. The Test: How Good is the Assistant?

After the training, they tested the AI. They didn't just ask, "Is it right?" They used three different "report cards":

1. The Word-Match Test (BLEU): This checks if the AI used the exact same words as a human expert.
   • Result: The score was low (0.12).
   • What it means: The AI didn't copy the human word-for-word. It used different words to say the same thing. In a strict spelling test, it would fail, but in real life, this is often okay.
2. The Sentence-Structure Test (ROUGE-L): This checks if the AI captured the main ideas and the order of events.
   • Result: The score was decent (0.50).
   • What it means: The AI got the general story right. It knew who the patient was and what was wrong, even if the sentence structure was slightly different.
3. The "Meaning" Test (BERTScore): This is the most important one. It asks, "Do these two texts mean the same thing, even if the words are different?"
   • Result: The score was very high (0.82).
   • What it means: This is the big win. The AI understood the meaning perfectly. If a human doctor read the AI's notes, they would understand the patient's condition just as well as if a human had written it.

5. The Conclusion: A Promising Start

The study shows that even with a tiny dataset and a difficult language, you can teach a powerful AI to act as a Digital Scribe for Finnish doctors.

• The Good News: The AI captures the meaning of medical conversations very well. It can turn messy spoken Finnish into clean, structured medical notes.
• The Reality Check: The dataset was very small (only 7 examples), and the AI isn't perfect at using the exact same medical terminology every time.
• The Future: The researchers believe that if they give the AI more data (more "lessons") and maybe combine it with other tools, it could become a real-world tool that saves doctors hours of typing every day, letting them focus on what matters most: the patient.

In a nutshell: They took a smart, general AI, gave it a crash course in Finnish medical talk using a tiny set of practice scripts, and proved that it can understand the meaning of doctor-patient conversations almost as well as a human, paving the way for a future where doctors in Finland can talk freely without worrying about the paperwork.

Technical Summary

1. Problem Statement

The paper addresses the critical issue of physician burnout caused by the administrative burden of Electronic Health Records (EHRs). In modern healthcare, doctors spend excessive time documenting patient interactions rather than engaging with patients. While human medical scribes and general AI tools exist, they face challenges:

• Human Scribes: High costs, scalability issues, and data privacy concerns.
• General AI/ASR: Most Large Language Models (LLMs) and Automatic Speech Recognition (ASR) systems are optimized for English. They struggle with low-resource languages like Finnish, which is morphologically agglutinative (complex word structures) and requires high accuracy for patient safety.
• Data Scarcity: There is a lack of high-quality, validated datasets for Finnish medical transcription, making it difficult to train domain-specific models without violating data privacy regulations (e.g., GDPR).

2. Methodology

The study proposes a pipeline to fine-tune an open-source LLM for Finnish medical transcription using a small, validated dataset.

A. Dataset Creation

• Source: A custom dataset was created by students from Metropolia University of Applied Sciences (nursing, podiatry, paramedic, etc.) acting as healthcare professionals and patients.
• Content: 7 simulated clinical conversations recorded as MP3 audio files with corresponding manual text transcriptions.
• Structure: Data was organized with a specific naming convention (I01-G01-C01) to track iterations, student groups, and clinical cases.
• Preprocessing: Audio was transcribed using Whisper Large-V3, then aligned and converted into a structured JSON format containing speaker roles (doctor/patient) and target documentation fields.

B. Model Selection

• Base Model: LLaMA 3.1-8B (Meta).
• Selection Criteria: The model was chosen for its open-source nature, adaptability, and ability to handle structured clinical documentation. It was preferred over domain-specific models like BioGPT or ClinicalBERT, which lack native Finnish support.
• Platform: Development and training were conducted on the Hugging Face ecosystem.

C. Experimental Setup & Fine-Tuning

• Hardware: CSC Puhti Supercomputer (Finland) using 1x NVIDIA A100 GPU (40GB VRAM), 16 CPU cores, and 128GB RAM.
• Techniques:
  • Quantization: 4-bit quantization to fit the model into GPU memory.
  • Parameter-Efficient Fine-Tuning (PEFT): LoRA (Low-Rank Adaptation) adapters were used to fine-tune the model without updating all parameters.
  • Precision: Half-precision (fp16) training.
  • Configuration: Batch size of 1, 3 epochs, max sequence length of 512 tokens.
• Validation Strategy: Due to the small dataset size (7 samples), the study employed 7-Fold Cross-Validation (Leave-One-Out). In each fold, one conversation served as the test set while the model was trained on the remaining six.

D. Evaluation Metrics

The performance was evaluated against human reference transcripts using three metrics:

1. BLEU: Measures n-gram precision (lexical overlap).
2. ROUGE-L: Measures the longest common subsequence (recall and completeness).
3. BERTScore: Measures semantic similarity using contextual embeddings, crucial for capturing medical meaning despite syntactic variations.

3. Key Contributions

1. Domain-Specific Finnish Model: Successfully demonstrated that an open-source LLM (LLaMA 3.1-8B) can be effectively fine-tuned for Finnish medical transcription despite the language's morphological complexity.
2. Small-Data Viability: Proved that high-quality, validated datasets (even as small as 7 samples) combined with cross-validation can yield stable results for low-resource domains, avoiding overfitting.
3. Privacy-Centric Approach: Utilized open-source models and local supercomputing (CSC Puhti) to ensure data privacy compliance (GDPR), offering a blueprint for secure clinical AI deployment.
4. Comprehensive Evaluation: Provided a multi-metric analysis showing that while lexical overlap (BLEU) is low, semantic similarity (BERTScore) is high, indicating the model captures the clinical meaning accurately even if the exact wording differs.

4. Results

The model was evaluated across the 7-fold cross-validation:

• Average BLEU Score: 0.1214 (Indicates low n-gram overlap, typical for low-resource languages and complex medical terminology).
• Average ROUGE-L Score: 0.4982 (Indicates moderate recall and reasonable structural overlap).
• Average BERTScore (F1): 0.8230 (Indicates high semantic similarity, suggesting the model successfully captures the clinical intent and meaning).

Interpretation: The results suggest that while the model may not generate word-for-word identical transcripts to the reference, it produces clinically equivalent documentation with high semantic integrity.

5. Significance and Future Work

• Significance: This study validates the feasibility of using fine-tuned LLMs to reduce administrative burdens in Finnish healthcare. It offers a cost-effective, privacy-compliant alternative to human scribes and proprietary English-centric AI tools.
• Limitations: The dataset is very small (7 samples), and the base model (LLaMA 3.1) is not natively pre-trained on Finnish, which may limit the capture of deep linguistic nuances.
• Future Directions:
  • Expanding the dataset with more authentic or synthetic Finnish clinical data.
  • Comparing LLaMA 3.1 with other models (e.g., FinBERT, SiloGen).
  • Implementing Retrieval-Augmented Generation (RAG) to reduce hallucinations.
  • Incorporating human-in-the-loop validation with medical professionals to assess clinical utility beyond automated metrics.

In conclusion, the paper establishes a foundational framework for developing privacy-oriented, domain-specific medical scribes for low-resource languages, proving that semantic accuracy can be achieved even with limited data through rigorous fine-tuning and cross-validation.