💬 NLP

Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations

This paper investigates the fine-tuning of the Finnish BERT model on unlabeled medical text to improve downstream classification performance and explores whether the geometric changes in embeddings during this process can predict the benefits of domain-specific pre-training in data-scarce healthcare scenarios.

Original authors: Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read student named FinBERT. This student has already read almost everything available in the Finnish language: news articles, Wikipedia, internet forums, and subtitles. They are very smart, but they haven't studied medicine yet.

Now, imagine you need this student to help you read and understand pathology reports (complex medical documents about tissue samples) to help doctors diagnose diseases. But there's a catch: you don't have enough labeled examples (answers) to teach them directly, and getting those labels takes months of waiting for expert doctors to review them.

The Big Question: Can we give the student a "crash course" on medical text before we even have the answers? And more importantly, can we tell during that crash course if it's actually helping, so we don't waste time?

This paper is the story of how the researchers tried to answer that question.

1. The "Cram Session" (Domain Fine-Tuning)

The researchers decided to give FinBERT a "cram session" using a pile of unlabeled medical texts. In the world of AI, this is called Domain Fine-Tuning.

Think of it like this:

The Base Model: FinBERT is like a generalist who knows the language perfectly but doesn't know medical jargon.
The Cram Session: They feed the model thousands of medical reports and ask it to guess missing words (a game called "Masked Language Modeling").
The Goal: To see if the model "wakes up" to the specific nuances of medical language.

2. The "Stress Test" (Watching the Loss Curve)

How do you know if the cram session is working? You can't wait for the final exam (the classification task) because you don't have the answers yet. So, the researchers watched the student's "stress levels" during the study session.

In AI terms, this is called the Training Loss.

The "Familiar" Texts: When they fed the model texts it already knew (like news from YLE), the "stress level" barely changed. It was like reading a book you've already memorized; nothing new is learned.
The "New" Texts: When they fed it histopathology (medical) reports, the stress level dropped drastically. This was a huge signal! It meant the model was struggling at first, then suddenly "got it," and learned a massive amount of new patterns.

The Analogy: Imagine a musician practicing scales. If they play a song they've played a thousand times, their heart rate stays steady. If they start learning a completely new, difficult genre (like Jazz), their heart rate spikes and then settles into a new rhythm. That spike and drop told the researchers: "Yes, this new data is teaching the model something important!"

3. The "Internal Map" (Embedding Geometry)

The researchers didn't just look at the stress levels; they also looked at how the student's brain changed.

They measured the "shape" of the student's understanding (called embeddings).

Before the Cram: The student's thoughts on medical words were scattered and fuzzy.
After the Cram: The thoughts became organized. Similar medical concepts (like "tumor" and "cancer") started grouping together tightly, while unrelated things moved apart.

They used fancy math tools (like CKA and Isotropy) to measure this. Think of it like looking at a messy room before and after cleaning.

Messy Room: Everything is scattered.
Clean Room: Books are on shelves, clothes in drawers.
The Finding: The more the "room" organized itself (became more isotropic or evenly distributed), the better the student performed on the final test.

4. The Final Exam (Downstream Classification)

Finally, they tested the student. They gave them a small set of medical reports with labels (the exam) and asked the model to classify them.

Result: The model that had the "cram session" on medical text did significantly better than the one that hadn't.
The Surprise: The model that had the biggest drop in "stress levels" (loss) and the biggest reorganization of its "brain" (embeddings) was the one that got the highest score.

The "Magic Crystal Ball"

The most exciting part of this paper is the idea of a predictive crystal ball.

Usually, in healthcare AI, you have to wait months to get labeled data to know if your model is any good. This paper suggests that you don't have to wait. By watching how the model learns during the cram session (the shape of the loss curve and the reorganization of its internal map), you can predict beforehand if the model will be useful for the final task.

In Simple Terms:
If you see the model's "stress" drop quickly and its internal "filing system" get organized, you can say with confidence: "Great! This model is learning the medical language, and it's going to be a star doctor's assistant."

Why This Matters

This is a game-changer for low-resource languages (like Finnish) and specialized fields (like medicine).

No More Guessing: You don't need to wait for expensive, slow labeling to know if your data is good.
Efficiency: You can use the "waiting time" to run these cram sessions and predict success.
Safety: It helps build better AI tools for healthcare without needing massive amounts of private patient data immediately.

The Bottom Line:
The researchers proved that by watching how an AI learns new text, we can predict how well it will perform on a difficult task, even before we have the answers. It's like being able to tell a student will ace the exam just by watching how quickly they organize their notes during study hall.

1. Problem Statement

The paper addresses a critical bottleneck in healthcare AI: the scarcity of labeled data and the long delays associated with acquiring expert medical labels (e.g., for histopathological reports).

The Challenge: In low-resource language settings (specifically Finnish), acquiring large, labeled datasets for specific medical domains is difficult due to privacy regulations and the need for specialized medical expertise.
The Question: Can researchers utilize the "wait time" before labels are available to perform Domain Fine-Tuning (DFT) on unlabeled data, and can they predict whether this DFT will benefit a downstream classification task solely by observing training-time signals (e.g., loss curves, embedding geometry) without access to ground-truth labels?

2. Methodology

2.1 Models and Data

Base Model: The study uses FinBERT (TurkuNLP/bert-base-finnish-cased-v1), a BERT model pre-trained on general Finnish text (news, discussions, web crawls) but not specifically on medical data.
Datasets: The authors curated a diverse set of Finnish text corpora to test DFT:
- Medical: Histopathological reports from the Central Finland Biobank (unlabeled).
- Familiar Domains: YLE (news), Wikipedia (sampled in 2025, post-dating FinBERT's 2019 pre-training).
- Novel/Partial Domains: FinLex (legal), Eduskunta (parliamentary), Gutenberg, Subtitles, Webcrawl.
Data Split: Data was split into sets A (DFT training), B (DFT testing/embedding analysis + classification training), and C (classification testing).

2.2 Training Procedure

Domain Fine-Tuning (DFT): The base FinBERT was fine-tuned using the Masked Language Modeling (MLM) task on the unlabeled "Set A" of each domain.
Downstream Classification: A simple classifier (Logistic Regression or k-Nearest Neighbors) was trained on the [CLS] embeddings of "Set B" to perform a classification task.
- Note: The goal was not to optimize classification accuracy but to measure the relative improvement (differential) provided by DFT compared to the base model.

2.3 Measurement and Analysis Techniques

To correlate DFT training signals with downstream performance, the authors analyzed:

Training Dynamics: Tracking training and evaluation loss curves over epochs.
Embedding Geometry: Comparing the [CLS] embedding vectors of the base model vs. the fine-tuned model across all 12 transformer layers using:
- Centered Kernel Alignment (CKA): Measures similarity in geometric structure.
- Procrustes Analysis: Measures optimal alignment between shapes.
- Representational Similarity Analysis (RSA): Compares correlation structures.
Internal Geometry Metrics:
- Isotropy: Measured via Partition Function $Z(\tau)$ and Effective Rank to determine how uniformly embeddings occupy the latent space.
- Clustering: K-means clustering metrics (Silhouette, ARI, NMI) to assess semantic grouping.

3. Key Contributions

Empirical Validation of Train-Time Predictors: The paper demonstrates that specific signals observed during DFT (without labels) can predict the magnitude of downstream classification improvement.
Loss Curve Analysis: It establishes that the magnitude of loss drop during DFT is a strong indicator of domain novelty.
- Observation: Datasets the model was already familiar with (YLE) showed flat loss curves. Novel domains (Histopathology, FinLex) showed drastic loss reductions.
Embedding Isotropy as a Proxy: The study identifies that an increase in embedding isotropy (embeddings becoming more evenly distributed in the latent space) and changes in effective rank correlate positively with downstream classification performance.
Layer-Wise Sensitivity: The authors found that changes in the lower layers (Layers 1–3) of the transformer, measured via CKA, correlate more strongly with downstream success than changes in the final layers.
Low-Resource Language Context: The work highlights the necessity of domain-specific adaptation for Finnish, showing that multilingual models or translation-based approaches are inferior to native domain adaptation.

4. Key Results

Loss Dynamics vs. Familiarity:
- YLE Datasets: Showed minimal loss change, confirming the model had already "seen" this data during pre-training. Consequently, DFT provided negligible or slightly negative benefits to downstream tasks.
- Histopathology & FinLex: Showed massive loss drops, indicating the model was learning new patterns. These domains showed the most significant downstream improvements.
Correlation Findings:
- Strong Positive Correlation: There is a strong correlation between Train Loss Relative Improvement and downstream classification performance (e.g., Logistic Regression F1 score).
- Isotropy: A positive delta in Isotropy Effective Rank (embeddings becoming less collapsed and more isotropic) correlates with better classification accuracy.
- Clustering: Improvements in clustering metrics (ARI, NMI) during DFT predict better downstream task performance.
- Layer Specificity: Large changes in the lower transformer layers (CKA) correlate with better downstream results, whereas changes in the final layers do not show a strong correlation.
Quantitative Example: In the Histopathology domain, the "Train Loss Relative Improvement" showed a correlation coefficient ( $r$ ) of 0.961 with Logistic Regression F1 improvement (500 samples).

5. Significance and Future Work

Practical Application for Healthcare AI: The findings offer a practical workflow for healthcare institutions. If labeled data is unavailable, practitioners can run a short DFT on unlabeled data. By monitoring the loss curve drop and embedding isotropy, they can predict if the investment in DFT will yield a useful model for future classification tasks, saving time and compute resources.
Theoretical Implications: The paper suggests that "capped power laws" for DFT need to be developed, accounting for the fact that the model has already seen a significant portion of the data during pre-training.
Limitations: The study is based on a small number of domains (9) and uses simple classifiers. The authors acknowledge that some correlations might be spurious and call for replication studies.
Future Directions: The authors propose extending these geometric analyses to model weights and attention patterns, and building a secure system to apply these methods to real-world healthcare datasets.

In summary, the paper provides a robust, data-driven argument that training-time signals (specifically loss reduction and embedding geometry changes) serve as reliable proxies for predicting the utility of domain fine-tuning in low-resource, unlabeled medical scenarios.