💬 NLP

Determinants of Training Corpus Size for Clinical Text Classification

This study utilizing MIMIC-III data demonstrates that while 600 annotated documents are generally sufficient to achieve near-optimal performance in clinical text classification, the specific learning curve and accuracy are significantly determined by the ratio of strong predictive words to noisy vocabulary rather than corpus size alone.

Original authors: Jaya Chaturvedi, Saniya Deshpande, Chenkai Ma, Robert Cobb, Angus Roberts, Robert Stewart, Daniel Stahl, Diana Shamsutdinova

Published 2026-01-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaya Chaturvedi, Saniya Deshpande, Chenkai Ma, Robert Cobb, Angus Roberts, Robert Stewart, Daniel Stahl, Diana Shamsutdinova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but initially blank-slate, robot how to recognize specific medical conditions just by reading hospital discharge notes. The big question the researchers asked was: "How many notes does the robot actually need to read before it becomes an expert?"

In the past, doctors and researchers often guessed that they needed to manually label anywhere from 200 to 500 documents to get a good result. But they didn't really know why that number was chosen, or if it was enough.

Here is what the study found, explained through simple analogies:

1. The "Smart Starter" vs. The "Blank Slate"

The researchers didn't start with a dumb robot. They used a pre-trained "Large Language Model" (like BERT). Think of this model as a robot that has already read millions of books and knows how language works, but it hasn't learned about specific medical diseases yet.

Because the robot already understands the "grammar" of language, it doesn't need to read a library's worth of notes to learn the basics. It just needs to learn the specific "dialect" of the disease it's looking for.

2. The Magic Number: 600 Notes

The team tested 10 different medical conditions (like diabetes, heart failure, or high blood pressure). They started with tiny groups of notes (100) and kept adding more, all the way up to 10,000.

The Discovery:
For every single condition they tested, once the robot read about 600 notes, it had learned almost everything it was ever going to learn.

The Analogy: Imagine you are trying to learn a new song. You might struggle with the first few verses, but by the time you've practiced 600 times, you know the song perfectly. Practicing 10,000 times doesn't make you 10 times better; it just makes you slightly more consistent. The study found that 600 notes got the robot to 95% of its maximum possible skill.

3. The "Signal" vs. The "Noise"

Why did some tasks get easier faster than others? The researchers looked at the words inside the notes. They found two types of words:

Strong Predictors (The Signal): These are the "smoking guns." For diabetes, words like "insulin," "sugar," or "metformin" are strong signals. They scream, "This is diabetes!"
Noisy Predictors (The Noise): These are words that appear in almost every note but don't help identify the specific disease. Words like "hospital," "patient," "admitted," or "with" are noise. They are like static on a radio.

The Analogy:
Imagine you are trying to find a specific person in a crowded room.

If the room is full of people wearing bright red hats (strong predictors), you find them instantly, even if the room is small.
If the room is full of people wearing grey shirts (noise) and only a few have red hats, you have to look at many more people to be sure you haven't missed anyone.

The study found that if a disease's notes were full of "noise" words, the robot needed more data to filter them out. If the notes were full of "strong" words, the robot learned very quickly.

4. What This Means for Researchers

The paper suggests that researchers don't need to waste time and money labeling thousands of documents if they don't have to.

The "600 Rule": If you are building a tool to classify medical text, aim for about 600 high-quality, labeled examples. That is usually enough to get near-perfect results.
Clean Up Before You Scale: Instead of just grabbing more data, it's often better to clean the data you have. If you can remove the "noise" (irrelevant words) from the notes, the robot learns faster. It's like cleaning a window rather than buying a bigger one; a clean small window lets you see better than a dirty big one.

Summary

The study proves that with modern AI tools, you don't need a massive library of labeled medical notes to build a good classifier. About 600 examples are usually enough, provided the notes contain clear "signal" words. If the notes are messy with "noise," you might need a bit more data, but the key is to focus on the quality of the words, not just the quantity of the documents.

Technical Summary: Determinants of Training Corpus Size for Clinical Text Classification

Problem Statement
Clinical text classification using Natural Language Processing (NLP) requires adequate training data to ensure optimal performance. While Large Language Models (LLMs) offer zero-shot capabilities, fine-tuning remains necessary for specialist cases or when superior accuracy is prioritized. In practice, clinical research studies often annotate between 200 and 500 documents due to time and cost constraints. However, this sample size is frequently chosen without rigorous justification regarding the relationship between corpus size, text vocabulary properties, and model performance. There is a lack of guidelines on the minimum sample size required to build generalizable machine learning models in the healthcare domain, particularly given the scarcity of openly available, clinician-annotated datasets.

Methodology
The study utilized the publicly available MIMIC-III dataset, comprising over 1.2 million de-identified clinical notes (e.g., discharge summaries) from over 50,000 hospital admissions. The authors selected 10 randomly chosen ICD-9 diagnosis codes to represent diverse binary classification tasks.

The experimental pipeline involved the following steps:

Data Preparation: For each diagnosis, hospital discharge notes were extracted. Labels were derived directly from ICD-9 codes to avoid the time-consuming process of manual annotation.
Feature Extraction: Pre-trained BERT embeddings were used to generate pooled-level vector representations for all documents. This approach leverages transfer learning to capture contextual relationships inherent in clinical narratives, distinguishing nuanced meanings that traditional bag-of-words or TF-IDF methods might miss.
Classification: A Random Forest classifier was trained on the BERT embeddings. The authors tested various architectures, including direct BERT fine-tuning and zero-shot approaches (Chat-GPT 3.5, Llama2, DistilBERT), but found the BERT-embedding-plus-Random-Forest pipeline to be superior for this specific task.
Variable Manipulation: Training corpus sizes ( $N$ ) were varied from 100 to 10,000 documents. For each size, models were trained 10 times on random stratified subsets and validated on a large hold-out dataset (5,000 documents) to report averaged performance metrics.
Vocabulary Analysis: To investigate the relationship between text properties and learning curves, the authors employed Lasso logistic regression on bag-of-words embeddings. Words were categorized as "strong predictors" (absolute coefficient > 0.1) or "noisy predictors" (below the threshold). SHAP (SHapley Additive exPlanations) values were also used to quantify feature contributions.

Key Results

Optimal Sample Size: Learning curves varied significantly across the 10 tasks despite identical preprocessing. However, the study found that 600 documents were sufficient to achieve 95% of the performance attainable with 10,000 documents across all tasks.
Vocabulary Impact: There was a clear correlation between vocabulary quality and learning curve steepness.
- Strong Predictors: An increase of 100 strong predictive words corresponded to an approximate 0.04 increase in maximum accuracy.
- Noisy Predictors: An increase of 100 noisy words corresponded to an approximate 0.02 decrease in accuracy at the largest training corpus size.
Feature Stability: SHAP analysis on the diabetes classification task demonstrated that models trained on 1,000 samples successfully identified core diagnostic terms (e.g., 'diabetes', 'metformin') with rankings similar to models trained on 12,000 samples. The additional data primarily served to filter out non-discriminative features rather than capture novel clinical concepts.

Significance and Contributions
The paper provides empirical evidence that 600 gold-standard annotations represent a practical threshold for achieving near-optimal performance in clinical text classification when using pre-trained LLM embeddings combined with tree-based classifiers.

The authors claim that understanding the relationship between vocabulary properties (specifically the ratio of strong to noisy predictors) and learning curve shapes can guide optimal sample size decisions. This suggests that:

Keyword Analysis as a Predictor: A preliminary analysis of keywords in a small annotated sample can predict model performance and the steepness of the learning curve.
Data Quality over Quantity: Data cleaning techniques focusing on document relevance and the isolation of relevant parts may be more valuable than simply increasing the training corpus size.
Resource Efficiency: Researchers can potentially save computational resources and time by targeting the 600-document threshold, provided the vocabulary is sufficiently discriminative.

The study acknowledges limitations, including potential bias from the random selection of ICD codes (which may include co-morbidities) and the focus on binary classification. The authors suggest that future work should explore LLMs as annotation assistants to generate initial labels for human review, further reducing annotation time while maintaining quality.