💬 NLP

Bridging the Version Gap: Multi-version Training Improves ICD Code Prediction, Especially for Rare Codes

This paper demonstrates that training a modified label-wise attention model on a combination of ICD-9 and ICD-10 data significantly improves ICD-10 code prediction performance, particularly for rare codes, by effectively bridging version gaps and addressing the long-tail problem with fewer parameters.

Original authors: Jinghui Liu, Anthony Nguyen

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jinghui Liu, Anthony Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: A Moving Target and a Long Tail

Imagine you are trying to organize a massive library of medical records. To do this, you have to assign every patient a specific "tag" (an ICD code) that describes their illness. There are two huge headaches in this job:

The Dictionary Keeps Changing: The official dictionary of medical tags (called ICD) is constantly being updated. Sometimes, a tag from the old dictionary (ICD-9) doesn't match up perfectly with a tag in the new dictionary (ICD-10). It's like trying to learn a new language while your old textbooks are still on the shelf. Most computer programs are trained to speak only one "version" of the language, so when the rules change, they get confused.
The "Long Tail" Problem: Most patients have common conditions (like the flu or high blood pressure), which are easy to tag. But there are thousands of rare diseases that only a handful of people have. In the world of data, these are the "long tail." Because there are so few examples of these rare diseases, computer models struggle to learn them, often ignoring them entirely.

The Solution: Mixing Old and New Books

The researchers asked a simple question: What if we taught the computer using both the old medical records (ICD-9) and the new ones (ICD-10) at the same time?

Usually, people think mixing these two would be a disaster because the tags don't match perfectly (only about 24% of old tags have a direct match in the new system). It's like trying to teach a student to speak English by mixing in some French words; you'd expect them to get confused.

However, the researchers built a special model called DUALLAAT and tried exactly this. They fed it a "smoothie" made of:

Old records from MIMIC-III (ICD-9)
Newer records from MIMIC-IV (ICD-9)
Newest records from MIMIC-IV (ICD-10)

What Happened? (The Results)

The results were surprisingly good, acting like a "secret sauce" for the computer model:

The Rare Disease Boost: The biggest win was for the rare diseases (the "long tail"). By adding the old data, the model's ability to spot rare ICD-10 codes jumped by 27%. It's as if the old records contained hidden clues about the concepts of the diseases, even if the specific tag names were different. The model learned the meaning of the illness, not just the label.
The Common Disease Boost: The model also got better at handling common diseases, improving its overall accuracy without needing to be made more complex.
Efficiency: Instead of needing three different models (one for each dataset version), they only needed one model to handle all of them. This saved a massive amount of computer memory and training time. It's like having one universal translator instead of needing a different dictionary for every country you visit.

The "Why" (The Secret Ingredient)

The paper suggests that the success didn't come from the tags matching perfectly. Instead, it came from shared clinical meaning.

Think of it like this: Even if the word for "heart attack" changed from "Myocardial Infarction" in the old book to "Acute Myocardial Infarction" in the new book, the description of the patient's symptoms in the notes remained similar. By reading both old and new notes, the model learned to recognize the story of the disease, making it much smarter than a model that only read the new notes.

Summary

The paper proves that you don't have to throw away old medical data when systems update. By mixing old and new records, you can train a single, smarter computer model that is better at spotting rare diseases and doesn't break when the coding system changes. It turns "outdated" data into a valuable training tool.

Problem Statement

The paper addresses two critical challenges in automated International Classification of Diseases (ICD) coding:

Version Evolution: ICD systems undergo continuous updates (e.g., ICD-9 to ICD-10, and the upcoming ICD-11). Models trained on a specific version with a fixed label space struggle to handle these shifts, leading to data distribution mismatches when new versions are adopted.
Long-Tail Distribution: ICD code distributions are heavily skewed. In datasets like MIMIC-IV, rare codes (appearing fewer than 10 times) constitute approximately 70% of the total code set (18,000 out of 26,000 codes). Current models often underperform on these rare codes due to insufficient training samples, creating a bottleneck for robust, implementable systems.

A key question remains unexplored: Is it viable to combine data annotated in different ICD versions (e.g., mixing ICD-9 and ICD-10) to train a single, version-independent model, or does the terminological mismatch introduce noise that harms performance?

Methodology

The authors propose a multi-version training approach using a modified Label-Wise Attention (LWA) model named DUALLAAT (Dual Label-wise Attention).

Model Architecture:
- Dual Encoders: Unlike standard LWA models that only encode clinical notes, DUALLAAT employs two text encoders: one for clinical notes and a separate one for ICD code descriptions.
- Code Embeddings: Instead of using fixed embeddings for codes, the model generates code embeddings dynamically from the textual descriptions of the ICD codes. This allows the model to process codes from different versions (ICD-9 or ICD-10) as long as they are provided as text strings.
- Attention Mechanism: The model uses label-wise attention to align relevant tokens in the clinical note with the specific code description, outputting a probability matrix for $N$ notes and $C$ codes.
- Variants: The authors tested two encoder configurations: DUALLAATcnn (using a basic CNN) and DUALLAAT (using an RNN/GRU).
Data Strategy:
- Datasets: The study utilized three MIMIC-derived datasets: MIMIC-III (ICD-9), MIMIC-IV (ICD-9), and MIMIC-IV (ICD-10).
- Label Space Expansion: The authors re-extracted all codes from MIMIC to include rare codes (frequency < 10), expanding the ICD-10 label space from 8,000 (frequent) to 26,000 (full set).
- Training Procedure: The training sets from all three datasets were aggregated. To handle batch training with mixed versions, the authors implemented a sampler that ensures each batch contains only one ICD version (either ICD-9 or ICD-10) while shuffling batches of different versions during the training process. This stabilizes the label space size while exposing the model to multi-version data.
Baselines:
- PLM-ICD: A strong baseline using in-domain BERT with sliding windows and LWA.
- CoRelation: A model utilizing bipartite graphs to model relations between codes.
- CAML & LAAT: Previous CNN and RNN-based LWA models.

Key Results

The experiments demonstrated that mixing ICD-9 data into ICD-10 training yields significant improvements, particularly for rare codes.

Rare Code Performance (ICD-10):
- Adding ICD-9 data to the training set resulted in a 27.4% relative increase in Micro F1 for 18,000 rare ICD-10 codes (improving from 8.4 to 10.7) compared to training on ICD-10 data alone.
- This improvement was achieved with a significantly smaller model (37M parameters for DUALLAAT vs. 137M for PLM-ICD).
Frequent Code Performance:
- For the 8,000 frequent ICD-10 codes, multi-version training improved the Macro F1 score by approximately 7% (from 27.9 to 29.9).
- The mixed-training DUALLAAT achieved comparable or superior performance to PLM-ICD and CoRelation on full code sets, with notably higher Macro F1 scores (11.2 vs. 6.3 for CoRelation), indicating better robustness across the long tail.
Efficiency:
- Parameter Efficiency: A single DUALLAAT model trained on mixed data can handle multiple ICD versions, whereas separate PLM-ICD models would be required for each version.
- Training Time: The mixed-data DUALLAAT converged in roughly 18 hours (57 hours for the full mixed dataset), significantly faster than the 95 hours required to train three separate PLM-ICD models for the three benchmarks.

Significance and Claims

The paper claims that combining multi-version datasets creates a synergistic effect for ICD coding, challenging the assumption that version mismatches necessarily introduce harmful noise.

Shared Semantics over Direct Overlap: The authors note that only ~24.3% of ICD-9 codes have exact matches in ICD-10. The observed performance gains suggest that the benefits arise from learning shared clinical semantics across versions rather than direct label overlap.
Legacy Data Utilization: The study demonstrates that legacy patient records (e.g., ICD-9) can be effectively leveraged to train more robust models for current systems (ICD-10), potentially solving cold-start issues for future versions (ICD-11).
Data vs. Architecture: The findings suggest that enhancing training data diversity (via multi-version mixing) can be as impactful as architectural improvements, which have been the primary focus of recent literature.
Practical Deployment: The approach offers a more efficient solution for low-resource settings by reducing the number of required model parameters and training time while maintaining high performance across different ICD versions.

The authors conclude that while their study is limited to MIMIC datasets and English ICD-CM, the principles of multi-version training offer a promising direction for scaling ICD coding research and handling ongoing system modifications.