Evaluation and LLM-Guided Learning of ICD Coding Rationales

This paper addresses the lack of systematic evaluation and dedicated training methods for ICD coding rationales by introducing a novel multi-granular dataset to assess faithfulness and plausibility across different rationale types, and subsequently leveraging high-quality LLM-generated rationales as distant supervision to significantly improve the plausibility of rationale generation in both large language models and specialized student models.

The Gist

Imagine you are a doctor trying to understand why a computer program (an AI) decided a patient has a specific disease. The AI says, "This patient has Type 2 Diabetes." But it doesn't just give you the answer; it needs to show you why. It needs to point to the specific sentences in the patient's medical notes that led to that conclusion.

In the world of medical coding, these "why" explanations are called rationales.

This paper is like a detective story investigating three different ways an AI can give these explanations, and then teaching the AI how to give better ones using a super-smart robot assistant (a Large Language Model, or LLM).

Here is the breakdown in simple terms:

1. The Problem: The AI is a "Black Box"

Currently, hospitals use AI to automatically turn messy doctor's notes into standard codes (like ICD-10 codes) for billing and records. The AI is great at getting the code right, but it's terrible at explaining how it got there.

• The Old Way: The AI used to just highlight random words based on "attention" (like a spotlight that gets a bit fuzzy). It's like a student guessing the answer on a test and then pointing to random words in the textbook to justify it, even if those words aren't actually the reason.
• The Issue: Doctors don't trust the AI if they can't see the real evidence. Plus, the old datasets used to test these explanations were outdated (like using a map from 1990 to navigate a city in 2026).

2. The New Tool: A Fresh, High-Quality Map

The researchers built a brand-new dataset called RD-IV-10.

• The Analogy: Imagine you are training a new chef. Instead of giving them a recipe book with torn pages and missing ingredients (the old data), you give them a pristine, modern cookbook with high-quality photos and step-by-step instructions.
• What they did: They took 150 real patient records and had human medical experts carefully highlight exactly which sentences proved a diagnosis. This created a "Gold Standard" to test if the AI's explanations are actually good.

3. The Three Types of Explanations (The Contest)

The researchers tested three different "students" to see who could give the best explanation:

• Student A (The Entity Linker): This is a robot that just looks for specific medical terms (like "diabetes" or "aspirin") and highlights them.
  • Verdict: It's okay, but it's like a search engine. It finds the words, but it doesn't understand the story or the context.
• Student B (The Attention Model): This is the old-school AI that highlights words based on mathematical weights.
  • Verdict: It's the worst. It often highlights random words that have nothing to do with the diagnosis. It's like a student highlighting the word "the" because it appears a lot, not because it explains the disease.
• Student C (The LLM - Large Language Model): This is a super-smart AI (like a very advanced chatbot) that reads the whole note and writes a summary of why the diagnosis fits.
  • Verdict: The Winner! It generated explanations that humans found the most convincing and logical. It understood the context, not just the keywords.

4. The Breakthrough: Teaching the AI with a "Tutor"

Since the super-smart LLM (Student C) was so good at explaining things, the researchers asked: "Can we use this smart robot to teach the other, simpler AI models how to explain themselves?"

They used a technique called Distant Supervision.

• The Analogy: Imagine a master chef (the LLM) writing down the perfect reasons for a dish. Then, they give those notes to a junior chef (the smaller AI model) and say, "Study these notes and try to learn how to explain your cooking."
• The Result: The junior models got much better at explaining their decisions. They didn't just get the right answer; they learned to point to the right evidence.

5. The "Cheat Sheet" Trick (Few-Shot Prompting)

The researchers also tried a clever trick. When asking the super-smart LLM to write explanations, they gave it a few examples of perfect human-written explanations first.

• The Analogy: It's like showing a student a sample essay before asking them to write their own.
• The Result: The LLM's explanations became even more accurate and human-like. It learned the "style" of a good medical explanation.

The Big Takeaway

This paper proves two main things:

1. Old AI explanations are often nonsense. We can't trust them yet.
2. New AI (LLMs) can act as excellent teachers. By using a super-smart AI to generate "model answers," we can train smaller, faster AI models to not only diagnose patients correctly but also explain their reasoning in a way that human doctors can trust.

In short: They built a better test, found that smart chatbots are the best teachers, and used those chatbots to train medical AIs to finally "show their work" in a way that makes sense to humans.

Technical Summary

1. Problem Statement

International Classification of Diseases (ICD) coding is a critical but labor-intensive process in healthcare, involving the mapping of unstructured Electronic Health Record (EHR) text to standardized codes. While deep learning models have achieved high accuracy in automated ICD coding, their "black box" nature undermines trust and transparency.

Existing research on explainability faces three major gaps:

1. Lack of Modern Datasets: Existing rationale-annotated datasets (e.g., MDACE) rely on outdated ICD-9 codes and suffer from significant distribution shifts when compared to modern ICD-10 benchmarks (MIMIC-IV).
2. Limited Rationale Types: Most studies focus exclusively on attention-based rationales, lacking systematic comparisons with other explanation types (e.g., entity linking or LLM-generated text).
3. Absence of Rationale Learning: Effective training of models to generate rationales is hindered by the scarcity of high-quality, human-annotated rationale datasets.

2. Methodology

A. Dataset Construction: RD-IV-10

The authors constructed a new rationale-annotated dataset, RD-IV-10, based on the MIMIC-IV database with ICD-10 coding.

• Annotation: 150 discharge summaries were annotated by medical professionals.
• Granularity: Unlike previous datasets that often provide a single rationale per code, RD-IV-10 offers multi-granular annotations (words, phrases, sentences) and captures both direct and indirect evidence.
• Alignment: The dataset maintains a high code distribution overlap (93.15% Full, 83.88% Top-50) with the original MIMIC-IV ICD-10 labels, addressing the distribution shift issues found in MDACE.

B. Evaluation Framework

The paper evaluates explainability across two dimensions:

1. Faithfulness (Model-Centric): Measures how well the rationale reflects the model's internal reasoning.
   • Metrics: Sufficiency (performance with only the rationale) and Comprehensiveness (performance drop when the rationale is removed).
   • Models Tested: CAML, LAAT, and PLM-ICD.
2. Plausibility (Human-Centric): Measures how convincing the rationale is to human experts.
   • Comparison: Rationales are compared against human annotations using Exact/Position-Independent Token and Span Matches.
   • Rationale Types Compared:
     • Naive Entity-Level: Derived from SNOMED CT entity linking.
     • Strong LLM-Generated: Extracted by prompting Large Language Models (Gemini 2-Flash, Gemini 1.5-Pro, LLaMA-3.3).
     • Model-Generated: Attention weights from ICD coding models.

C. LLM-Guided Rationale Learning

To overcome the scarcity of human-annotated rationales, the authors propose using LLMs as a source of "distant supervision."

1. Weak Supervision: LLMs (specifically Gemini 2-Flash) generate rationales for the entire MIMIC-IV dataset.
2. Learning Approaches:
   • Multi-Objective Learning: Jointly optimizes ICD coding loss and a rationale generation loss (minimizing the discrepancy between attention weights and LLM-generated masks).
   • NER Formulation: Reformulates the task as Named Entity Recognition, where rationales are treated as entities and ICD codes as class labels.
3. Few-Shot Prompting: The authors incorporate a small number of human-annotated examples from RD-IV-10 into the LLM prompts to improve the quality of the generated weak labels, which in turn improves the training of student models.

3. Key Contributions

1. RD-IV-10 Dataset: A high-quality, modern rationale dataset for ICD-10 coding that provides richer, multi-granular annotations compared to existing resources.
2. Comprehensive Rationale Evaluation: A systematic comparison showing that LLM-generated rationales (Gemini 2-Flash) significantly outperform both naive entity linking and traditional attention-based model rationales in terms of plausibility.
3. LLM-Guided Learning Framework: Demonstration that LLM-generated rationales can serve as effective distant supervision signals.
4. Few-Shot Enhancement: Proof that incorporating few-shot human examples into LLM prompts significantly boosts both the quality of generated rationales and the performance of rationale learning models.

4. Key Results

Faithfulness

• PLM-ICD (Transformer-based) demonstrated the best faithfulness, requiring fewer tokens to maintain prediction accuracy compared to CAML and LAAT.
• However, even the best attention-based models showed low faithfulness when compared to the full document, suggesting attention mechanisms are not always reliable indicators of model reasoning.

Plausibility

• LLM Superiority: Gemini 2-Flash achieved the highest plausibility scores (e.g., 37.3% PI Token Match), significantly outperforming entity linking (6.1%) and model-generated rationales (<9%).
• Model Limitations: Attention-based rationales (CAML, LAAT, PLM-ICD) had extremely low plausibility (F1 < 1%), indicating they do not align with human explanations.
• Few-Shot Impact: Adding few-shot human examples to LLM prompts improved plausibility scores by an average of ~40-50% across metrics.

Rationale Learning Performance

• Multi-Objective Learning: Improved plausibility by ~1% without degrading ICD coding accuracy.
• NER Formulation:
  • Achieved the highest span-level plausibility, even surpassing the teacher model (Gemini 2-Flash).
  • Trade-off: While it excelled at extracting plausible rationales, it suffered a drop in overall ICD coding accuracy (F1-macro dropped from 68.18% to 53.46%) because the model optimized for entity recognition rather than document classification.
• Few-Shot in Learning: Training NER models on LLM-generated rationales enhanced by few-shot prompts yielded further improvements in plausibility (average gain of ~6% in F1).

5. Significance and Conclusion

This paper fundamentally shifts the paradigm for ICD coding explainability by:

1. Validating LLMs as Rationale Generators: It establishes that modern LLMs can generate high-quality, human-aligned rationales, serving as a scalable alternative to expensive manual annotation.
2. Bridging the Data Gap: It provides a solution to the lack of rationale-annotated data for modern ICD-10 systems, enabling future research in explainable AI for healthcare.
3. Practical Application: The proposed LLM-guided learning approaches allow for the development of models that are not only accurate but also capable of providing convincing, human-understandable explanations, thereby fostering trust in AI-driven clinical decision support systems.

The authors conclude that while attention mechanisms are insufficient for explainability, a combination of LLM-generated weak supervision and few-shot human prompting offers a robust pathway to training models that generate both accurate codes and plausible rationales.