Can Reasoning LLMs Enhance Clinical Document Classification?

Technical Summary: Can Reasoning LLMs Enhance Clinical Document Classification?

Problem Statement
Clinical document classification is a critical yet challenging process in healthcare, requiring the conversion of unstructured medical narratives into standardized ICD-10 diagnoses. This task is hindered by the complexity of medical language, including domain-specific terminology, abbreviations, and varying institutional writing styles. Furthermore, strict privacy regulations and a scarcity of high-quality annotated datasets impede the development of robust models. While Large Language Models (LLMs) have emerged as transformative tools for natural language understanding, a significant research gap exists regarding the systematic quantification of whether reasoning-enhanced LLMs offer tangible benefits over standard non-reasoning LLMs in the specific context of clinical coding. It remains unclear if the additional computational complexity of reasoning architectures translates to improved diagnostic coding accuracy and consistency.

Methodology
The study evaluated eight LLMs—four reasoning models (Qwen QWQ, Deepseek Reasoner, GPT o3 Mini, Gemini 2.0 Flash Thinking) and four non-reasoning models (Llama 3.3, GPT 4o Mini, Gemini 2.0 Flash, Deepseek Chat)—on a balanced subset of the MIMIC-IV dataset.

• Dataset Construction: The dataset comprised 3,000 discharge summaries, evenly distributed across the top 10 ICD-10 codes (derived from both primary and secondary diagnoses). For each code, 150 positive samples (confirmed diagnosis) and 150 negative samples (no diagnosis) were selected to ensure class balance.
• Preprocessing: Unstructured clinical narratives were processed using cTAKES (clinical Text Analysis and Knowledge Extraction System). This tool tokenized the text into structured SNOMED codes, capturing contextual nuances such as affirmation or negation, thereby reducing document length and improving processing efficiency.
• Experimental Protocol: Each model was tasked with classifying the 3,000 tokenized summaries into "Yes" or "No" for specific ICD-10 codes. To ensure robustness, each model was executed in three independent experimental runs. Final predictions were determined via majority voting across the three runs.
• Evaluation Metrics: Performance was assessed using Accuracy, F1 Score, and Consistency (the proportion of summaries receiving identical classifications across all three runs).

Key Contributions

1. Comparative Analysis of Reasoning vs. Non-Reasoning: The study provides a direct empirical comparison between reasoning and non-reasoning LLM architectures in a clinical coding context, addressing a gap in the literature regarding the specific utility of reasoning capabilities for ICD-10 classification.
2. Consistency vs. Accuracy Trade-off: The research identifies a distinct trade-off where reasoning models achieve higher accuracy but exhibit lower stability, while non-reasoning models demonstrate superior consistency.
3. Granular Performance Mapping: The study categorizes ICD-10 codes based on model performance, distinguishing between well-defined conditions (where models perform well) and abstract diagnostic categories (where performance varies significantly).

Results

• Overall Performance: Reasoning models outperformed non-reasoning models in classification accuracy and F1 scores. Reasoning models achieved an average accuracy of 71% and an F1 score of 67%, compared to 68% accuracy and 60% F1 for non-reasoning models.
• Top Performers: Gemini 2.0 Flash Thinking (reasoning) achieved the highest accuracy (75%) and F1 score (76%). Conversely, GPT 4o Mini (non-reasoning) recorded the lowest performance (64% accuracy, 47% F1).
• Consistency: Non-reasoning models demonstrated higher stability, with an average consistency of 91%, compared to 84% for reasoning models. Models like GPT o3 Mini and Deepseek Chat showed high consistency (95%), whereas reasoning models like Qwen QWQ and Gemini 2.0 Flash Thinking showed lower consistency (78%).
• Code-Specific Variations:
  • Well-defined conditions (e.g., Sepsis/A41, Myocardial Infarction/I21): All models showed strong consistency and F1 scores.
  • Variable Performance (e.g., E78, I13): Consistency remained high, but F1 scores varied widely across models.
  • Abstract Categories (e.g., Z79, Z87): Both consistency and F1 scores fluctuated significantly. Notably, some models (e.g., GPT o3 Mini) achieved 100% consistency but 0% F1, indicating they consistently misclassified all records as negative.

Significance and Claims
The paper claims that while reasoning LLMs offer superior classification accuracy and F1 scores—particularly in complex clinical cases—they introduce greater variability in results compared to the more stable non-reasoning models. The authors suggest that the optimal approach for automated clinical coding may be a hybrid strategy that leverages the high accuracy of reasoning models alongside the reliability of non-reasoning models.

The study concludes that reasoning capabilities enhance the ability to capture complex clinical relationships but do not guarantee stability. Therefore, future deployment in healthcare settings requires balancing accuracy with reliability. The authors propose that future work should focus on multi-label classification, domain-specific fine-tuning, and ensemble modeling to further enhance performance and generalizability in real-world applications.