Automatic identification of diagnosis from hospital discharge letters via weakly supervised Natural Language Processing

Technical Summary: Automatic Identification of Diagnosis from Hospital Discharge Letters via Weakly Supervised NLP

Problem Statement
Identifying patient diagnoses from hospital discharge letters is critical for large-scale cohort selection and epidemiological research. However, traditional supervised Natural Language Processing (NLP) approaches require extensive manual annotation of document-level labels, a process that is labor-intensive, costly, and often impractical for large textual datasets. This challenge is exacerbated in clinical settings by the heterogeneity of medical documents, specialized lexicons, ambiguous abbreviations, and the scarcity of manually labeled data, particularly for low-resource languages like Italian. While existing NLP tools are optimized for English, their transferability to Italian is limited due to a lack of domain-specific linguistic resources and annotated corpora. Furthermore, existing weakly supervised methods often rely on structured Electronic Health Record (EHR) variables (e.g., diagnosis codes, medication records) or aggregate data across multiple encounters, which may be incomplete or unavailable in emergency department admissions where discharge summaries are the primary diagnostic source.

Methodology
The authors propose a three-step weakly supervised NLP pipeline designed to automatically identify diagnoses from Italian hospital discharge letters without requiring document-level manual annotation. The pipeline operates solely on free-text clinical documents.

1. Transformer Pre-training for Italian Medical Documents (TPT):
   To address the lack of domain-specific language models for Italian, the authors perform additional pre-training on a transformer-based model (specifically UmBERTo, XLM-RoBERTa, and RemBERT variants) using the European Clinical Case Corpus (E3C). This corpus includes multilingual medical documents (journal articles, admission tests, patient leaflets, thesis abstracts). Non-Italian documents are automatically translated into Italian using the Google Translate API. The model undergoes additional Masked Language Modeling (MLM) pre-training on this aggregated Italian medical corpus to improve domain adaptation before being used in subsequent steps.

2. Semi-Automatic Labelling (SAL):
   This step generates weak labels for a subset of the data through a clustering-based approach:

   • Extraction: Regular expressions are used to extract diagnosis strings from the designated diagnosis sections of discharge letters.
   • Clustering: Extracted strings are normalized and embedded using the domain-adapted transformer encoder. A two-level clustering procedure is applied:
     • First Level: HDBSCAN clusters the high-dimensional embeddings (after PCA dimensionality reduction) to group semantically similar diagnosis strings.
     • Second Level: Each cluster is summarized by extracting representative keywords. These keyword summaries are re-embedded and clustered again to merge semantically related clusters, refining the groupings.
   • Mapping to Weak Labels: Clinicians define diseases of interest using sets of positive and negative keywords. Clusters are mapped to disease labels if their keyword summaries satisfy these definitions. Documents with diagnosis strings assigned to "positive" clusters receive a weak positive label (1); those assigned to other clusters receive a weak negative label (0). Documents with no extracted string or unclustered strings are excluded from the weakly labeled training set.

3. Weakly Supervised Classification (WLC):
   The weakly labeled dataset is used to train a transformer-based binary classifier. The model consists of the domain-adapted encoder with a linear classification layer added on top. The model is trained to minimize binary cross-entropy loss against the weak labels. The input is the full discharge letter (truncated to 512 tokens if necessary), allowing the model to learn from the entire narrative context, not just the extracted diagnosis section.

Key Contributions

• First Application to Italian Discharge Letters: This is the first study addressing diagnosis identification directly from Italian hospital discharge letters using a weakly supervised approach.
• Domain-Adapted Pre-training: The authors demonstrate the efficacy of additional pre-training on a multilingual medical corpus (translated to Italian) to improve performance on clinical text in a low-resource language setting.
• Two-Level Clustering for Weak Label Generation: The pipeline introduces a robust method for generating weak labels by clustering diagnosis strings and summarizing them via keywords, reducing the noise inherent in direct keyword matching on raw text.
• Scalability without Document-Level Annotation: The approach eliminates the need for manual document-level labeling, requiring only clinician input for keyword definitions, which is significantly less demanding than full annotation.

Results
The pipeline was evaluated on two datasets from the Pedianet network in the Veneto Region, Italy:

1. Bronchiolitis Dataset: 33,176 discharge letters (2017–2020) with 335 manually annotated positive cases.
2. Bronchitis Dataset: 3,188 discharge letters (2020–2025) with 97 manually annotated positive cases.

• Performance: The best weakly supervised model (WS-UmBERTo-TPT) achieved an AUROC of 77.68% and an AUPRC of 73.13% on the bronchiolitis dataset. This performance surpassed unsupervised rule-based baselines and zero-shot Large Language Model (LLM) approaches, while approaching the performance of fully supervised models trained on gold labels (which achieved an AUPRC of 77.57%).
• Comparison: The weakly supervised approach outperformed a baseline using the XPRESS framework (which applied keywords directly to full text) and standard rule-based methods. The domain-adapted pre-training (TPT) consistently improved performance across all transformer models.
• Robustness: The model demonstrated robustness to variations in cluster selection and showed smooth performance degradation when the proportion of gold labels in training was reduced, indicating resilience to label noise.
• Efficiency: The method reduced the need for manual annotation by over 1,500 hours for a dataset of this size compared to a fully supervised approach.

Significance and Claims
The paper claims that weakly supervised NLP methods offer a practical and scalable framework for disease identification from clinical discharge letters, particularly in settings where manual annotation is infeasible. The authors emphasize that while there is a modest performance gap (4–6% AUPRC) compared to fully supervised models, this trade-off is acceptable for large-scale cohort selection where the goal is to identify groups of potentially relevant patients rather than achieve perfect individual-level classification.

The study highlights that the pipeline effectively learns from the entire text of discharge letters, making it advantageous when automatic extraction of diagnosis strings fails or when diagnoses are not explicitly stated in the designated section. The authors conclude that their approach provides a generalizable solution for clinical NLP in Italian, reducing the burden of manual labeling while maintaining competitive performance. They acknowledge limitations regarding the single-annotator gold standard and the small size of the secondary validation dataset, suggesting that future work should include double annotation and validation on more diverse, multi-regional datasets.