Towards Multidisciplinary Summarization of Hospital Stays: Efficient Sentence-Level Clinical Provenance Categorization

Technical Summary: Towards Multidisciplinary Summarization of Hospital Stays

Problem Statement
Effective "all-team" clinical summarization in high-complexity environments, such as the Neonatal Intensive Care Unit (NICU), is hindered by the unbalanced nature of Electronic Health Records (EHR). Extensive documentation by physicians often overshadows contributions from other disciplines (nurses, therapists), leading to incoherent outputs when heterogeneous text is simply pooled. To enable structured summarization that preserves multi-disciplinary insights, the authors identify the need for accurate, sentence-level categorization of clinical provenance (identifying the source discipline or section of a note) across diverse clinical notes. While previous approaches have moved from rule-based matching to context-aware supervised models, these often struggle to generalize across specialized clinical settings due to a lack of domain-specific annotation and schema alignment.

Methodology
The study proposes a pipeline using Supervised Fine-Tuning (SFT) of Large Language Models (LLMs) to categorize sentence-level spans based on clinical provenance.

• Datasets:
  • Source Domain (Adult ICU): The authors utilized MedSecId, a corpus of 2,002 MIMIC-III notes. Annotations were distilled into 17 high-prevalence categories (e.g., History of Present Illness, Hospital Course), resulting in 1,562 notes. The data was formatted as a structured chat-based instruction-tuning task with 134,640 sentence-level spans, where each instance included the target sentence, its immediate context (predecessor and successor), and a system instruction to assign a single header.
  • Target Domain (NICU): A gold-standard "NICU Pilot Evaluation Set" was constructed containing 227 sentence-level spans derived from three de-identified, multidisciplinary discharge summaries. This schema includes 25 fine-grained categories (e.g., Feeding-Nutrition, Respiratory, discipline-specific summaries) that only partially overlap with adult ICU headers, creating a stringent test for semantic generalization under schema shift.
• Model Architecture and Training:
  • Two Llama-3 models were evaluated: Llama-3.1-8B-Instruct and Llama-3.3-70B-Instruct.
  • Fine-Tuning Strategy: The models were fine-tuned using QLoRA (Quantized Low-Rank Adaptation) to reduce memory usage by ~75% relative to 16-bit storage.
  • Quantization: Models were quantized to 4-bit Normal Float (NF4) precision.
  • Adaptation Scope: LoRA adapters (rank $r=16$, $\alpha=32$) were attached to all linear projections (MLP gate, up, and down layers) rather than restricting adaptation to attention modules, increasing trainable parameter coverage.
  • Hardware: The 70B model was trained on four A100 GPUs (80GB memory each) using data-parallel training.
  • Hyperparameters: Learning rates were set to $2 \times 10^{-4}$ for the 8B model and $1 \times 10^{-4}$ for the 70B model.

Key Results

• In-Domain Performance (MedSecId): Both models achieved strong alignment with the clinical provenance schema, yielding nearly identical weighted F1-scores of 0.94 and Macro F1-scores between 0.92 and 0.93. Performance was consistent across high- and low-frequency categories, with no statistically significant advantage observed for the 70B model over the 8B model within the source domain.
• Cross-Domain Transfer (NICU): A distinct divergence in generalization capabilities emerged when evaluating on the NICU Pilot Set:
  • 70B Model: Demonstrated a scale-dependent transfer effect. The fine-tuned 70B model improved its Macro F1 from 0.52 (Base) to 0.59 and its Weighted F1 from 0.56 to 0.60. Notably, the quantized fine-tuned 70B model outperformed its full-precision baseline while reducing computational requirements.
  • 8B Model: Showed only marginal gains. While Weighted F1 improved slightly (0.47 to 0.50), the Macro F1 remained essentially unchanged (0.53 to 0.52). This suggests the 8B model reinforced performance on high-frequency classes but failed to generalize to minority categories under domain shift.
• Statistical Significance: The authors note that while the performance differences are numerically meaningful, a McNemar exact test yielded a p-value of 0.24. This lack of statistical significance is attributed to the limited sample size (227 spans) and high intra-note correlation.

Significance and Claims
The paper claims that sufficient model capacity is critical for preserving semantic flexibility during cross-domain clinical transfer. While smaller models (8B) can learn complex schemas in-domain, they exhibit limited ability to translate structured supervision into balanced cross-domain performance. In contrast, larger-capacity models (70B) successfully internalize structured constraints while maintaining the flexibility to recognize specialized, low-prevalence categories in unseen domains.

Furthermore, the study demonstrates that efficient quantized adaptation (using QLoRA and 4-bit quantization) enables the use of large-capacity models for structured provenance modeling within standard clinical research hardware constraints (single 80GB GPU). The authors position this work as a foundational step toward a provenance-aware retrieval pipeline, which could serve as a backbone for downstream "all-team" summarization by accurately segmenting notes into discipline-specific zones, thereby reducing hallucinations and improving clinical utility.

Limitations
The authors explicitly acknowledge that the NICU Pilot Set is small (227 spans from three summaries) and that creating such gold-standard datasets is resource-intensive due to the need for manual cross-referencing of diverse discipline-specific notes. Consequently, the statistical power of the cross-domain comparison is limited.