Technical Summary: Clinician-Centered Evaluation of LLM-Generated Discharge Summaries for Longer Hospitalizations

Problem Statement

Hospital discharge summaries are critical for safe transitions of care, yet they impose a significant cognitive burden on clinicians, particularly during longer hospitalizations (7–21 days). These extended stays generate high volumes of fragmented, repetitive, and complex documentation, making the synthesis of a concise, accurate transition document difficult. While Large Language Models (LLMs) have shown promise in generating discharge summaries, prior evaluations have largely focused on shorter stays and have rarely examined the specific priorities of receiving clinicians (Primary Care Physicians, or PCPs) or the reporting of incidental findings. Furthermore, existing literature lacks evidence on whether LLMs can effectively process Electronic Health Records (EHRs) that exceed current model context window limitations, a common challenge in long-stay cases.

Methodology

The study conducted a clinician-centered evaluation comparing LLM-generated discharge summaries against human-authored summaries for 60 Internal Medicine hospitalizations lasting between 7 and 21 days.

Data Processing and Pipeline:

• Cohort: 60 encounters were stratified from a pool of 400 historical inpatient cases at Stony Brook University Hospital (Jan 2023–Dec 2024).
• Input Data: The pipeline extracted 370 unique note types (including ED notes, consults, progress notes, and imaging reports) from the Oracle Cerner EHR. This resulted in an average 55% reduction in text volume per encounter after filtering.
• LLM Architecture: To address context window limitations, the authors developed a scalable pipeline using a sliding window approach. Clinical documents were segmented into 50,000-token chunks. An iterative refinement process was employed where the LLM (GPT-5.2) generated an initial chronological, bulleted draft from the first segment and updated it as successive segments were processed.
• Incidental Findings: A separate prompt workflow was designed to identify and extract clinically relevant incidental findings from radiology reports.
• Final Output: The system consolidated the chronological draft and incidental findings into a final narrative summary.

Evaluation Protocol:

• Reviewers: 12 attending physicians (6 hospitalists and 6 PCPs) independently evaluated the summaries.
• Workflow Alignment: Hospitalists reviewed all inpatient notes before evaluating the summaries, while PCPs reviewed the summaries first and then corroborated findings with inpatient notes to simulate real-world transition-of-care workflows.
• Metrics:
  • Quantitative: Reviewers rated summaries on a 5-point Likert scale for quality, readability, factuality, and completeness. PCPs provided additional ratings on ease of understanding, verbalization to patients, and follow-up utility.
  • Error Annotation: Reviewers annotated inaccuracies and omissions, rating the potential for harm and likelihood of harm using an adapted Agency for Healthcare Research and Quality (AHRQ) scale.
  • Incidental Findings: Reviewers assessed the accuracy and appropriateness of reported incidental findings.
• Qualitative: Post-review semi-structured debriefing sessions were conducted to gather qualitative feedback on workflow and trust.

Key Results

Performance and Preference:

• Preference: Physician reviewers preferred the LLM-generated summaries in 95% of the 60 encounters.
• Quality Metrics: LLM-generated summaries received significantly higher mean scores than human-authored summaries across all metrics:
  • Quality: 4.71 vs. 3.16 ($p = 1.40 \times 10^{-17}$)
  • Readability: 4.66 vs. 3.39 ($p = 3.25 \times 10^{-15}$)
  • Factuality: 4.62 vs. 3.73 ($p = 1.08 \times 10^{-13}$)
  • Completeness: 4.69 vs. 3.42 ($p = 2.67 \times 10^{-15}$)
• Consistency: LLM summaries demonstrated greater consistency, with a mean absolute deviation in Likert scores less than half that of human-authored summaries (0.49 vs. 1.07).

Error Analysis:

• Frequency: LLM-generated summaries contained fewer total errors (mean 0.42) compared to human-authored summaries (mean 0.73; $p = 0.011$).
• Error Types: The reduction in errors was primarily driven by fewer omissions in LLM summaries. There was no significant difference in the rate of inaccuracies between the two groups.
• Harm Potential: There was no statistically significant difference in the estimated potential for harm or the likelihood of harm between errors in LLM-generated and human-authored summaries.

Stakeholder-Specific Findings:

• PCP Perspective: PCPs rated LLM summaries significantly higher than hospitalists did for readability and completeness. PCPs identified significantly more omissions in both summary types than hospitalists and assigned higher likelihood-of-harm scores to errors, reflecting their responsibility for longitudinal post-discharge care.
• Incidental Findings: Of 31 LLM-identified incidental findings, 93.5% were rated as factually correct and 87.1% as appropriate for reporting.

Qualitative Insights:

• Reviewers cited reduced cognitive burden, higher consistency, and clearer presentation of treatment rationale and unresolved follow-up issues as primary reasons for preferring LLM summaries.
• Hospitalists noted that human-authored summaries often suffered from redundancy and physician-dependent variability, whereas LLM summaries were more uniform and succinct.
• PCPs emphasized that LLM summaries better facilitated the transition of care by highlighting actionable items and pending results.

Significance and Claims

The paper claims that this study extends prior work by demonstrating the clinical utility of LLMs in summarizing longer, more complex hospitalizations, a scenario previously underexplored due to technical and documentation volume challenges.

• Stakeholder-Centered Design: The results highlight that designing transition documents around the informational priorities of receiving clinicians (specifically PCPs) yields significant improvements in perceived quality and utility. The study suggests that prompting strategies aligned with PCP workflows (e.g., focusing on action items and follow-up) are crucial.
• Technical Scalability: The study provides evidence that LLMs can effectively process EHRs exceeding standard context windows using iterative sliding window techniques, maintaining high-quality output even for 21-day hospitalizations.
• Safety and Oversight: While LLM-generated summaries showed fewer omissions and higher quality, the authors emphasize that they are not error-free. The findings support a model of supervised LLM-assisted summarization, where the AI reduces cognitive burden and improves documentation quality, but clinician oversight remains essential for verification, particularly regarding high-risk content like medication changes and incidental findings.

The authors conclude that LLM-assisted discharge summarization holds potential to reduce clinician burden and enhance transition-of-care communication, provided it is implemented with appropriate clinician verification workflows.