💬 NLP

DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

This paper introduces DistillNote, a functional evaluation framework that assesses the clinical utility of LLM-generated summaries by measuring their ability to retain diagnostic signal in downstream prediction tasks, demonstrating that highly compressed summaries can preserve up to 97% of the original diagnostic performance for heart failure detection.

Original authors: Heloisa Oss Boll, Antonio Oss Boll, Leticia Puttlitz Boll, Ameen Abu Hanna, Iacer Calixto

Published 2026-02-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Heloisa Oss Boll, Antonio Oss Boll, Leticia Puttlitz Boll, Ameen Abu Hanna, Iacer Calixto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor rushing through a hospital. You have a patient with a serious heart condition, but their medical file is a massive, 50-page novel filled with handwritten notes, lab results, and long stories from the patient. You don't have time to read the whole book before you need to make a life-saving decision.

Enter Large Language Models (LLMs). Think of these as super-smart AI librarians. You hand them the 50-page novel, and they instantly write a one-page "Cliff's Notes" version for you. This is great for saving time, but it raises a scary question: Did the AI throw away the most important clues while summarizing? If the AI accidentally leaves out a tiny detail about an allergy or a specific symptom, the doctor might make the wrong diagnosis.

This paper introduces a new way to test these AI summaries, called DistillNote. Instead of just asking, "Does this summary sound nice?" or "Does it look like the original?", they ask a much more practical question: "If a doctor (or another AI) reads this summary, can they still make the correct diagnosis?"

Here is the breakdown of their experiment using simple analogies:

1. The Experiment: The "Compression" Game

The researchers took thousands of real patient admission notes from a public database (MIMIC-IV). They asked different AI models to summarize these notes in three different ways, creating three levels of "compression":

The "One-Step" Summary (36% compression): Like asking a friend to quickly tell you the plot of a movie. It's shorter, but still has most of the details.
The "Structured" Summary (53% compression): Like organizing that plot into bullet points: "Beginning," "Middle," "End." It's more organized but cuts out some fluff.
The "Distilled" Summary (79% compression): This is the extreme version. It's like the AI trying to summarize the entire 50-page novel into a single, dense paragraph. It's about 20 times shorter than the original!

2. The Test: The "Heart Failure" Challenge

To see if these summaries actually work, the researchers didn't just ask humans to grade them (which is slow and expensive). Instead, they used a functional test.

Imagine you have a detective who is very good at solving "Heart Failure" cases.

First, they let the detective read the full 50-page novel. How good are they at solving the case? (Score: 94/100).
Next, they let the detective read the short summaries. Can they still solve the case just as well?

The Result:
Even with the most extreme summary (the 20x shorter "Distilled" version), the detective could still solve the heart failure cases 97% as well as if they had read the full novel. The AI managed to keep the "critical clues" (the diagnostic signals) while throwing away the "noise" (the repetitive or irrelevant text).

3. The Twist: The "Judge" vs. The "Reality"

The researchers also tried the old-school way of testing: asking an AI to act as a "Judge" and asking human doctors to read the summaries.

The AI Judge loved the "Distilled" summaries because they were factually accurate and didn't make up stories.
The Human Doctors actually preferred the "One-Step" summaries because they felt those were more useful for making quick decisions.

The Big Lesson:
The "Judge" (who just checks if the text looks good) and the "Reality" (can we actually use this to save a life?) gave different answers. The paper argues that we must test AI summaries by seeing if they help us do the actual job, not just by seeing if they sound pretty.

Why This Matters

This study is like a safety check for the future of hospital AI. It proves that we can shrink massive medical records down to tiny, manageable summaries without losing the life-saving information inside.

However, it also warns us: There is a trade-off. The more you shrink the summary, the slightly lower the performance gets. Doctors and hospitals need to decide: Do we want a summary that is 99% perfect but long, or 97% perfect but incredibly short and fast?

DistillNote gives them the data to make that choice safely, ensuring that when AI summarizes a patient's history, the doctor isn't missing the one detail that matters most.

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed to summarize lengthy Electronic Health Records (EHR) to reduce clinician cognitive load and burnout. However, current evaluation methods for these summaries suffer from critical limitations:

Lexical/Semetric Focus: Standard metrics (e.g., ROUGE, BERTScore) measure lexical overlap or semantic similarity but fail to assess whether diagnostic signals (critical clinical information) are preserved.
Subjectivity & Scalability: Human expert evaluation is the gold standard but is costly, time-consuming, and difficult to scale.
LLM-as-Judge Limitations: Using LLMs to evaluate other LLMs (LLM-as-judge) offers scalability but often lacks consistency and reliability regarding clinical utility.
The Gap: There is no scalable framework to determine if a compressed summary retains enough information to support complex downstream clinical tasks (e.g., diagnosis or risk prediction).

2. Methodology: The DistillNote Framework

The authors propose DistillNote, a functional evaluation framework that assesses summary quality by measuring the retention of predictive signal in a downstream clinical task.

A. Dataset and Task

Data Source: MIMIC-IV (Medical Information Mart for Intensive Care IV), specifically using 64,734 admission notes from 49,869 patients.
Downstream Task: Heart Failure (HF) Diagnosis. This was chosen because it is a complex diagnostic challenge requiring the integration of diverse clinical signals (symptoms, history, labs, physical exams).
Goal: Fine-tune LLMs on summaries and compare their diagnostic performance (AUROC) against models trained on full original notes.

B. Summarization Strategies

The study generated over 192,000 summaries using three distinct strategies with increasing compression rates:

One-step: Direct summarization of the full admission note in a single pass.
- Compression: ~36% (Avg 262 words).
Structured (Divide-and-Conquer): The note is split into four specific clinical insights (Chief Complaint, Medical History, Physical Exam, Social/Family History). Each is summarized separately and concatenated.
- Compression: ~53% (Avg 195 words).
Distilled: The four structured summaries are fed into a second LLM pass to synthesize a single, highly condensed summary.
- Compression: ~79% (Avg 87 words; ~20x smaller than original).

C. Models Used

Three state-of-the-art LLMs were utilized for both summarization and downstream fine-tuning:

Deepseek-R1-70B: Reasoning-focused.
OpenBioLLM-70B: Biomedical-focused.
Phi-4-14B: General-purpose (smaller model).

D. Evaluation Metrics

The study employed a multi-lens evaluation approach:

Functional Evaluation (Primary): Fine-tuning LLMs on the summaries to predict Heart Failure. Performance measured via AUROC, AUPRC, and F1-score.
LLM-as-Judge: Using Phi-4 to score summaries on a 1–5 scale for Relevance, Factual Fabrication, and Clinical Actionability.
Clinician Validation: Two board-certified clinicians performed blinded pairwise comparisons on 18 cases to validate LLM judge scores.

3. Key Results

A. Functional Performance (Diagnostic Signal Retention)

Despite significant text compression, LLM-generated summaries retained the vast majority of diagnostic utility:

Baseline (Full Notes): AUROC of 0.941 (Phi-4).
One-step Summary: AUROC 0.929 (98.7% retention).
Structured Summary: AUROC 0.926 (98.4% retention).
Distilled Summary: AUROC 0.917 (97.4% retention).
Key Finding: Even the most compressed summaries (79% reduction, ~20x shorter) retained 97% of the predictive performance of the original notes. The "compression-to-performance" trade-off was highly favorable, with AUROC efficiency ratios ranging from 19.8x to 30.3x.

B. Model Performance Differences

Phi-4 (14B) performed best on One-step and Structured summaries.
Deepseek-R1 (70B) performed best on Distilled summaries, suggesting reasoning models handle high-level synthesis better.
OpenBioLLM did not outperform the others in any strategy.

C. Evaluation Method Comparison

LLM-as-Judge vs. Functional: There was a discrepancy. LLM judges rated "Distilled" summaries highly for factual accuracy, yet these summaries showed slightly lower functional performance (AUROC) compared to One-step summaries.
Clinician vs. LLM: Clinician preferences correlated moderately with LLM judges (Spearman $\rho$ = 0.67). Clinicians generally preferred One-step summaries for relevance but found Distilled summaries sufficient for clear cases.
Conclusion: Subjective metrics (relevance, hallucination) do not fully predict functional utility. A summary can be "factually accurate" but miss subtle signals required for complex prediction.

4. Key Contributions

DistillNote Framework: A novel, scalable, task-based evaluation method that quantifies clinical utility by measuring signal retention in downstream prediction tasks, moving beyond lexical similarity.
Empirical Trade-off Analysis: The first large-scale quantification of the compression-to-performance trade-off in clinical summarization, demonstrating that extreme compression (up to 79%) is feasible with minimal diagnostic loss.
Public Dataset: Release of a dataset containing >192,000 LLM-generated clinical summaries across three strategies and three models to support future clinical NLP research.
Evaluation Insight: Evidence that functional evaluation is necessary to complement subjective metrics, as high "factuality" scores do not guarantee high clinical utility.

5. Significance and Implications

Clinical Deployment: The results suggest that LLM summaries can safely replace full notes for specific downstream tasks (like risk stratification) in real-time settings, significantly improving speed and scalability without compromising patient safety.
Resource Efficiency: High compression rates reduce computational costs for storage and inference, making LLM integration more viable for resource-constrained healthcare environments.
Future Research: The framework is domain-agnostic and can be adapted to other clinical tasks (e.g., mortality prediction, sepsis detection) to guide the deployment of AI summarizers.

Conclusion: DistillNote demonstrates that LLMs can distill clinical notes into highly compressed summaries that preserve the critical diagnostic signals necessary for complex medical decision-making, providing a robust, objective basis for deploying these tools in healthcare.