Technical Summary: Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale

Problem Statement

Large Language Models (LLMs) are increasingly utilized to generate or synthesize clinical text for applications ranging from documentation assistance to data augmentation. However, existing evaluations of LLM-generated clinical notes typically focus on narrow aspects, such as textual similarity to human-written references (intrinsic evaluation) or performance on a single downstream task (extrinsic evaluation). These approaches are rarely presented jointly on a consistent, large-scale dataset. Consequently, there is a lack of systematic understanding regarding how well LLMs preserve core clinical information, maintain predictive utility, and adhere to factual accuracy when rephrasing real-world clinical notes at scale.

Methodology

The authors conducted a multi-angle, systematic evaluation of synthetic clinical notes generated by rephrasing 4.6 million de-identified notes from the MIMIC-III and MIMIC-IV databases. The study utilized three efficient, open-source LLMs (~8B parameters): Llama-3.1, Mistral-v0.3, and Qwen-2.5.

The methodology comprised five distinct evaluation pillars:

1. Data Synthesis: The authors compared two rephrasing strategies:

   • Note-level: Rephrasing the entire clinical note as a single context.
   • Chunk-level: Splitting long notes into smaller segments (~150 words) and rephrasing them individually to mitigate context length limitations.
     The resulting corpus, MIMIC-EXT-SYNNOTES, contains 9.1 billion words.

2. Intrinsic Evaluation: The synthetic notes were compared against human-written references using:

   • Linguistic Features: Sentence length, part-of-speech distributions, and readability scores (Flesch-Kincaid, SMOG).
   • Textual Similarity: ROUGE and BERTScore metrics.
   • Semantic Fidelity: Extraction of UMLS Concept Unique Identifiers (CUIs) and their mapping to semantic types (TUIs) to measure the preservation of medical concepts.

3. Extrinsic Evaluation (Downstream Utility): The authors trained clinical NLP models on synthetic data and evaluated them on four key healthcare tasks:

   • Risk Stratification: In-hospital mortality and 30-day readmission prediction.
   • Medical Coding: Diagnosis-Related Group (DRG) prediction and ICD-10 coding.
   • Fairness: Assessment of bias across age, gender, and ethnicity subgroups using Average Absolute Odds Difference (AAOD) and Statistical Parity Difference (SPD).

4. Fact-Checking and Error Analysis: Using an "LLM-as-judge" approach (Gemini-2.5), the authors decomposed notes into atomic facts to calculate precision (claims entailed by the original) and recall (original claims entailed by the synthetic). They also performed a qualitative taxonomy of error types.

5. Data Augmentation: The utility of task-agnostic synthetic notes was tested by augmenting the training set for the ICD coding task, specifically examining performance on rare ICD codes.

Key Results

Intrinsic Quality and Semantic Shift

Synthetic notes were generally shorter with longer sentences and higher readability scores than human notes. While they preserved broad semantic distributions, they exhibited a shift toward general terms (e.g., "Natural Phenomenon or Process") and omitted specific anatomical and specialist terminology (e.g., "Body Space or Junction," "Medical Device").

Downstream Utility

• Coarse-Grained Tasks: For single-label tasks (mortality, readmission, DRG), synthetic notes achieved performance comparable to human-written notes, with results overlapping 95% confidence intervals. This indicates that core clinical information regarding patient severity is preserved.
• Fine-Grained Tasks: The ICD coding task (multi-label, 8K classes) showed a significant performance drop (4–10% decrease in Micro F1) when using note-level synthetic data.
• Impact of Chunking: Rephrasing by chunks rather than whole notes substantially mitigated the performance gap in ICD coding, suggesting that the loss of fine-grained details in note-level rephrasing is driven by the difficulty of integrating information from long contexts.

Factuality and Error Patterns

• Precision vs. Recall: Synthetic notes exhibited high factual precision but significantly lower recall, indicating a tendency to omit details rather than invent them. This was exacerbated in long notes (>2K words), where recall dropped to 0.3.
• Chunking Trade-off: While chunking improved recall by preserving more details, it slightly reduced precision and increased hallucination rates. This occurred because LLMs, lacking full context in a single chunk, sometimes generated plausible but unsupported claims based on information present in other chunks.
• Error Taxonomy: The dominant failure mode was misinterpretation of clinical context, followed by measurement errors, temporal confusion, and fabrication of claims (e.g., inventing names or dates).

Fairness and Augmentation

• Fairness: Models trained on synthetic notes showed no significant increase in bias compared to those trained on human notes across demographic subgroups.
• Augmentation: Despite being task-agnostic, synthetic notes effectively augmented training data for ICD coding. The most significant gains were observed for rare ICD codes, where Micro F1 improved from 0.03 to 0.10.

Iterative Synthesis

Repeated synthesis (iteratively rephrasing the output of the previous round) did not lead to immediate "model collapse" in downstream tasks, though ICD coding performance declined progressively over five iterations.

Significance and Contributions

The paper claims that its primary contribution is a systematic, multi-angle evaluation of synthetic clinical text at a million-note scale, bridging the gap between intrinsic similarity and extrinsic utility.

Key takeaways emphasized by the authors include:

1. Complementarity of Evaluation: Intrinsic and extrinsic metrics provide complementary views; high similarity does not guarantee utility for fine-grained tasks, and high utility does not guarantee factual completeness.
2. The Long-Context Bottleneck: The study highlights that while LLMs can preserve coarse clinical information, they struggle with fine-grained details in long contexts. Chunking is a viable strategy to recover detail but introduces a trade-off with factual precision due to context fragmentation.
3. Task-Agnostic Utility: Synthetic notes generated without specific task instructions can still serve as valuable resources for data augmentation, particularly for underrepresented classes (rare codes).
4. Error Patterns: LLM errors in clinical rephrasing are not random but follow recurring patterns (context misinterpretation, temporal confusion), suggesting that future improvements should target these specific failure modes rather than general generation quality.

The authors conclude that while LLM-generated clinical notes have potential for data augmentation and documentation support, their usage requires careful assessment of the application context, particularly regarding the tolerance for detail loss versus hallucination. They release the MIMIC-EXT-SYNNOTES corpus to support future research.