Technical Summary: A Pipeline for Generating Longitudinal Synthetic Clinical Notes Using Large Language Models

Problem Statement

The development and evaluation of clinical AI systems—such as summarization tools, diagnostic support, and automated coding models—are hindered by restricted access to real-world patient data due to stringent regulatory and ethical requirements. While privacy-enhancing techniques like anonymization exist, they often fail to satisfy institutional approval processes or fully mitigate privacy risks. Consequently, there is a critical need for high-quality synthetic clinical data that mimics the structure, content, and longitudinal consistency of real patient records without containing genuine patient information. Existing synthetic data approaches often struggle to balance internal consistency across a patient's journey with the necessary variation in writing styles, note structures, and clinical details found in real-world settings.

Methodology

The authors present a modular, five-stage pipeline designed to generate synthetic clinical notes with longitudinal consistency. The pipeline prioritizes speed and privacy, utilizing Large Language Models (LLMs) for semi-structured and unstructured data generation, supplemented by structured data from the open-source simulator Synthea.

1. Patient Generation

The pipeline begins by generating patient demographics using Synthea UK, a modified version of Synthea tailored to align with UK population demographics and addresses. An LLM (GPT-4o) is then employed to generate additional details, such as next-of-kin contact information. To ensure diversity and realism, the pipeline iteratively refines prompts based on clinical feedback; for instance, it now randomly samples allergies based on national prevalence rather than defaulting to common hallucinations like penicillin allergies. All outputs are strictly formatted as JSON, with automated cleaning functions (using regex and secondary LLM calls) to handle parsing errors.

2. Admission Generation

The system generates two types of admissions: elective and emergency. Admission reasons are sampled from a curated table of common reasons stratified by sex and age, though the architecture allows for integration with SNOMED or ICD coded datasets. The LLM generates admission details including specialty, ward location, current medications, and past medical history, followed by an estimation of the length of stay.

3. Patient Journey Simulation

This stage constructs a longitudinal timeline of events from admission to pre-discharge.

• Initial Generation: A "simple" journey is generated first, containing event types (e.g., admission, ward rounds, operations), timestamps, and brief summaries.
• Complexity Injection: Once the simple journey is complete, complexity is added to each event, including staff assignments, detailed descriptions, and next-step decisions.
• Validation Loop: An LLM validator assesses the realism of the journey. If a journey is terminated early or deemed unrealistic, the pipeline loops to rewrite sections or extend the timeline.
• Filtering: Unwanted events, such as the discharge event itself (to avoid generating discharge summaries prematurely), are filtered out.

4. Clinical Note Generation

For each event in the journey, a specific clinical note is generated using GPT-4o.

• Contextual Prompting: Prompts include patient demographics, the full history of events up to that point, and expected output formats designed with clinician input.
• Clinical Personas: To introduce realistic variation in writing style, staff members are assigned random "personas" (e.g., Concise, Narrative, Bullet Points, Note, ABCDE). This ensures consistent stylistic variation throughout a patient's journey.
• Dynamic Content: The pipeline dynamically selects appropriate "red-flag" symptoms to check based on the admission reason and determines necessary legal examinations.

5. Augmentation and Validation

• LLM Validation: A validator LLM checks the faithfulness of generated notes against the patient journey and context, ensuring no critical details are omitted or altered. This step operates in a loop to iteratively improve output quality.
• Noise Injection: The pipeline adds realistic noise, including:
  • Abbreviations: An LLM intelligently inserts clinical abbreviations into sentences rather than using simple find-and-replace.
  • Typos: The python-typo package simulates realistic typing errors (character swaps, omissions, double spaces), with individual staff members assigned random typo rates.
  • Sign-offs: Staff signatures are added to notes.

Key Contributions

1. A Modular Pipeline: A configurable pipeline for generating synthetic clinical notes that maintains longitudinal consistency across patient records while capturing variation in writing style and clinical detail.
2. The Synthetic Dataset: A publicly released dataset comprising 70 synthetic patients, each with 20–50 clinical notes spanning a full hospital journey. The dataset is provided at multiple validation levels:
   • Silver (Released): Generated via the validated pipeline and LLM without clinician review.
   • Gold (Planned): Includes additional review and validation by clinicians.
   • Bronze (Planned): Generated with validated pipelines but potentially different LLMs and no additional validation.
3. Evaluation Framework: An integrated evaluation suite that measures realism, consistency, and variation using both traditional metrics and LLM-based judges.

Results and Evaluation

The paper details an evaluation pipeline used to iteratively improve the system during development:

• Readability: Measured using Flesch Reading Ease and Dale Chall scores to ensure different clinical personas produce notes with varying but appropriate readability levels.
• LLM-as-a-Judge: LLMs score the generated notes on Fluency (grammar and coherence), Groundedness (alignment with patient journey facts), and Relevance (inclusion of vital information). Scores range from 1 to 5.
• Temporal Coherence: A binary check ensures all events for a patient occur in chronological order.
• Human Evaluation: Clinical feedback identified specific issues (e.g., missing red-flag documentation, unrealistic abbreviations, verbose notes), which were subsequently addressed through prompt engineering and pipeline modifications.

The dataset supports the development of AI tools for summarization, coding, decision support, and bias testing (via a gender-swapping capability).

Significance and Claims

The authors claim that this work provides a practical solution for developing and testing clinical AI systems without reliance on real patient data, thereby bypassing the slow and often unsuccessful institutional approval processes associated with real-world data.

The paper emphasizes that while the pipeline generates high-quality synthetic journeys and notes, it is not a perfect replica of real-world complexity. The authors modestly acknowledge limitations, including:

• Loss of Real-World Complexity: The inability to fully capture the richness of conflicting observations, staff availability issues, and context-dependent decision-making.
• Edge Case Underrepresentation: A tendency to reflect "average" cases rather than the long tail of rare or adverse events.
• Bias and Hallucination: The risk that synthetic data may inherit biases from the underlying LLM or introduce plausible but factually incorrect clinical information.

The significance of the work lies in its provision of a scalable, privacy-preserving dataset that enables the testing of AI tools (including bias detection and multimodal prediction) while adhering to ethical constraints. The code is available on GitHub, and the data is hosted on Hugging Face.