Technical Summary: MedSynth – Realistic, Synthetic Medical Dialogue-Note Pairs

Problem Statement

Physicians dedicate a significant portion of their workday (52–102 minutes daily) to clinical documentation, a burden that contributes to professional burnout and detracts from direct patient care. While automated tools for medical documentation (specifically Dialogue-to-Note and Note-to-Dialogue tasks) offer a promising solution, their development is severely hampered by a lack of large, open-access, and privacy-compliant datasets. Existing datasets are often limited in scope (covering few medical conditions), lack adherence to standard medical formats (such as SOAP), or are unavailable due to privacy restrictions. This scarcity restricts the training of robust models capable of generalizing across diverse medical scenarios.

Methodology

The authors introduce MedSynth, a novel synthetic dataset designed to bridge the gap in training data for medical documentation automation. The methodology involves a multi-stage pipeline driven by Large Language Models (LLMs) and informed by real-world epidemiological data.

1. Data Resource and Disease Distribution

To ensure the synthetic data reflects real-world prevalence, the authors utilized the IQVIA PharMetrics Plus database, a US medical insurance claims database containing over 3 billion claims.

• Filtering: The data was filtered to include only claims with ICD-10 codes (post-2015), resulting in 800 million records and 54,985 unique codes.
• Sampling Strategy: The authors selected the top 2,000 most frequent ICD-10 codes. Despite the natural skew in disease frequency, they employed uniform sampling (generating 5 dialogue-note pairs per code) to prevent the dataset from being dominated by common conditions, thereby ensuring the resulting models are robust across a diverse spectrum of diseases.

2. Quality Factors

Before generation, a survey was conducted with two medical professionals (a general physician and a bio-statistician) to identify 11 critical variables for note quality. These included medical outcomes, history, symptom descriptions, demographics, patient behavior, clinical setting, and treatment disparities. Two additional variables (Physical Exams and Investigations/Tests) were added post-survey.

3. Data Generation Pipeline

The pipeline generates notes first, then derives dialogues from them, utilizing GPT-4o and In-Context Learning (ICL) with Chain-of-Thought (CoT) and role-playing strategies.

• Note Generation Pipeline (Four Agents):

  1. Scenario Provider: Receives an ICD-10 description, selects a relevant physician role, and generates a scenario incorporating the 13 quality variables. It uses One-Shot ICL with examples from the Aci-Bench dataset.
  2. Scenario Judge: Evaluates the generated scenario for three conditions:
     • Diversity: At least 4 of the 13 variables must differ from previously approved scenarios for the same ICD-10 code.
     • Medical Accuracy: Symptoms, tests, diagnosis, and treatment must be logically consistent with the ICD-10 code.
     • Plausibility: The scenario must be realistic for a medical setting.
     • Feedback Loop: If rejected, the Judge provides specific feedback to the Provider to regenerate the scenario.
  3. Note Writer: Generates a medical note adhering to the SOAP (Subjective, Objective, Assessment, Plan) format based on the approved scenario and physician role.
  4. Note Polisher: Refines the note to ensure information is correctly categorized into SOAP sections (e.g., ensuring imaging orders appear in "Plan" if pending, or "Objective" if completed).

• Dialogue Generation Pipeline (Two Agents):

  1. Dialogue Generator: Creates a doctor-patient dialogue corresponding to the medical note using Few-Shot ICL (3 examples from Aci-Bench).
  2. Dialogue Polisher: Enhances realism by adding social chatter and modal particles (e.g., "hmm," "okay") while strictly ensuring all medical information from the note is accurately reflected in the dialogue to prevent hallucinations.

Key Contributions

1. MedSynth Dataset: A comprehensive, privacy-compliant dataset containing 10,035 dialogue-note pairs covering 2,001 unique ICD-10 codes. It is the largest publicly available dataset of its kind that adheres to the standard SOAP structure.
2. Novel Generation Pipeline: A multi-agent, iterative pipeline that ensures diversity, medical accuracy, and structural adherence (SOAP) in synthetic data, extensible to other domains.
3. State-of-the-Art (SotA) Models: The authors release fine-tuned models (based on Llama-3-8B-Instruct and Mistral-7B) that achieve SotA performance in both Dial-2-Note and Note-2-Dial tasks.

Results

The authors evaluated MedSynth by fine-tuning models and testing them on the Aci-Bench test set, using an LLM-based "jury" (Prometheus, GPT-4o, Qwen2.5) and human medical experts for evaluation.

• Dial-2-Note Performance: Models trained on MedSynth (alone or combined with Aci-Bench) were preferred by the LLM jury over models trained on NoteChat or PriMock57 in 60% of comparisons. Notably, models trained only on MedSynth significantly outperformed those trained only on NoteChat (95% preference), primarily because NoteChat lacks structured SOAP notes, leading to poor format adherence in the latter.
• Note-2-Dial Performance: Similarly, MedSynth-trained models achieved a 55% preference rate over NoteChat-trained models when combined with Aci-Bench, and an 87.5% preference rate when compared in isolation.
• Human Evaluation: In a study with three medical professionals, human annotators showed a slight preference (54.2%) for MedSynth-trained models, aligning with the LLM jury's 60% preference.
• Ablation Studies: Removing the Scenario Judge Agent reduced performance, confirming its necessity for maintaining diversity and quality. Furthermore, using GPT-4o as the backbone for the generation pipeline yielded better results than using Llama-3.3-70B or Qwen2.5-32B.

Significance and Claims

The paper claims that MedSynth addresses the critical scarcity of diverse, privacy-compliant, and structurally standard training data for medical AI. By providing a dataset that mimics primary care conditions and adheres to SOAP standards, MedSynth enables the development of automation tools that can reduce the documentation burden on physicians.

The authors emphasize that while the dataset is synthetic and generated via LLMs, it is designed to be a tool for model development rather than a source of reliable medical information for clinical use. They acknowledge limitations, including the potential inability of synthetic data to capture all nuances of real-world human interaction and the need for further validation in real-world clinical workflows. However, they assert that MedSynth represents a significant step forward in advancing the Dialogue-to-Note and Note-to-Dialogue tasks, offering a foundation for future tools to mitigate physician burnout.