A Typology of Synthetic Datasets for Dialogue Processing in Clinical Contexts

This paper reviews the creation, evaluation, and application of synthetic clinical dialogue datasets while proposing a novel typology to classify data synthesis levels, aiming to guide their effective use and generalization in healthcare NLP tasks.

The Gist

Imagine you are trying to teach a robot how to be a doctor. You need it to learn how to talk to patients, ask the right questions, and understand symptoms. The best way to teach it would be to show it thousands of real conversations between real doctors and real patients.

But here's the problem: Real patient conversations are like gold dust. They are incredibly valuable, but they are also locked away in vaults because of privacy laws. You can't just walk into a hospital, grab a recording of a patient's visit, and feed it to a computer. It's a huge privacy risk.

So, what do researchers do? They build fake conversations. They create "synthetic" datasets.

This paper is essentially a user manual and a classification guide for these fake medical conversations. The authors are saying, "We have a lot of these fake datasets, but we don't have a good way to talk about how fake they are or how they were made. Let's fix that."

Here is the breakdown of their ideas, using some everyday analogies:

1. The Problem: The "Fake" vs. "Real" Confusion

Right now, people tend to think of data as a simple switch: it's either Real (a real conversation) or Fake (a computer made it up).

The authors say this is like thinking a cake is either "made from scratch" or "bought from a store." In reality, there's a whole spectrum.

• Maybe you bought a cake mix (real ingredients) but added your own frosting (synthetic).
• Maybe you baked a cake from scratch but used a recipe written by a computer.
• Maybe you bought a cake, but the store changed the name of the baker on the box.

In the world of medical data, "synthetic" isn't a binary switch. It's a continuum. Some datasets are just real conversations with names changed (like blurring a face in a photo). Others are completely made up by a computer, but they sound very real.

2. The Solution: A New "Menu" for Data

To fix the confusion, the authors propose a new Typology (a fancy word for a classification menu). They look at two main ingredients to decide what kind of "fake" a dataset is:

1. Who did the work? (A Human or a Machine?)
2. What did they do? (Did they just tweak an existing conversation, or did they invent a brand new one?)

They break it down into three "Types":

• Type 1: The "No-Touch" Zone.

  • Analogy: You walk into a forest, take a photo of a tree, and put it in a book. You didn't change the tree; you just captured it.
  • Data: Real conversations recorded in the wild. No changes made.

• Type 2: The "Edit" Zone.

  • Analogy: You take a real photo of a tree, but you use Photoshop to blur the license plate of a car in the background or change the color of the sky. The tree is still real, but you altered specific details.
  • Data: Real conversations where names are changed, or the text is translated into another language, or specific words are swapped out to protect privacy. The core conversation is still "real."

• Type 3: The "Invention" Zone.

  • Analogy: You hire an actor to pretend to be a tree, or you draw a picture of a tree that never existed.
  • Data: A conversation that never happened. It might be written by a human actor role-playing a patient, or generated entirely by an AI (like a Large Language Model) based on a prompt like, "Write a dialogue between a doctor and a patient with a headache."

3. Why This Matters: The "Uncanny Valley" of Data

The authors point out that just because a dataset is "Type 3" (completely made up) doesn't mean it's useless. It depends on what you are using it for.

• The "Scripted" vs. "Improvised" Trap:
  Imagine you want to teach a robot how to handle a medical emergency.

  • Scenario A: You give the robot a script written by a doctor (Human Type 3). It's perfect grammar, but it sounds stiff and robotic.
  • Scenario B: You give the robot a recording of two actors improvising a scene based on a medical case (Human Type 3, but more natural).
  • Scenario C: You give the robot a conversation generated by an AI (Machine Type 3).

  If you only care about what medical terms are used, all three might work. But if you care about how people panic, stutter, or interrupt each other (the "pragmatics" of speech), the AI or the scripted version might fail miserably. The authors argue we need to be honest about which "flavor" of fake we are using so we don't build a robot that sounds like a robot when it talks to a real human.

4. The Cultural "Lost in Translation" Problem

The paper also warns about a subtle danger: Context.

Imagine you take a real conversation between a doctor and a patient in the US (about insurance and dialysis) and use a computer to translate it into Arabic.

• The Words: The translation might be perfect.
• The Context: The meaning might be broken. In the US, the conversation is about insurance coverage. In a different country, the conversation might be about family support or different healthcare systems.

If you use that translated "fake" dataset to train a robot for the Middle East, the robot might learn the wrong social rules. It's like teaching someone how to drive in the US, then handing them the manual and saying, "Now go drive in the UK." The car is the same, but the rules of the road are different.

The Big Takeaway

This paper is a call to action for scientists and developers. It says: "Stop just calling everything 'synthetic data.' Be specific!"

Before you use a dataset to train your AI, ask:

1. Was this made by a human or a machine?
2. Did they just tweak real data, or did they invent it from scratch?
3. Does the "fake" conversation actually feel like the real thing in terms of culture and emotion?

By using their new "menu" (Typology), researchers can stop guessing and start knowing exactly what they are feeding their AI, ensuring that the robots we build to help us are actually helpful, safe, and realistic.

Technical Summary

1. Problem Statement

The field of Clinical Natural Language Processing (NLP) faces significant barriers in acquiring authentic data due to privacy regulations (e.g., HIPAA), data governance issues, and the sensitive nature of patient-provider interactions. Consequently, researchers increasingly rely on synthetic datasets to train and evaluate models for tasks like automated documentation, chatbots, and information extraction.

However, the term "synthetic" is ill-defined in the context of linguistic data. Unlike structured numerical data, where synthetic generation is often a matter of statistical perturbation, linguistic data involves complex semantic, pragmatic, and discursive properties. The current literature lacks a unified framework to:

• Distinguish between different degrees of "syntheticness" (e.g., anonymized real data vs. AI-generated fiction).
• Classify the roles of human vs. machine intervention in dataset creation.
• Evaluate the validity and representativeness of these datasets for specific clinical applications.

2. Methodology

The authors employed a mixed-method approach combining conceptual framework development with a systematic literature review.

A. Conceptual Framework Development

The authors critiqued existing definitions of synthetic data (often borrowed from the U.S. Census Bureau) and argued that "synthetic" is not a binary property but a continuum. They proposed a new conceptual definition of a dataset as a "teleological" collection of information-bearing objects intended for a specific representational function.

Based on this, they developed a Novel Typology (Table 1 in the paper) to classify datasets along two dimensions:

1. Agent of Intervention: Human vs. Machine.
2. Type of Intervention:
   • Type 1 (No Intervention): Naturalistic data reflecting genuine exchanges.
   • Type 2 (Alteration): Perturbation of existing microdata (e.g., anonymization, translation, paraphrasing) while maintaining lineage to the source.
   • Type 3 (Generation): Creation of entirely new microdata de novo (e.g., screen-written scripts, LLM-generated dialogues).

This results in a matrix (e.g., "Human Type 3, Machine Type 1") that allows for granular description of a dataset's composition.

B. Systematic Literature Review

To validate the typology and survey the landscape, the authors conducted a structured search across three databases: PubMed, ACL Anthology, and dblp.

• Search Scope: Papers published between 2019 and April 2025 focusing on clinical dialogues (spoken or written) that are synthetic or natural.
• Filtering Process:
  • Initial retrieval: 1,626 papers.
  • LLM-Assisted Filtering: Used LLMs (Gemma 3 27b and ChatPDF) to filter for papers describing the creation of reusable clinical dialogue datasets.
  • Manual Review: Final selection of 20 unique papers representing 25 datasets (some papers contained multiple datasets).
• Exclusion Criteria: Non-dialogue data (e.g., EHR notes), datasets where only metadata was synthetic, and papers where synthetic data was generated solely for internal analysis without release.

3. Key Contributions

A. The Human/Machine Microdata Intervention Typology

The paper's primary contribution is the proposed typology, which moves beyond the "real vs. synthetic" binary. It categorizes datasets based on:

• Human Type 1: Genuine, unmodified clinical conversations.
• Human Type 2: Real conversations altered by humans (e.g., redaction, translation).
• Human Type 3: Human-written scripts or role-play simulations (e.g., standardized patients).
• Machine Type 1: Real conversations processed by machines (e.g., ASR transcription).
• Machine Type 2: Real conversations altered by machines (e.g., machine translation, automated anonymization).
• Machine Type 3: Fully machine-generated dialogues (e.g., LLMs simulating patients/providers).

B. Empirical Landscape Analysis

The review revealed a trend toward fully synthetic (Type 3) datasets, particularly following the widespread availability of Large Language Models (LLMs) in 2023.

• Language Diversity: English dominates, but significant work exists in Chinese, Arabic, Korean, and German.
• Domain Specificity: Most datasets are generic (outpatient visits), though some target specific specialties (cardiology, psychology, OB/GYN).
• Task Diversity: Datasets are used for Information Extraction, Summarization, ASR training, Named Entity Recognition (NER), and dialogue generation.

C. Identification of Limitations in Current Synthesis

The authors highlight that current typologies fail to capture:

• Modality: The difference between "imagined" text (screen-written) and "transcribed" improvisation (actors role-playing). Both may be Type 3, but their pragmatic validity differs.
• Sociolinguistic Context: Synthetic data (e.g., machine-translated dialogues) may preserve lexical/syntactic validity but fail to capture cultural norms, insurance logistics, or social expectations, reducing ecological validity.

4. Results

• Dataset Classification: The authors successfully mapped 20 unique datasets to their typology.
  • Example: MedDialog (Zeng et al., 2020) is classified as Human Type 1 / Machine Type 1 (real data, machine-transcribed).
  • Example: ACI-bench (Yim et al., 2023) is Human Type 3 / Machine Type 2 (actors role-playing, machine-transcribed).
  • Example: DoPaCo (Chen et al., 2023a) is Human Type 1 / Machine Type 3 (real data used to fine-tune an LLM, which then generates new dialogues).
• Prevalence: There is a clear shift toward Machine Type 3 datasets (LLM-generated) in recent literature, driven by the ease of generation and the difficulty of obtaining real data.
• Evaluation Gap: While many papers claim their synthetic data is "realistic," few provide rigorous evaluation of pragmatic or discursive validity, often relying only on statistical metrics like perplexity.

5. Significance and Future Directions

• Standardization: The typology provides a common language for researchers to explicitly state the provenance and degree of synthesis in their datasets, facilitating better comparison and reproducibility.
• Validity Assessment: It helps practitioners understand the limitations of a dataset. For instance, a "Human Type 3" dataset (screen-written) may be sufficient for training a chatbot on medical terminology but insufficient for training a model on the nuances of patient empathy or turn-taking.
• Contextual Awareness: The paper argues that future work must extend beyond microdata (words/sentences) to consider extra-linguistic factors (cultural context, social norms, healthcare system logistics). Synthetic data that ignores these factors may be statistically sound but clinically invalid.
• Policy Implication: As LLMs become ubiquitous in generating "hybrid" synthetic datasets, the paper calls for rigorous assessment of their representativeness before deployment in high-stakes clinical environments.

In summary, this paper bridges the gap between the technical mechanics of synthetic data generation and the practical requirements of clinical dialogue processing, offering a structured framework to navigate the complexities of "real" vs. "synthetic" in healthcare AI.