📄 health informatics

Generation of Synthetic Data in Health Surveys Using Large Language Models

This study demonstrates that a large language model can generate psychometrically plausible and demographically consistent synthetic data for Peruvian health surveys when conditioned on user personas, though the presence of specific deviations necessitates rigorous validation before such data can be used for policy-making.

Original authors: Villarreal-Zegarra, D., Bellido-Boza, L.

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Villarreal-Zegarra, D., Bellido-Boza, L.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a chef trying to perfect a new recipe for a massive national soup. Usually, to test if the soup tastes right, you have to gather thousands of real people to taste it, which takes a lot of time, money, and effort. You also have to be very careful not to reveal who ate what, to protect their privacy.

This paper is about a new way to test that soup: using a "digital taste-tester" instead of real people.

Here is the story of what the researchers did, broken down simply:

The Big Idea: The "Digital Twin"

The researchers wanted to see if a smart computer program (called a Large Language Model, or LLM) could pretend to be real people answering a health survey. They didn't just ask the computer random questions; they gave it a "user persona."

Think of a "user persona" like a detailed character sheet for a video game. The computer was told: "You are now a 45-year-old woman from Lima who has high blood pressure and feels a bit down." The computer then had to answer the survey questions as if it were that specific person.

The Experiment: The "Fake" Survey

The researchers took data from a real 2016 health survey in Peru (called ENSUSALUD). They used this real data to build their "character sheets." Then, they asked the computer to generate answers for 1,000 of these characters.

They tested the computer on three main things:

Did it stay in character? If the character was a man, did the computer answer like a man? If the character had diabetes, did the computer remember that?
Did the "feelings" make sense? They asked the computer to fill out depression and anxiety scales (like the PHQ-9 and GAD-7). They checked if the answers looked like a real human's pattern of feelings.
Did the numbers look real? They checked if the distribution of answers (how many people said "yes" vs. "no") looked like the real world.

The Results: A Mostly Perfect Impression

The computer did an amazing job, almost like a method actor who never breaks character:

Staying in Character: When the researchers checked if the computer remembered the character's age, gender, and chronic diseases, it was right 99% to 100% of the time. It rarely forgot who it was pretending to be.
The "Feelings" Test: The depression and anxiety scores the computer generated looked very realistic. They had the same internal structure and reliability as real surveys. In fact, the computer's answers about depression correlated strongly with real-life depression scores from other studies.
The "Cost" of the Fake Survey: It was incredibly cheap and fast. The computer took about 5 seconds to answer one question and cost less than a penny per question. If they had to do this for the whole country, it would have taken about 20 hours and cost less than $100.

The Glitches: Where the "Actor" Stumbled

Even though the performance was great, the researchers found a few cracks in the mask:

The "Obesity" Mix-up: The computer tended to overestimate how many people were obese. It seemed to assume that if someone had diabetes, they must also be obese, which isn't always true in real life.
The "Height" Glitch: The computer was good at remembering facts given to it (like age), but when asked to invent new facts (like height), the numbers didn't follow a natural, bell-curve pattern. They looked a bit "off."
The "Suicide" Question: One specific question about suicidal thoughts had a lot of missing answers. The computer seemed to hesitate or skip this sensitive topic more often than real people do.

The Bottom Line

The researchers concluded that AI can be a very useful "stand-in" for real people when designing and testing health surveys. It can help researchers check if their questions make sense before they go out to the real world, saving time and money while protecting privacy.

However, they warned that you can't just trust the AI blindly. Because the computer made small mistakes (like overestimating obesity or skipping sensitive questions), you can't use these "fake" answers to make final decisions about public health policy yet. It's a powerful tool for practice and planning, but it still needs a human supervisor to check the work before it's used for real-world decisions.

In short: The computer is a brilliant actor that can mimic a crowd of people for a rehearsal, but it's not quite ready to replace the real audience for the final show.

Technical Summary: Generation of Synthetic Data in Health Surveys Using Large Language Models

Problem Statement
Access to high-quality, anonymized health survey data is often restricted due to privacy concerns, ethical constraints, and logistical barriers, particularly in low- and middle-income countries. While synthetic data generation using Artificial Intelligence (AI) offers a solution to reduce costs and expand data access, existing methods (e.g., Variational Autoencoders, GANs) often struggle to maintain the contextual coherence and psychometric validity required for complex health surveys. Furthermore, there is a lack of systematic evaluation regarding the ability of Large Language Models (LLMs) to generate plausible, representative synthetic responses for national health surveys while preserving the statistical properties of latent variables like depression and anxiety scales.

Methodology
The study employed a cross-sectional design utilizing secondary data from the 2016 National Health Satisfaction Survey (ENSUSALUD) in Peru, specifically targeting ambulatory health service users.

Data Source & Persona Construction: Researchers extracted sociodemographic and clinical variables (sex, age, chronic conditions) from the original dataset to construct narrative "user personas." These profiles were systematized into CSV files to serve as the ground truth for the AI.
Model Architecture: The study utilized the GPT-OSS-20B, a 20-billion-parameter open-access LLM. The model was executed locally on a server using the Ollama framework with a temperature of 0.0 to ensure deterministic outputs. No fine-tuning or Retrieval-Augmented Generation (RAG) was applied; the model relied solely on its pre-trained knowledge and the provided prompts.
Prompt Engineering: The experimental design used a two-level interaction: a system message defining the user persona (e.g., "simulate being a 45-year-old female with hypertension") and a user message containing specific survey questions. The system enforced strict constraints, requiring categorical responses to match predefined options and numerical responses to fall within logical ranges. Invalid outputs triggered automatic retries; persistent failures were classified as missing data.
Variables Generated: The study generated responses for:
- Consistency Variables: Sex, age, and presence of chronic diseases (hypertension, diabetes, obesity, depression).
- Latent Variables: Full versions of the Patient Health Questionnaire (PHQ-9, PHQ-8, PHQ-2) and the Generalized Anxiety Disorder scale (GAD-7, GAD-2), which were not fully available in the original ENSUSALUD dataset.
- Numerical/Categorical Variables: Height, weight, IQ, and sleep hours.
Evaluation Metrics:
- Consistency: Accuracy, precision, recall, F1-score, and AUC were calculated to measure alignment between the generated response and the persona's profile.
- Psychometric Validity: Confirmatory Factor Analysis (CFA) using Weighted Least Squares Mean and Variance Adjusted (WLSMV) estimators and polychoric matrices assessed the structural validity of PHQ-9, PHQ-8, and GAD-7. Internal consistency was measured via Cronbach's $\alpha$ and McDonald's $\omega$ .
- Distribution & Stability: Normality was tested via the Jarque–Bera test. Stability was assessed by correlating synthetic measures with real-world PHQ-2 and EQ-5D scores from the original dataset.

Key Results

Profile Consistency: The LLM demonstrated exceptional consistency with the user personas. Accuracy, precision, recall, and F1-scores for sex, age, and chronic disease status were near 1.00. Specifically, sex and chronic disease status achieved 100% consistency, while obesity and hypertension showed slight deviations (98.6% and 97.3% consistency, respectively). Overall, 95.9% of cases were fully consistent across all variables.
Psychometric Properties: The synthetic data for PHQ-9, PHQ-8, and GAD-7 exhibited optimal model fit indices (CFI and TLI > 0.99; SRMR < 0.02; RMSEA < 0.04) and high internal consistency ( $\alpha$ and $\omega$ > 0.93).
Correlation with Real Data: Synthetic measures showed strong, significant positive correlations with real PHQ-2 scores (e.g., synthetic PHQ-9 vs. real PHQ-2: $r = 0.806$ ) and negative correlations with EQ-5D quality-of-life scores, mirroring expected clinical patterns.
Distributional Anomalies: While weight followed a parametric distribution, height did not exhibit the expected normal distribution. Additionally, single-item symptom scores displayed implausible distributions with pronounced central peaks. Notably, Item 9 of the PHQ-9 (suicidal ideation) had a high missing value rate (30.1%) compared to other items.
Bias Detection: A sensitivity analysis revealed an overestimation of obesity prevalence in the synthetic group ( $p = 0.012$ ), which the authors attributed to the model assuming comorbidities (e.g., linking diabetes to obesity) rather than strictly adhering to the profile.

Significance and Claims
The authors claim that this study is the first to simulate responses to a national health survey and evaluate the psychometric properties of such data generated by an LLM. The primary significance lies in demonstrating that LLMs, when conditioned on realistic user personas, can generate synthetic data that preserves high coherence with demographic and clinical characteristics and maintains adequate psychometric properties for depression and anxiety scales.

The paper posits that this approach offers a viable strategy for:

Instrument Development: Piloting and refining questionnaires (e.g., identifying confusing wording) before field implementation.
Data Accessibility: Generating anonymized datasets that resemble real surveys, facilitating research in contexts where access to primary data is restricted.
Cost and Time Efficiency: Reducing the resources required for preliminary studies and hypothesis testing.

Limitations and Caveats
The authors maintain a modest stance regarding the utility of the data for inferential purposes. They explicitly note that:

The study did not incorporate complex survey design weights (stratification, clustering), assuming a uniform distribution of simulated cases.
The model was not fine-tuned for the specific Peruvian context, relying instead on general pre-training.
Deviations in distribution (e.g., height, obesity overestimation) and missing values in sensitive items (suicidal ideation) indicate that the data is not a perfect replica of reality.
The authors conclude that while the method is promising for survey design and planning, rigorous validation and bias control are essential before using such synthetic data for public policy or clinical inference. The data should not replace empirical collection but rather complement it.

The Big Idea: The "Digital Twin"

The Experiment: The "Fake" Survey

The Results: A Mostly Perfect Impression

The Glitches: Where the "Actor" Stumbled

The Bottom Line

More like this