Metadata-based Methods to Generate Synthetic Health Data: A Scoping Review
This scoping review identifies existing metadata-based methods for generating synthetic health data and highlights critical gaps in their standardized evaluation, ethical guidance, and specific applicability to code development in federated analysis contexts.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery, but the police have locked the crime scene. You can't touch the real evidence because it contains private information about real people, and the rules say you can't take it home. This is exactly the situation for researchers studying health data. They need to test their computer scripts and theories on "Electronic Health Records" (EHRs)—digital files of patient visits, diagnoses, and medications—but they can't use the real files without breaking privacy laws. So, they need a stand-in. They need "synthetic data": fake patient records that look and act just like the real ones, but belong to no one.
Usually, to make these fake records, scientists try to copy-paste the real data into a complex machine learning robot. But this robot is a "black box": it works in secret, and if it accidentally leaks a real person's secret, it's a disaster. Plus, in many modern research setups called "federated analysis," the real data is locked in different hospitals that never share their files. The robot can't work if it can't see the data. This is where a different approach comes in: "metadata-based" methods. Instead of looking at the messy, private details of individual patients, these methods use a "recipe" or a "blueprint." They look at the summary statistics (like "50% of patients are over 50") and public knowledge to build fake patients from scratch, without ever needing to see the real, private files.
This paper is a scoping review, which is like a giant map-making expedition. The authors, a team of researchers from Utrecht, wanted to see what tools exist for building these "blueprint-only" fake health records. They didn't just look at the tools; they also looked at the rulebooks. They asked: How do we know if the fake data is any good? Are there rules to make sure it's safe? And most importantly, can we use these fake records to test our code before we ever touch the real data?
The team scoured the scientific literature and found 23 relevant articles. Twelve of these described specific methods for making the data, and eleven were reviews discussing how to judge the data and the legal rules around it. Here is what they discovered:
The Tools: Building with Blueprints
The researchers found six main ways to generate this "blueprint" data.
- Synthea: This is the most famous tool. It's like a video game character creator. It starts with a population based on public census data (like the US Census). Then, it simulates a patient's life step-by-step. It uses a "Monte Carlo" method (a fancy way of saying it rolls digital dice based on probabilities) to decide if a patient gets an ear infection, how old they are when they do, and what medicine they get. It builds the patient's entire medical history from public knowledge and user settings, never needing a real patient file.
- OSIM: This tool is designed to speak the language of the "OMOP Common Data Model," a standard format for health data. It takes summary tables (like "how many people have diabetes") and builds a fake dataset that fits that standard.
- SASC: A web-based app specifically for COVID-19 data. It takes summary tables and generates fake patient records with lab results and outcomes.
- Davis et al.'s Method: This one is unique because it simulates "learning." It imagines that doctors get better at treating patients over time. It builds fake data that includes this improvement curve, which is great for testing how new treatments perform in the real world.
- ODM Generator: This tool is a bit different. It doesn't try to make the data look statistically real; it just makes sure the data follows the correct "grammar" (syntax) for clinical trials. It's like a spell-checker for data structure.
- Barr et al.'s Method: A new approach using a Large Language Model (like GPT-4o). The researchers gave the AI a list of statistics (e.g., "average age is 45") and asked it to write a table of fake patients. It's a very new, experimental way to do it.
The Problem: We Have Tools, But No Rulebook
Here is the twist in the story. While the authors found these tools, they found a massive gap in how we check if they work.
- The "Fidelity" Trap: Almost every time these methods were tested, the researchers only checked "fidelity." This means they asked, "Does the fake data look like the real data?" They compared averages and charts. But they rarely checked "utility." Utility asks, "Does this fake data actually help me write better code or solve a problem?"
- The Missing Link: The paper explicitly notes that no one has really tested these metadata-based methods for their specific use case: code development in federated analysis. We don't know if the fake data is "good enough" to test a script that will later run on real, locked-down hospital data.
- The Evaluation Chaos: The authors looked at eleven other reviews about how to judge synthetic data. They found a mess of different definitions. Some people call it "fidelity," others "resemblance" or "similarity." Some include "fairness" (making sure the fake data doesn't bias against minorities), while others ignore it. There is no single, agreed-upon rulebook.
- The Legal Grey Area: The review found very little discussion on the legal and ethical side. While we have rules for real data, the rules for this "fake" data are still fuzzy. The authors point out that major regulators like the FDA and EMA haven't given clear guidance yet.
The Verdict
The paper concludes that we have a toolbox full of interesting ways to build fake health data without seeing real patients. However, we are flying blind. We haven't standardized how to test these tools, and we haven't figured out the specific rules for using them to test computer code in secure, federated environments.
The authors suggest that future research needs to stop just asking "Does it look real?" and start asking "Does it work for the job?" They call for a new, standardized framework that matches the method of generation with the intended use. Until then, using these tools to build code for real-world health research is a bit like building a model airplane without knowing if the wings will actually hold up in a real wind tunnel. The potential is huge, but the safety checks are still being written.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.