From Statistical Fidelity to Clinical Consistency: Scalable Generation and Auditing of Synthetic Patient Trajectories

This paper presents an integrated pipeline combining knowledge-grounded generative modeling with automated LLM-based auditing to produce clinically consistent, privacy-preserving synthetic patient trajectories that overcome the limitations of existing methods by eliminating clinical inconsistencies while maintaining high statistical fidelity and downstream utility.

The Gist

Imagine you are a doctor trying to teach a new medical student how to diagnose patients. You want to show them thousands of real patient files to learn from. But there's a problem: privacy laws say you can't share real people's medical records. It's like trying to teach someone to drive using a real car, but the car is locked in a vault because the owner doesn't want their personal history exposed.

For years, scientists tried to solve this by creating "Synthetic Patients"—fake medical records made by computers. Think of these as "clones" of real patients that look and act the same but don't belong to anyone.

However, there was a major catch with these old clones. They were like bad actors in a movie: they looked the right age and had the right clothes (statistics), but their dialogue made no sense. They might say, "I'm a 70-year-old man," and then immediately say, "I just gave birth to twins." The computer got the numbers right, but the story was broken.

This paper introduces a new system called Coogee (named after a bird, implying it's a natural, organic creation) that fixes this problem. Here is how it works, broken down into simple steps:

1. The "Knowledge-First" Writer (Generation)

Old computer models were like students who only memorized flashcards. If they saw the word "Diabetes" and the word "Insulin" often, they just threw them together randomly.

Coogee is different. It's like a medical student who has read every textbook.

• The Analogy: Before writing a single fake patient record, Coogee studies a massive "Medical Knowledge Graph" (a giant map of how diseases, drugs, and body parts connect).
• The Result: Instead of just guessing, it understands that if a patient has Type 2 Diabetes, they might need insulin, but they won't need a pregnancy test. It builds the patient's story from the ground up, ensuring every medical term is real and fits together logically.

2. The "Strict Editor" (Auditing)

Even with a smart writer, mistakes happen. Sometimes the computer gets carried away and writes a scene where a patient takes a drug that is dangerous for their specific condition.

This is where the second step comes in: The Automated Editor.

• The Analogy: Imagine a very strict, super-smart editor (an AI trained like a Chief Medical Officer) who reads every single fake patient file before it's released.
• The Job: This editor checks for "plot holes."
  • Did a male patient get a hysterectomy? Cut! (Deleted).
  • Did a patient get a drug that kills people with their specific heart condition? Cut!
  • Does the timeline make sense? (e.g., Did the surgery happen before the diagnosis?) Cut!

The paper found that without this editor, about 45-60% of the fake records had these logical errors. With the editor, the records became almost indistinguishable from real ones.

3. The "Double-Check" (Testing)

The researchers tested this system in three ways:

1. The Math Check: They compared the fake data to real data using statistics. The fake data matched the real data almost perfectly (like a perfect photocopy).
2. The Human Check: They asked real doctors to read the fake records. At first, the doctors could tell which ones were fake because of the "plot holes." But after the "Strict Editor" fixed the records, the doctors couldn't tell the difference anymore.
3. The "Usefulness" Check: They trained a new AI on the fake data and tested it on real patients. The AI performed just as well as if it had been trained on real data. This proves the fake data is useful for training future medical AI.

Why Does This Matter?

• Privacy: We can now share "fake" patient data across the world without ever risking a real person's privacy. It's like sharing a recipe without revealing the chef's secret family history.
• Safety: It prevents the "bad actor" problem where AI learns from nonsense data.
• Speed: Instead of hiring hundreds of doctors to manually check millions of fake records (which would take forever), this system uses AI to do the checking in seconds.

The Big Takeaway

The paper argues that being statistically accurate isn't enough. A fake patient record needs to be clinically consistent. You can't just have the right numbers; the story has to make sense.

Coogee is the first system to successfully combine a "smart writer" (who knows the medical rules) with a "strict editor" (who checks the logic), creating a massive library of fake patients that are safe, private, and perfectly realistic for training the next generation of medical AI.

Technical Summary

1. Problem Statement

The generation of synthetic Electronic Health Records (EHRs) is critical for privacy-preserving data sharing, digital twin simulations, and validating clinical AI. However, existing methods face a fundamental "consistency gap":

• Statistical vs. Clinical Fidelity: Current deep learning models (e.g., GANs, VAEs, Transformers) often achieve high statistical fidelity (matching aggregate distributions of real data) but fail to capture clinical consistency. They frequently produce records that are statistically probable but clinically impossible (e.g., a male patient with pregnancy-related procedures, or medications prescribed without a corresponding diagnosis).
• Structural Integrity Issues: Many models use sub-word tokenization strategies that fragment medical codes (e.g., splitting E11.9 into E11 and .9), leading to the generation of non-existent, "hallucinated" medical concepts when tokens are recombined.
• Scalability of Validation: Manual review by clinicians is the gold standard for detecting clinical inconsistencies but is subjective, time-consuming, and impossible to scale to large synthetic datasets.

2. Methodology: The Coogee Framework

The authors propose Coogee, a two-step integrated pipeline designed to generate and audit synthetic patient trajectories at scale.

A. Knowledge-Grounded Generative Model

Instead of relying purely on statistical co-occurrence, the generative component is anchored in biomedical knowledge to ensure structural integrity.

• Atomic Tokenization: Unlike sub-word approaches, Coogee treats every distinct clinical concept (ICD-10, ATC, Lab codes) as a single, indivisible token. This prevents the generation of invalid codes. The vocabulary covers ~32,000 distinct concepts, including the "long tail" of rare diseases.
• Knowledge-Aware Embeddings: The model integrates two layers of context via the PrimeKG knowledge graph and ClinicalBERT:
  1. Structural Knowledge: Uses a Relational Graph Convolutional Network (RGCN) to encode biological relationships (e.g., drug-disease pathways).
  2. Semantic Knowledge: Encodes textual definitions of medical codes.
  • These embeddings are frozen during training to preserve ontological structure, while the model learns to project and gate these representations.
• Architecture: A decoder-only Transformer (50M parameters, 13M trainable) featuring:
  • Rotary Position Embeddings (RoPE) for better temporal generalization.
  • Grouped-Query Attention (GQA) and SwiGLU activations for efficiency.
  • Discrete Time-Gap Tokens: Explicitly models irregular time intervals between events (minutes to months).
  • Constrained Decoding: Enforces structural rules (e.g., lab values must map to valid quantile bins).

B. Scalable Automated Auditing

To address the consistency gap, Coogee employs a Large Language Model (LLM) auditor to filter out clinically invalid records.

• LLM Auditor: Utilizes a local instance of Qwen3-30B prompted to act as a Chief Medical Officer.
• Evaluation Dimensions: The auditor scores records (1–10 scale) based on:
  1. Demographic Alignment: Consistency between biological attributes (sex, age) and clinical events.
  2. Clinical Reasoning: Adherence to causal chains (e.g., diagnosis $\to$ treatment).
  3. Temporal Plausibility: Feasibility of event sequences and time gaps.
• Filtering: Only trajectories scoring $\ge$ 7/10 ("Mostly Realistic" to "Indistinguishable") are retained.

3. Key Contributions

1. Shift in Paradigm: Demonstrates that statistical fidelity is necessary but insufficient; clinical consistency requires a framework-level approach combining knowledge-grounded generation with automated auditing.
2. Atomic Tokenization & Long-Tail Coverage: Successfully models nearly 32,000 clinical concepts (vs. ~9,000 in prior benchmarks) without code fragmentation, ensuring every generated token is a valid medical concept.
3. Scalable Auditing: Introduces an automated LLM-based auditing module that acts as a scalable proxy for human clinicians, effectively filtering logical inconsistencies that probabilistic models miss.
4. Privacy Preservation: Validates that the synthetic data does not leak membership information, performing indistinguishably from random guessing in membership inference attacks.

4. Results

The framework was evaluated using the MIMIC-IV dataset (180,712 real patients), generating 18,071 synthetic records.

• Statistical Fidelity:
  • High correlation ($R^2 = 0.99$) between real and synthetic event probabilities.
  • Minimal mean bias (0.00) in Bland-Altman analyses.
  • Accurate reproduction of complex syndromes (e.g., Cardio-Renal Syndrome) and temporal distributions (Length of Stay, visit intervals).
• Clinical Consistency (The "Gap"):
  • Before Auditing: A random sample of synthetic records showed 45–60% inconsistency when reviewed by clinicians (e.g., logical violations like male pregnancy).
  • After Auditing: The automated filter significantly reduced the distinguishability between real and synthetic data. Cohen's $d$ effect size dropped from 1.60 to 0.67 for rule-checking clinicians and 1.13 to 0.71 for LLM auditors.
  • Human reviewers found it significantly harder to distinguish audited synthetic data from real data ($p=0.57$ for narrative reviewers).
• Downstream Utility (TSTR Protocol):
  • Models trained on audited synthetic data achieved performance parity or superiority on real-world test sets for tasks like phenotyping (AUROC 0.91 vs 0.96), mortality prediction, and readmission.
• Privacy:
  • Membership Inference Attack (MIA) F1-score was 0.51 (random guessing baseline), confirming no data leakage.

5. Significance and Implications

• Trustworthy Synthetic Data: Coogee bridges the gap between "statistically plausible" and "clinically sound," making synthetic data viable for high-stakes applications like clinical trial emulation and digital twins.
• Standard for Governance: The paper argues that automated auditing should become a standard component of synthetic data pipelines to ensure trust and governance, replacing reliance on aggregate statistics alone.
• Scalability: By replacing manual clinical review with LLM-based auditing, the framework enables the generation of massive, high-quality synthetic cohorts that were previously infeasible to validate.
• Future Directions: The authors suggest future work in lifelong patient modeling (state-space models), retrieval-augmented auditing (connecting LLMs to external guidelines), and multimodal data integration.

In summary, Coogee represents a significant advancement in synthetic health data by prioritizing clinical logic alongside statistical accuracy, ensuring that generated patient trajectories are not just mathematically correct, but medically valid.