🤖 machine learning

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Memisis is a unified tool that leverages large language models to orchestrate and evaluate synthetic tabular health data generation, enabling users to specify high-level goals while automatically managing the workflow across multiple synthesizers and metrics to ensure privacy, utility, and fairness.

Original authors: Nitish Nagesh, Mahdi Bagheri, Arshia Harish Puthran, Pengbao Zhou, Muhjaazee Love, Aadi Sharma, Ian Harris, Amir M. Rahmani

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nitish Nagesh, Mahdi Bagheri, Arshia Harish Puthran, Pengbao Zhou, Muhjaazee Love, Aadi Sharma, Ian Harris, Amir M. Rahmani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to train a new AI assistant to help diagnose mental health conditions. You have a massive, real-world notebook full of patient records. But, you can't share that notebook with the AI trainer because it contains private secrets (names, addresses, specific medical histories). If you share it, you break privacy laws.

The Problem: You need a "fake" notebook that looks and acts exactly like the real one, but contains no real people. This is called synthetic data. However, making a fake notebook is tricky. If the fake data is too different from the real data, the AI won't learn anything useful (low utility). If the fake data accidentally learns to treat certain groups of people (like specific races or genders) unfairly, the AI will make biased mistakes (low fairness).

The Solution: Memisis
The paper introduces a tool called Memisis. Think of Memisis as a super-smart, automated project manager for creating these fake notebooks.

Here is how it works, using simple analogies:

1. The "Conductor" (Orchestration)

Usually, making synthetic data is like trying to build a house by asking three different contractors to do the job without talking to each other. One builds the foundation, another paints the walls, and a third tries to install the roof, but they never check if the house is actually livable or safe.

Memisis acts as the orchestra conductor. It doesn't just build the data; it coordinates the whole process. You tell it what you want in plain English (e.g., "Make me a fake dataset that looks like the real one but is fair to everyone"). Memisis then:

Picks the best "builder" (a specific AI model like CTGAN, TVAE, or GaussianCopula).
Tells it how long to work.
Checks the work immediately.

2. The "Taste Testers" (Evaluation)

Memisis doesn't just trust the builder. It uses two specialized "taste testers" to grade the fake data before you ever see it:

The Realism Chef (SDMetrics): This tester checks if the fake data tastes like the real thing. Does it have the same mix of ages, genders, and symptoms? If the fake data looks nothing like the real world, this tester gives it a low score.
The Fairness Judge (Fairlearn): This tester checks if the data is biased. Imagine the fake data is a test for a hiring manager. If the fake data suggests that people of one race are "sick" 3 times more often than people of another race (even if that's not true in reality), the Fairness Judge slams the gavel.

3. The "Scorecard" (Composite Scoring)

Memisis combines these two tests into one final score. It uses a clever rule:

If the data is perfectly realistic but unfair, the score gets a heavy penalty.
If the data is fair but nonsense, the score is low.
The goal is a "Goldilocks" score: data that is both realistic and fair.

The "No Cheating" Rule:
A key feature of Memisis is that the "Judge" (Evaluator) and the "Builder" (Generator) are kept in separate rooms. The Judge cannot whisper to the Builder to "make the numbers look better." They work independently. The Judge only reports back to the "Supervisor" (the main AI), who then decides: "This batch is good, send it to the user," or "This batch is unfair, go back and try again."

What Did They Find?

The team tested Memisis using a real dataset about schizophrenia (a mental health condition) that included race and gender. They tried three different "builders":

GaussianCopula: This builder made data that looked very realistic (91% match), but it was unfair. It made the "Hispanic" group look much sicker than the "White" group in the fake data. Because of this unfairness, its final score dropped.
TVAE: This builder made data that was perfectly "fair" (everyone looked the same), but it was actually broken. It was so uniform that it didn't teach the AI anything useful.
CTGAN: This was the winner. It found the best balance. The data looked realistic enough to be useful, and it treated different groups fairly enough to pass the test.

Why Does This Matter?

Memisis is a tool for researchers and data owners. It lets them say, "I need fake medical data," and the tool handles the complex math, the privacy checks, and the fairness audits automatically. It ensures that when we use AI to help doctors, the AI isn't learning from a biased or broken version of reality.

In short: Memisis is the automated quality control manager that ensures our "fake" medical data is both a good copy of the real thing and a fair copy for everyone.

Technical Summary: Memisis – Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Problem Statement
Synthetic data is increasingly vital in healthcare to create datasets that mirror original data distributions while mitigating privacy concerns. However, generating high-quality synthetic data requires a delicate balance across three critical dimensions: privacy, utility, and fairness. Existing platforms often lack a unified workflow that allows users to easily orchestrate generation, validate distributional fidelity, and evaluate fairness metrics simultaneously. Furthermore, current approaches often rely on manual tuning of "knobs" (e.g., epochs, sample sizes) rather than allowing users to specify high-level generation goals. There is a need for an agentic workflow that can reason about these trade-offs, particularly in domains like healthcare where intersectional bias in diagnosis is clinically documented.

Methodology
The authors present Memisis, an interactive agent-based tool designed to orchestrate and evaluate synthetic data generation for tabular health datasets. The system leverages Large Language Models (LLMs) and state-of-the-art evaluation metrics to create a unified workflow.

Architecture & Workflow:
- Backend Integration: Memisis connects to the Synthetic Data Vault (SDV) to utilize existing synthesizers, specifically CTGAN, TVAE, and GaussianCopula.
- Agent Orchestration: The system employs a LangGraph ReACT (Reasoning and Acting) framework. A Supervisor agent partitions the workflow into a Generator subgraph and an Evaluator subgraph. This separation ensures that scoring mechanisms cannot silently influence the synthesis process.
- User Interaction: Users interact via a Streamlit UI using natural language to specify generation goals (e.g., target distribution, fairness constraints) rather than tuning low-level parameters. The agent translates these goals into specific tool calls.
- Evaluation Paradigm: The system utilizes the Train-on-Synthetic, Test-on-Real (TSTR) paradigm.
  - Utility: Measured using SDMetrics (specifically the SingleTableQualityReport), which aggregates marginal column-shape fidelity and pairwise-correlation fidelity into a score between 0 and 1.
  - Fairness: Measured using Fairlearn, focusing on False Positive Rates (FPR) across protected attributes (e.g., race, gender).
- Composite Scoring: To jointly capture the fairness-utility trade-off, the authors propose a composite score:
  $\text{synth\_score} = \text{quality} \times \text{fairness\_mult}$
  The fairness_mult acts as a penalty. If the ratio of the highest group FPR to the lowest group FPR (max_rel_FPR) exceeds a threshold of 2, the multiplier discounts the quality score proportionally. If the ratio is $\le 2$ , the multiplier is 1.0.
Deployment Stack:
- Server: Python-based server on Streamlit Cloud connected to GitHub.
- LLM: Llama3.2 (running locally via Ollama) serves as the chat model.
- Hardware: NVIDIA RTX 3090 GPU used for training synthesizers.

Key Contributions

Unified Agentic Workflow: Memisis provides a novel interface where users define goals in natural language, and the agent orchestrates the selection of synthesizers, training, and evaluation without requiring manual hyperparameter tuning.
Separation of Concerns: The multi-agent architecture strictly separates the generation process from the evaluation process, preventing feedback loops where scoring metrics might inadvertently bias the generation mechanism.
Composite Metric for Healthcare: The paper introduces a specific composite scoring mechanism that penalizes synthetic data which achieves high distributional fidelity but fails to maintain parity in downstream classification errors (FPR) across demographic groups.
Demonstration on Health Data: The tool is demonstrated on an open-source schizophrenia dataset (OpenML 45040) containing protected attributes (Race, Sex) and clinical labels, showcasing its ability to detect intersectional disparities.

Results
The authors evaluated three synthesizers (CTGAN, TVAE, GaussianCopula) on the schizophrenia dataset (1,000 real training rows, 500 synthetic rows, 20 epochs):

GaussianCopula: Achieved the highest distributional fidelity (SDMetrics quality = 0.91). However, it suffered a significant fairness penalty because the Hispanic group's FPR was 2.67× higher than the White group's FPR (exceeding the 2.0 threshold). This resulted in a lower composite score (0.68).
CTGAN: Produced the best balanced composite score (0.77). It maintained a max_rel_FPR of 1.57 (within the acceptable range), demonstrating that it successfully balanced fidelity with fairness.
TVAE: Achieved a max_rel_FPR of 1.0, but the authors note this reflects a "degenerate downstream classifier" (predicting all positive) rather than true parity. Its composite score was 0.60.

Significance and Claims
The paper claims that Memisis offers a flexible, natural language-driven mechanism for generating synthetic health data, shifting the burden of curation and validation to an agent with human-in-the-loop oversight. The authors emphasize that high distributional fidelity alone is insufficient for healthcare applications; synthetic data must also preserve fairness in downstream tasks. The tool allows researchers and data owners to explore these trade-offs explicitly.

The authors remain modest regarding their current scope, noting that the tool is currently used by a small community of researchers. They acknowledge challenges in architecture finalization, token limits for cloud providers, and coordination among multiple agents. Future work focuses on improving interoperability, deploying on persistent servers, ensuring compliance with standards like HL7/HIPAA, and expanding experiments to more diverse datasets.