Quantifying Membership Disclosure Risk for Tabular Synthetic Data Using Kernel Density Estimators

This paper proposes a practical Kernel Density Estimator-based method to quantify membership disclosure risk in tabular synthetic data by modeling nearest-neighbor distances, demonstrating through empirical evaluation that it outperforms existing baselines in accuracy and efficiency without requiring computationally expensive shadow models.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Picture: The "Fake ID" Problem

Imagine you work for a hospital. You have a massive database of patient records (real data) that is incredibly valuable for research but contains sensitive secrets like HIV status or mental health history. You can't share the real data because it violates privacy laws.

So, you use a smart computer program to create Synthetic Data. Think of this as a "digital twin" or a "fake ID" for the entire population. It looks and acts exactly like the real data, but none of the people in it actually exist. It's safe to share, right?

The Problem: Even though the people are fake, a clever hacker might still be able to figure out if a specific real person (like your neighbor, Bob) was part of the original group used to teach the computer how to make the fakes. If the hacker can say, "Yes, Bob was in the training data," they might learn something sensitive about Bob (like, "Oh, Bob has a rare disease"). This is called a Membership Inference Attack (MIA).

The Old Way: The "Shadow Puppet" Show

Previously, to check if your fake data was safe, researchers used a method called Shadow Modeling.

• The Analogy: Imagine you want to test if your fake ID is good. To do this, you hire a team of actors to create hundreds of their own fake IDs based on the same rules. Then, you hire a detective to try to guess which IDs are real and which are fake.
• The Downside: This is incredibly slow, expensive, and requires a lot of computing power. It's like hiring an entire movie production crew just to test one prop.

The New Way: The "Distance Detective" (KDE)

The authors of this paper propose a much faster, smarter way to check for these leaks. They call it a Kernel Density Estimator (KDE) approach.

The Analogy: The "Closest Neighbor" Game
Imagine you have a bag of real marbles (Real Data) and a bag of fake marbles (Synthetic Data).

1. The Setup: You take a specific marble (let's call it "The Suspect").
2. The Measurement: You measure the distance between "The Suspect" and its closest neighbor in the bag of fake marbles.
3. The Logic:
   • If the Suspect is very close to a fake marble, it's likely the Suspect was used to make that fake marble. (High risk of a leak).
   • If the Suspect is far away from all fake marbles, it's likely the Suspect was never part of the group. (Low risk).

The Innovation:
Old methods just drew a line in the sand: "If the distance is less than 5 inches, it's a leak. If more, it's safe." This is a "Yes/No" answer.

The authors' new method uses KDE to draw a smooth probability curve instead of a hard line.

• The Analogy: Instead of a stop sign, imagine a thermometer.
  • "At 2 inches, there is a 90% chance this person was in the training data."
  • "At 4 inches, there is a 40% chance."
  • "At 6 inches, there is a 5% chance."

This gives data owners a nuanced risk score rather than a simple pass/fail. It tells them how confident they can be that their data is safe.

The Two Types of "Hacks" Tested

The paper tests this method against two types of attackers:

1. The "God Mode" Attacker (True Distribution Attack):

   • Scenario: The attacker knows exactly who was in the original training data and who wasn't. They have the answer key.
   • Result: This is the "worst-case scenario" test. It tells us the absolute maximum risk possible.

2. The "Realistic" Attacker (Realistic Attack):

   • Scenario: The attacker doesn't have the answer key. They only have a public dataset that looks similar to the training data (like a public census). They have to guess who is who based on how close the data points are.
   • Result: This is the test that matters most for real life. Surprisingly, the authors found that sometimes this "guessing" attacker performs better than the "God Mode" attacker in specific situations, proving that even without perfect info, the risk is real.

Why This Matters (The Takeaway)

1. It's Fast: You don't need to train hundreds of shadow models. You just measure distances and run a quick math formula. It's like using a metal detector instead of digging up the whole beach to find a coin.
2. It's Precise: It gives you a probability (a percentage) rather than a guess. This helps data custodians (the people holding the data) decide: "Is the risk low enough to release this data to researchers?"
3. It Reveals Hidden Dangers: Sometimes, the average risk looks low (e.g., "50% accuracy"), which sounds safe. But this new method looks at the "worst-case" scenarios (low false alarms) and finds that for specific individuals, the risk is actually huge. It's like saying, "The average weather is sunny, but there's a 100% chance of a tornado for your specific house."

Summary

This paper introduces a fast, mathematical "thermometer" for synthetic data. Instead of asking "Is this data safe?" (Yes/No), it asks "How likely is it that a specific person's secret is leaking?" This allows companies and hospitals to release synthetic data with much greater confidence, knowing exactly where the privacy cracks are before they share the data with the world.

Technical Summary

Here is a detailed technical summary of the paper "Quantifying Membership Disclosure Risk for Tabular Synthetic Data Using Kernel Density Estimators."

1. Problem Statement

The increasing adoption of synthetic data in sensitive domains (healthcare, finance, demographics) aims to preserve utility while protecting privacy. However, synthetic datasets are not immune to Membership Inference Attacks (MIAs), where adversaries attempt to determine if a specific individual's record was part of the original training data used to generate the synthetic dataset.

Existing methods for assessing MIA risk have significant limitations:

• Shadow Modeling: State-of-the-art approaches train multiple "shadow" generative models and meta-classifiers. While effective, this is computationally expensive and impractical for large-scale, dynamic datasets (e.g., billions of records).
• Distance-Based Thresholding: Simpler methods calculate nearest-neighbor distances between synthetic and real data, applying a hard threshold to classify members vs. non-members. These methods produce binary labels rather than probabilistic scores, preventing comprehensive risk analysis (e.g., ROC curves) and often underestimating worst-case privacy leakage.

2. Methodology

The authors propose a non-parametric, distance-based framework using Kernel Density Estimators (KDEs) to model the distribution of nearest-neighbor distances. This approach generates probabilistic membership predictions, enabling robust statistical evaluation.

The methodology involves two distinct attack models:

A. True Distribution Attack (Privileged Access)

This model assumes the data custodian has access to the true training set ($R$) and unseen holdout data ($U$).

1. Data Construction: An attack dataset ($D_{attack}$) is formed by combining $R$ (labeled as members) and $U$ (labeled as non-members).
2. Distance Calculation: Gower's distance is computed between every record in $D_{attack}$ and its nearest neighbor in the synthetic dataset ($S$).
3. KDE Modeling: Two separate KDEs are fitted to the distance distributions:
   • $KDE_{member}$: Distances for records known to be in the training set.
   • $KDE_{non-member}$: Distances for records known to be outside the training set.
4. Probabilistic Inference: Using Bayes' Theorem, the probability that a query record is a member given its distance $d$ is calculated as:
   $$P(member|d) = \frac{KDE_{member}(d)}{KDE_{member}(d) + KDE_{non-member}(d)}$$
   This allows for classification based on a probability threshold (e.g., 0.5) and enables full ROC curve analysis.

B. Realistic Attack (Adversarial Perspective)

This model simulates an adversary who does not have access to true membership labels but possesses auxiliary data from the same population.

1. Label Approximation: Without ground truth, the adversary partitions the auxiliary data into "supposed members" (distances below a threshold $\tau$) and "supposed non-members" (distances above $\tau$).
2. KDE Modeling: Separate KDEs are fitted to these "supposed" groups.
3. Evaluation: The model uses Equation 1 to estimate membership probabilities. While this introduces label noise, the paper demonstrates that under certain conditions (specifically higher thresholds), this attack can outperform traditional hard-threshold methods.

3. Key Contributions

1. KDE-Based Privacy Scoring Framework: A novel method to quantify membership disclosure risk by modeling distance distributions rather than relying on hard thresholds or expensive shadow models.
2. Probabilistic Output & ROC Analysis: Unlike previous distance-based methods, this approach provides membership probabilities, enabling the generation of Receiver Operating Characteristic (ROC) curves. This is crucial for evaluating worst-case scenarios (high True Positive Rates at low False Positive Rates), which average metrics like F1 or Accuracy often mask.
3. Practical Efficiency: The method eliminates the need for computationally intensive shadow model training, making it feasible for data custodians to perform post-generation risk assessments on large, dynamic datasets.
4. Dual Attack Models: The introduction of both a "True Distribution" (upper bound) and "Realistic" (practical) attack provides a comprehensive view of risk from both the data holder's and the adversary's perspectives.

4. Experimental Results

The authors evaluated their method on four real-world datasets (MIMIC-IV, UK Census, Texas-100X, Nexoid) using six synthetic data generators (CTGAN, ADS-GAN, DPGAN, TabDDPM, TVAE, Bayesian Network).

• Performance vs. Baselines: The KDE-based approach consistently achieved higher F1 scores than the prior distance-partitioning baseline (Method 1), particularly in the "Realistic Attack" scenario at higher distance thresholds.
• Generator Vulnerability:
  • Bayesian Networks were found to be the most vulnerable across most datasets, showing high accuracy and F1 scores.
  • TVAE showed mixed results: high vulnerability on MIMIC-IV (88.9% accuracy) but low vulnerability on UK Census (near-baseline accuracy).
• ROC Analysis Insights:
  • Average metrics (Accuracy/F1) often suggested low risk (e.g., TVAE on UK Census had ~50% accuracy).
  • However, Log-ROC curves revealed severe worst-case vulnerabilities, showing True Positive Rates (TPR) up to $10^5$ times higher than False Positive Rates (FPR) at low FPRs. This highlights that average-case metrics can dangerously underestimate privacy leakage.
• Realistic Attack Anomaly: In some cases (e.g., Texas-100X), the "Realistic Attack" outperformed the "True Distribution Attack" at specific thresholds. This occurred because the "supposed member" grouping in the realistic attack inadvertently captured a higher ratio of True Positives to False Positives as thresholds increased.

5. Significance and Conclusion

This paper provides a critical tool for data custodians to assess the privacy safety of synthetic data before release.

• Post-Generation Assessment: It allows for risk quantification using only the training data, the synthetic data, and a reference dataset, without needing to retrain complex models.
• Risk Characterization: By shifting from binary classification to probabilistic inference, the method uncovers "hidden" risks where average metrics appear benign but worst-case leakage is high.
• Scalability: The computational efficiency of KDEs compared to shadow modeling makes this approach viable for enterprise-level data governance.

The authors conclude that while the method offers a robust framework, future work should address relaxing the assumption of balanced datasets (to reflect real-world population ratios) and exploring theoretical guarantees for the distance-to-probability mapping. The code and datasets are publicly available to facilitate further research.