A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective

Technical Summary: A Closer Look on Memorization in Tabular Diffusion Model: A Data-Centric Perspective

Problem Statement
While diffusion models have demonstrated superior performance in generating high-quality tabular data compared to traditional GANs and VAEs, they carry significant privacy risks through the inadvertent memorization of exact training samples. Existing mitigation strategies primarily rely on dataset-level data augmentation (e.g., feature mixing) or nearest-neighbor distance-based detection. However, these approaches suffer from two critical limitations: they fail to explain when or which specific samples are memorized during the training process, and they apply mitigation uniformly across the dataset, ignoring the heterogeneous nature of sample-level memorization risks. This lack of granularity hinders the development of proactive, targeted interventions to prevent privacy leakage in sensitive domains such as healthcare and finance.

Methodology
The authors adopt a data-centric perspective to conduct the first fine-grained analysis of memorization dynamics in tabular diffusion models. The methodology proceeds in three stages:

1. Empirical Characterization of Memorization:
   The study quantifies memorization for each real training sample by counting how many generated samples are flagged as its replicas using a relative distance ratio metric ($r(x) < \tau$, where $\tau = 1/3$). The authors define specific training-time events, including "First Memorization," "Forget Events," and "Cumulative Memorization Count." They introduce Mem-AUC (Memorization Area Under Curve) to quantify the intensity and persistence of memorization for individual samples over training epochs.

2. Key Observations:
   Through extensive analysis on datasets like Adult, Default, and Shoppers, the authors identify a heavy-tailed distribution of memorization counts. A small subset of training samples (the "Top" group) is disproportionately memorized compared to the majority. Furthermore, these high-risk samples exhibit distinct temporal dynamics:

   • They are memorized earlier in the training process than non-top samples.
   • They show stronger memorization signals (higher Mem-AUC) during early training stages.
   • They exhibit volatile retention, frequently alternating between memorized and non-memorized states (early forgetting) compared to the more stable Non-Top group.

3. Proposed Solution: DynamicCut and DynamicCutMix:
   Based on the insight that memorization-prone samples can be identified via early training signals, the authors propose DynamicCut, a model-agnostic, two-stage mitigation method:

   • Ranking: During an initial warm-up phase (e.g., the first $T$ epochs), the model tracks the Mem-AUC for each sample.
   • Scoring: A memorization score is computed for each sample based on the mean of its top-k highest Mem-AUC values during this early phase.
   • Pruning: The top fraction ($p$, default 10%) of samples with the highest scores are identified as high-risk and removed from the training set.
   • Retraining: The model is retrained on the filtered dataset.

   Additionally, the authors propose DynamicCutMix (DCM), which integrates DynamicCut with existing feature-level augmentation techniques (TabCutMix) to further reduce memorization while preserving data diversity.

Key Results
The proposed methods were evaluated across multiple datasets (Adult, Default, Shoppers, Cardio, Wilt) and generative models (TabDDPM, CTGAN, TVAE).

• Memorization Reduction: DynamicCut consistently reduces memorization ratios across all architectures. For instance, on the Adult dataset with TabDDPM, it achieved a 13.58% relative reduction in memorization. When combined with augmentation (DCM), reductions reached up to 23.75% on CTGAN for the Adult dataset.
• Data Quality Preservation: Crucially, the removal of high-risk samples did not significantly degrade downstream utility. Metrics such as Shape Score, Trend Score, and Machine Learning Efficiency (MLE) remained stable or improved. The synthetic data maintained high fidelity in marginal distributions and feature correlations.
• Transferability: A significant finding is the cross-model transferability of memorization signals. Labels identifying high-risk samples derived from a diffusion model (TabDDPM) were effective in reducing memorization when applied to filter training data for GANs (CTGAN) and VAEs (TVAE), demonstrating that memorization patterns are robust across different generative architectures.
• Comparison with Baselines: DynamicCut and DCM outperformed standard data augmentation baselines (SMOTE, TabCutMix, TabCutMixPlus) in memorization mitigation while maintaining competitive or superior performance in data quality metrics.

Significance and Claims
The paper claims to provide the first data-centric study revealing that memorization in tabular diffusion models is not uniform but concentrated in a small, early-memorizing subset of data. The significance of this work lies in:

1. Proactive Mitigation: Shifting from reactive, dataset-wide augmentation to proactive, sample-level filtering based on real-time training dynamics.
2. Efficiency: Demonstrating that modest, targeted interventions (removing ~10% of data) can substantially reduce privacy risks without large-scale data pruning or compromising generation quality.
3. Generalizability: Establishing that memorization signals are transferable across different generative model families, allowing for efficient privacy protection even when full training traces of the target model are unavailable.
4. Practicality: Offering a lightweight, model-agnostic framework (DynamicCut) that can be easily integrated into existing training pipelines to enhance the safety of synthetic tabular data generation.

The authors conclude that their approach offers a principled solution to control memorization risk, balancing the fidelity and safety of tabular generative models, while acknowledging limitations regarding the dependency on early-stage patterns and the need for theoretical explanations of why specific samples are prone to memorization.