PuckTrick: A Library for Making Synthetic Data More Realistic

1. Problem Statement

The increasing reliance on machine learning (ML) for decision-making requires high-quality training data. However, access to real datasets is often restricted due to privacy concerns, proprietary limitations, and data incompleteness. Consequently, Synthetic Data Generation (SDG) has become a primary alternative.

While SDG successfully replicates the statistical properties of real data, it suffers from a critical deficiency: synthetic data is often "too clean." Real data is inherently imperfect and contains missing values, noise, outliers, misclassified labels, and structural inconsistencies. Models trained on idealized, error-free synthetic data often fail to generalize when deployed in real-world scenarios where such imperfections are common. There is a lack of systematic tools to introduce controlled, realistic errors into synthetic datasets to test and improve model robustness.

2. Methodology: The PuckTrick Library

To bridge this gap, the authors developed PuckTrick, a Python library designed to systematically contaminate synthetic datasets with controlled errors. The library operates in two distinct modes:

• New Mode: Adds a predefined percentage of errors to a clean dataset.
• Extended Mode: Further corrupts an already contaminated dataset, enabling the stepwise introduction of errors while maintaining a target error distribution.

The library supports six specific categories of data errors applicable at the data level or feature level, covering various data types (categorical, numerical, boolean, date/time):

1. Missing Data: Inserting "null" values to simulate sensor failures or incomplete capture.
2. Noisy Data: Random distortions (e.g., Gaussian noise for continuous features, value swapping for categorical integers, synthetic category generation for strings).
3. Outliers: Generating extreme data points using the 3-Sigma method (for continuous/integer data) or by introducing unknown categories (for categorical data).
4. Incorrect Label Classification: Reversing binary labels or randomly reassigning multi-class labels.
5. Duplicated Data: Random or targeted row duplication to simulate data leaks or oversampling.
6. Imbalanced Data: (Implied by duplication and class manipulation) to test model bias.

3. Experimental Setup

The authors evaluated PuckTrick using a dedicated pipeline with real financial data:

• Datasets: Five datasets containing financial metrics for over 4,000 US stocks from 2014–2018. The target variable was a binary classification of stock performance (price variation > 0).
• Synthetic Generation: The GReaT algorithm (using a distilgpt2 LLM) was used to generate synthetic datasets that preserved the statistical properties of the original data.
• Contamination: 61 synthetic datasets were created, including a baseline (clean) and datasets contaminated with specific error types (noise, missing values, outliers, labels) at a fixed error rate of 30%.
• Models: 10 diverse ML algorithms were trained on these datasets, including:
  • Tree-based: Extra Trees (ET), Random Forest (RF), Decision Tree (DT).
  • Linear/Kernel: SVM, Logistic Regression (LR), Linear/Quadratic Discriminant Analysis (LDA/QDA).
  • Others: K-Nearest Neighbors, Naive Bayes, Multilayer Perceptron (MLP).
• Evaluation: The models were tested on the original real test set. Performance was measured using Accuracy, F1-Score, Precision, Recall, and AUC.

4. Key Results

The experiments compared models trained on purely synthetic data with those trained on PuckTrick-contaminated synthetic data.

• Overall Performance: Models trained on contaminated synthetic data generally performed better or equally well compared to those trained on clean synthetic data. This confirms that exposure to realistic imperfections during training improves model generalization.
• Algorithm-Specific Insights:
  • Tree-based Models (ET, RF): Showed the most significant resilience and improvement, particularly with missing data and outliers. Extra Trees (ET) consistently outperformed other models in handling incomplete data.
  • Linear/Kernel Models (SVM, LR): SVM showed the highest improvement when trained on noisy data, suggesting that noise acts as a form of regularization helping SVMs find optimal decision boundaries.
  • Incorrect Label Classification: Results were highly model-dependent. While some models (like LDA) handled label noise well in certain years, others struggled, indicating the need for additional correction mechanisms (e.g., semi-supervised learning).
• Quantitative Impacts:
  • Noise: 100% of models maintained accuracy in the 2014 dataset, with 90% showing improvement.
  • Missing Data: 82% of models maintained or improved accuracy.
  • Outliers: 70–79% of models maintained or improved accuracy.
  • Label Errors: 90% maintained accuracy, but only 10–20% achieved direct gains, underscoring the difficulty of this specific error type.

5. Key Contributions

1. PuckTrick Library: The first systematic, open-source Python library specifically designed to introduce controlled, realistic imperfections into synthetic datasets.
2. Dual-Mode Operation: Unique ability to both initialize errors in clean data and progressively corrupt already contaminated data.
3. Empirical Validation: Comprehensive evidence that "dirty" synthetic data is superior to "clean" synthetic data for training robust ML models.
4. Benchmarking Tool: Provides a standardized way to evaluate data cleaning algorithms and model resilience against specific error types.

6. Significance and Future Work

The paper states that synthetic data must mimic real-world imperfections to be truly useful for training ML models at production level. By training on data containing noise and missing values, models become more robust and adaptable to the unpredictability of real-world deployment.

Future Directions:

• Schema Modification: Moving beyond value-level errors to structural changes (e.g., splitting a feature into two) to simulate schema mismatches between training and production environments.
• Generalized Error Strategies: Developing more flexible error insertion strategies to better mimic complex real-world scenarios.
• Feature Selection: Using PuckTrick to identify the minimal set of essential features required for high accuracy even under imperfect data conditions.

In summary, PuckTrick bridges the gap between the theoretical perfection of synthetic data and the chaotic reality of operational data, offering a crucial tool for developing more reliable and robust machine learning systems.