Technical Summary: Data Safety: Synthetic Data Quality Analysis Using CIFAKE Dataset

Problem Statement

The rapid deployment of high-performance image classification models in critical domains (e.g., healthcare, autonomous driving) necessitates vast training datasets. However, acquiring sufficient real-world images is often hindered by cost, privacy, ethical constraints, and annotation burdens. Consequently, synthetic data generated by models such as GANs, diffusion models, and variational autoencoders is increasingly utilized to compensate for these shortages.

Despite the visual fidelity of modern generative models, synthetic images are not inherently equivalent to real images for training purposes. Previous research, such as studies on the CIFAKE dataset, has indicated that while models trained on synthetic data may achieve comparable accuracy to those trained on real data, they often exhibit significant disparities in metrics reflecting class imbalance and misclassification tendencies (e.g., F1 scores). These discrepancies suggest that relying solely on synthetic data can degrade model reliability and safety upon deployment. While existing literature focuses heavily on detecting deepfakes (binary classification of real vs. fake), there is a lack of frameworks designed to safely utilize synthetic data as a trainable resource. Specifically, there is insufficient quantitative understanding of how generative data differs from real data across multiple levels and how these differences impact the learning process and final performance.

Methodology

This study proposes a multi-factor quality analysis framework to evaluate synthetic data safety and effectiveness. The research utilizes three heterogeneous datasets:

1. REAL: 60,000 images from the CIFAR-10 dataset (10 classes).
2. CIFAKE1: 60,000 synthetic images generated by Stable-Diffusion-v1-4.
3. CIFAKE2: 60,000 synthetic images generated by fine-tuning the Elucidating the Design Space of Diffusion-Based Generative Models (EDM).

The analysis is conducted across three primary perspectives:

1. High-Dimensional Feature Space Analysis

The study employs three pre-trained feature extraction models (CLIP, DINOv2, and DINOv3) to map images into high-dimensional spaces. Dimensionality reduction techniques (UMAP and t-SNE) are used to visualize distribution differences. The research quantifies the distance between the centroids of synthetic and real data clusters for each class to assess feature alignment.

2. Low-Level Statistical Analysis

Statistical properties of low-level features are examined, specifically focusing on illuminance components (V channel in HSV, Y channel in YCrCb, and L* channel in CIELab). The study measures first-to-fourth-order statistics (mean, standard deviation, skewness, kurtosis) and entropy to compare the distributions of synthetic data against the REAL dataset.

3. Model Learning Process Analysis

The internal representations of three classification architectures (ResNet50, MobileNetV2, and Vision Transformer ViT16) are analyzed during training.

• Layer-wise Divergence: Cosine similarity and L2 distances are calculated between layers of models trained on synthetic data versus those trained on real data to identify where feature representations diverge.
• Realistic Mixing Scenarios: Three experimental configurations simulate real-world deployment constraints:
  • One-class replacement: Replacing a specific underperforming class in synthetic data with real data.
  • Randomized mixture: Systematically varying the ratio of real to synthetic data (10% to 90%) across all classes.
  • Additional fine-tuning: Fine-tuning a model pre-trained on synthetic data with a small amount (10%) of real data over extended epochs.

Key Results

Feature Space and Statistical Differences

• CIFAKE1 vs. CIFAKE2: CIFAKE1 exhibits significant divergence from real data in high-dimensional feature spaces, particularly for classes like "frog," "bird," and "cat," where centroid distances are large. In contrast, CIFAKE2 clusters much closer to real data centroids, with DINO-based models (DINOv2/v3) showing clearer separation of appropriate features compared to CLIP.
• Low-Level Statistics: Analysis of illuminance channels reveals that CIFAKE1 has larger distribution differences in standard deviation, skewness, and entropy compared to real data. A strong positive correlation (Pearson's coefficient > 0.75 for MobileNet) was found between the entropy of the L* channel (CIELab) and model performance (F1 score and Recall). Classes with smaller distribution differences in illuminance entropy were classified more accurately.

Learning Process and Performance

• Layer Divergence: Models trained on CIFAKE1 diverge from real-data-trained models starting immediately after the input layer, indicating that differences in low-level statistical features (color, shape) affect the earliest stages of learning. CIFAKE2 shows a similar trend but with significantly less divergence, particularly in early layers.
• Mixing Scenarios:
  • One-class replacement: Replacing only the worst-performing class (e.g., "frog") with real data improved Recall for that class but decreased Precision, resulting in no significant overall F1 score improvement.
  • Randomized mixture: Introducing just 10% real data significantly improved performance for poorly performing classes. A mixture of 30–40% real data yielded performance comparable to training on 100% real data, effectively bridging the distribution gap.
  • Fine-tuning: Adding 10% real data to a CIFAKE1-trained model showed a sharp initial performance increase, but further training (up to 150 epochs) yielded diminishing returns, failing to reach the performance levels of models trained solely on real data.

Contributions and Significance

This study makes the following contributions to the field of AI safety and synthetic data utilization:

1. Multifaceted Evaluation Framework: It establishes a systematic approach to analyze synthetic data quality not just for detection, but for safe integration into training pipelines. This framework connects high-dimensional feature discrepancies and low-level statistical deviations directly to downstream classification performance.
2. Quantification of Distribution Gaps: The research quantifies the "acceptable usage range" for synthetic data. It demonstrates that the distance between synthetic and real data centroids in feature space and the entropy of illuminance channels serve as predictive metrics for performance degradation.
3. Practical Utilization Guidelines: The study provides empirical evidence that synthetic data alone may induce data shifts and performance degradation. It proposes a blending strategy where incorporating approximately 30% to 40% real data per class is necessary to mitigate distribution bias and achieve performance parity with real-data-only training.
4. Insight into Learning Dynamics: By analyzing layer-wise divergence, the paper challenges the notion that performance degradation is solely a "final layer" phenomenon. It highlights that poor-quality synthetic data can corrupt the learning of low-level features (color and shape) from the very first layers of the network.

Conclusion

The paper concludes that while synthetic data offers a viable solution for data scarcity, its application requires rigorous pre-assessment. The degree of performance degradation can be predicted by quantifying feature vector distances (using DINOv3) and low-level statistical entropy differences. To ensure the reliability and safety of image classification models in real-world operations, practitioners should adopt a data blending strategy that compensates for distribution gaps, rather than relying exclusively on synthetic data or attempting to fine-tune with insufficient real-world samples. This research aims to enhance the safety of AI systems by providing a quantitative basis for evaluating and integrating synthetic data.