🤖 AI

No Free Lunch for Synthetic Images under Data Scarcity Conditions

This study proposes a multidimensional evaluation framework to assess the trade-offs between fidelity, privacy, and utility in synthetic image generation under data scarcity, revealing that GANs and DDPMs maintain superior robustness compared to VAEs when differential privacy mechanisms are applied.

Original authors: Borja Arroyo Galende, Alejandro Almodóvar, Patricia A. Apellániz, Juan Parras, Silvia Uribe, Santiago Zazo

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Borja Arroyo Galende, Alejandro Almodóvar, Patricia A. Apellániz, Juan Parras, Silvia Uribe, Santiago Zazo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to teach a robot how to cook a specific dish, like a perfect lasagna. But there's a problem: you only have a tiny, crumbly scrap of the original recipe, and you can't show the robot the real ingredients because they contain secret family secrets (privacy).

So, you decide to teach the robot to "dream up" new, fake recipes that look and taste like the real thing, but without revealing the original secrets. This is what this paper is about: creating fake images (synthetic data) that are useful for training AI, but safe enough to protect privacy.

The researchers tested three different "robot chefs" (AI models) to see which one could do this best when they had very little information to start with:

The VAE (Variational Autoencoder): Think of this as a chef who tries to memorize every single detail of the tiny scrap of recipe they have.
The GAN (Generative Adversarial Network): This is a chef who learns by playing a game against a critic. The chef tries to make a fake dish, and the critic tries to spot the fake. They keep playing until the chef gets really good.
The DDPM (Denoising Diffusion Model): This chef starts with a bowl of pure noise (static) and slowly cleans it up, step-by-step, until a clear image emerges.

The Three Big Rules of the Game

The paper says you can't have it all. You have to balance three things, and improving one usually hurts the others:

Fidelity (The Look): Does the fake image look like the real one?
Utility (The Use): If you use the fake image to train a new AI, does that new AI actually learn how to do the job (like recognizing a digit or a disease)?
Privacy (The Secret): Did the robot accidentally memorize the original secret recipe and spit it back out?

The Experiment: Adding "Noise" to Protect Secrets

To protect privacy, the researchers added a special kind of "static" or "noise" to the training process (called DPSGD). Imagine sprinkling a little bit of pepper into the chef's ingredients so they can't taste the exact original recipe anymore. This makes the data safer, but it also makes it harder for the chef to cook a perfect dish.

They tested these chefs on three different "kitchens":

MNIST: Simple drawings of numbers (0-9).
OCTMNIST: Medical images of eyes (retinas).
OrganAMNIST: Medical images of organs (like kidneys and livers).

What Happened? (The Results)

1. The "Over-Thinker" (VAE) Crumbled
The VAE chef was very fragile. When the researchers added even a tiny bit of privacy "pepper," the VAE's cooking fell apart. The images became blurry, and the fake recipes were so bad that the new AI couldn't learn anything from them. It was like the chef got so scared of revealing secrets that they forgot how to cook entirely.

2. The "Game Player" (GAN) Was the Most Resilient
The GAN chef was the toughest. Even when the researchers added a lot of privacy noise, the GAN kept producing images that looked decent and were still useful for training other AIs.

A Surprising Twist: In some cases, adding a little bit of noise actually helped the GAN! It stopped the GAN from getting stuck on just one type of dish (a problem called "mode collapse") and forced it to explore more variety. It was like the pepper forced the chef to get creative instead of just copying the one scrap of paper they had.

3. The "Slow Cleaner" (DDPM) Was in the Middle
The DDPM chef was better than the VAE but not as strong as the GAN. It handled the noise okay, but as the noise got louder, its images started to lose their structure and shape.

The Medical Reality Check

When they moved to the medical images (eyes and organs), the results were harsher.

Because the medical data was so complex and the amount of data was so small, none of the chefs could produce fake images that were good enough to train a doctor's AI to diagnose diseases. The fake images were too blurry or too random.
This highlights a hard truth: If you don't have enough real data to begin with, even the best "fake data" might not be useful for serious medical tasks.

The Big Takeaway

The paper concludes that there is "No Free Lunch." You cannot magically get perfect privacy, perfect image quality, and perfect usefulness all at once, especially when you are short on data.

If you need to protect privacy in a data-scarce environment, GANs seem to be the most reliable "chef" to keep the balance between looking real and staying safe.
However, if the data is too complex (like detailed medical scans) and too scarce, synthetic data might simply not be good enough to do the job yet.

The researchers built a "scorecard" (a framework) to measure these three things together, helping future developers choose the right tool for their specific situation without breaking the rules of privacy.

Technical Summary: No Free Lunch for Synthetic Images under Data Scarcity Conditions

Problem Statement

Synthetic data generation is increasingly utilized to address data scarcity and privacy concerns, particularly in sensitive domains like healthcare. However, generating high-quality synthetic data while guaranteeing privacy introduces a fundamental trade-off: enhancing privacy often degrades the fidelity (statistical similarity to real data) and utility (performance in downstream tasks) of the generated data. This phenomenon aligns with the "no-free lunch" theorem for privacy-utility trade-offs.

While previous evaluation frameworks exist, they are predominantly designed for tabular data. There is a lack of systematic, multidimensional evaluation for high-dimensional image data under conditions of data scarcity and differential privacy (DP). Furthermore, the behavior of different deep generative architectures—specifically Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), and Denoising Diffusion Probabilistic Models (DDPM)—under these constrained conditions remains under-explored.

Methodology

The study proposes a unified empirical framework to jointly assess three core dimensions of synthetic image quality: Fidelity, Privacy, and Utility.

1. Generative Models and Training Setup

Three widely used architectures were evaluated:

VAE: Models the data distribution explicitly via a latent variable $z$ , optimizing the Evidence Lower Bound (ELBO).
GAN: Uses a game-theoretic approach where a generator competes with a discriminator.
DDPM: Learns to reverse a gradual noise corruption process.

All models were trained using Differentially Private Stochastic Gradient Descent (DPSGD) to enforce privacy. The implementation of DPSGD varied by architecture:

For VAE and DDPM, noise was added to the gradients of the entire architecture.
For GAN, noise was applied only to the discriminator's gradients, following established practices.
Conditional sampling was employed to ensure class balance and prevent bias, allowing the generation of underrepresented classes.

2. Evaluation Dimensions

Privacy: Evaluated using a formalization of differential privacy tailored for generative models. The metric $\epsilon$ $ϵ$ quantifies the likelihood ratio of a query sample $x_q$ $x_{q}$ (present in the "victim" training set but absent in the "ablated" set) being generated by the victim model versus the ablated model.
- Challenge: Accurate density estimation in high-dimensional, data-scarce spaces is difficult. The authors employed Isomap for dimensionality reduction (selected for its superior preservation of pairwise distance rankings) followed by k-Nearest Neighbors (kNN) for density ratio estimation.
- Additional Metric: The empirical Cumulative Distribution Functions (ECDF) of minimum distances between synthetic/holdout samples and training data were compared using Kolmogorov-Smirnov (KS) distance and area between curves to detect memorization.
Fidelity: Assessed using a suite of image similarity metrics:
- FID (Fréchet Inception Distance): Measures distributional alignment.
- IS (Inception Score): Evaluates quality and diversity.
- PSNR, LPIPS, SSIM: Measure pixel-level and perceptual similarity.
Utility: Measured by training a classifier exclusively on the synthetic data and evaluating its accuracy on a held-out real test set. This isolates the impact of synthetic data quality on downstream generalization.

3. Datasets

The framework was applied to three datasets:

MNIST: General-purpose handwritten digits (10 classes).
OCTMNIST: Medical retinal OCT images (4 classes).
OrganAMNIST: Medical abdominal CT images (11 classes).
Experiments were conducted under strict data scarcity (e.g., 10 samples per class) to simulate real-world constraints.

Key Results

1. Model Robustness and Trade-offs

VAE: Demonstrated the least robustness. While it achieved high initial fidelity with no noise, its performance degraded rapidly as DPSGD noise increased. Fidelity metrics (FID, IS) dropped sharply, and utility (classification accuracy) collapsed to near-random guessing levels.
DDPM: Showed intermediate behavior. It maintained better stability than VAE but suffered from significant loss of structural similarity (SSIM) and perceptual quality (LPIPS) under privacy constraints.
GAN: Emerged as the most robust architecture. It maintained higher fidelity and utility across a wider range of noise levels. Notably, in the MNIST and OrganAMNIST experiments, moderate noise levels actually improved GAN utility. The authors attribute this to DPSGD acting as an implicit regularizer, mitigating the "mode collapse" phenomenon often seen in GANs trained on small datasets.

2. Privacy vs. Utility in Medical Data

In the OCTMNIST dataset, all models struggled to produce synthetic data with meaningful utility for downstream classification, with accuracy hovering near random guessing. This highlights a critical limitation: in highly complex, small medical datasets, synthetic data may not retain sufficient predictive power regardless of the model used.
In OrganAMNIST, GANs again outperformed others in maintaining utility under privacy constraints, while VAE degraded immediately.

3. Dimensionality Reduction and Privacy Estimation

The study confirmed that standard density estimation methods fail in high-dimensional, sparse settings. The combination of Isomap (for dimensionality reduction) and kNN (for density estimation) provided a robust approach for estimating privacy leakage ( $\epsilon$ ) in these scenarios.

Significance and Claims

The paper claims to provide the first empirical comparison of deep generative architectures (VAE, GAN, DDPM) across fidelity, privacy, and utility specifically for image data under data scarcity.

Guidance for Model Selection: The results suggest that GANs are currently the most suitable architecture for balancing privacy and quality in image synthesis under differential privacy, particularly because they can leverage noise to improve generalization and diversity.
Limitations of Synthetic Data: The study emphasizes that synthetic data generation is not a panacea. In small, complex medical datasets (like OCTMNIST), the utility of synthetic data for downstream tasks is severely limited, regardless of the model or privacy mechanism.
The "No Free Lunch" Reality: The findings reinforce that improving one dimension (privacy) inevitably compromises others (fidelity/utility), though the severity of this trade-off varies significantly by architecture.
Regulatory Relevance: The framework offers a practical tool for evaluating synthetic data in regulated domains (e.g., under EHDS and HIPAA), helping stakeholders make evidence-based decisions on model selection and privacy parameters.

The authors conclude that while DPSGD can sometimes act as a beneficial regularizer for GANs, careful monitoring of privacy leakage remains essential, and future work should focus on developing models tailored to specific tasks that adhere to the principle of least privilege.