🤖 AI

Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation

This paper introduces RPSG, a novel framework that leverages private seed texts and integrates formal differential privacy mechanisms to generate high-fidelity synthetic data while ensuring strong privacy protection.

Original authors: Qian Ma, Sarah Rajtmajer

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qian Ma, Sarah Rajtmajer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a secret diary full of your most vulnerable moments—times you were broke, sick, or scared. You want to share these stories with researchers so they can build better tools to help people like you, but you can't hand over the actual diary. If you do, your identity could be stolen, or worse, your secrets could be used against you.

For a long time, scientists tried to solve this by either hiding the diary behind a heavy fog (adding so much noise to the data that the story becomes gibberish) or by asking a robot to rewrite the story (which often just copies the original too closely, leaving the secrets exposed).

Enter RPSG (Realistic and Privacy-Preserving Synthetic Data Generation). Think of RPSG as a masterful "story alchemist." It takes your private diary entry (the "seed") and runs it through a magical three-step process to create a brand-new story that feels exactly like yours but belongs to a completely different person.

The Three-Step Magic Trick

Step 1: The Abstracting Ghost
First, the system takes your raw story and runs it through a "Ghost Filter." This filter strips away the specific names, addresses, and dates (like "I live at 123 Maple Street") and replaces them with placeholders like [MASK]. But here's the clever part: it doesn't just delete things; it rewrites the story into a few different "abstract" versions that keep the feeling (the sadness, the anger, the hope) exactly the same.

To make sure no one can guess which abstract version came from your specific diary, the system plays a game of "privacy roulette." It adds a tiny bit of mathematical static (noise) to the scores of these versions and picks one at random. This is the Differential Privacy step. It's like shuffling a deck of cards so thoroughly that even if you know the order of the cards, you can't tell which card was the Ace of Spades.

Step 2: The Creative Writer
Next, a powerful AI (a Large Language Model) looks at that chosen abstract version and writes a brand-new, full-length story based on it. It's not just copying; it's improvising. It creates a "synthetic variant" that sounds human, uses the right slang, and captures the emotional tone of the original, but it's built from the abstracted pieces, not the raw secrets.

Step 3: The Privacy Detective
Finally, the system acts like a strict editor. It checks the new story to make sure the AI didn't accidentally "memorize" a piece of your original diary and paste it back in. It uses a special test (measuring how "surprised" the AI is by the words) to filter out any sentences that look too much like your original. If a sentence is too perfect, too similar to the secret source, it gets tossed. The result is a clean, safe story that feels real but reveals nothing about the original author.

Why the Old Ways Didn't Work

The paper tested RPSG against other methods and found some big problems with the competition:

The "Foggy" Method (DP-SGD): This approach tries to train a model on private data while adding noise. The paper found that this often results in a model that produces "word salad"—nonsense that is safe but useless. It's like trying to read a book where every third word is replaced by static; you're safe, but you can't learn anything.
The "Prompt" Method (AUG-PE): This asks an AI to "write a story about poverty" without giving it a specific seed. The paper found this creates generic, boring stories that miss the specific, gritty details researchers need. It's like asking a chef to "make a soup" without giving them any ingredients; they might make something edible, but it won't taste like the specific dish you wanted.
The "Rewrite" Method (RUPTA): This tries to rewrite your story one-to-one. While the stories sound good, the paper showed they are dangerous. Because they are so similar to the original, a hacker could easily figure out, "Hey, this story came from that specific person's diary."

The Results: Safe and Useful

The researchers tested this on real data from Reddit (posts about financial hardship) and PubMed (medical abstracts).

Privacy: When they tried to attack the new data with "Membership Inference Attacks" (tricks to guess if a specific person's data was used), RPSG was incredibly hard to crack. The attackers' success rate hovered around 50%—which is basically just guessing like flipping a coin. In contrast, the other methods let attackers guess correctly much more often (sometimes over 70%).
Usefulness: The synthetic data generated by RPSG was actually useful. When researchers used it to train new AI models, the models performed much better than those trained on the "word salad" from the foggy method. For example, on the Reddit data, RPSG helped models achieve 32.8% accuracy, while the prompt-based method only got 19.9%.
Speed: It was also faster. Generating 1,000 samples took about 1.2 to 1.4 times less time (in GPU hours) compared to the prompt-based method.

The Catch

The authors are careful to note that while RPSG is a huge step forward, it's not a magic wand that solves everything forever. They found that the system is sensitive to how you tune the knobs (like the temperature of the AI). Also, while it protects individual stories, they admit they haven't fully tested what happens if a bad actor collects thousands of these synthetic stories to piece together a profile of a community. That's a puzzle for future researchers to solve.

In short, RPSG suggests a new way to share our secrets without losing them: turn the secret into a ghost, let a robot dream a new story from that ghost, and then double-check that the dream doesn't accidentally reveal the dreamer. It's a way to keep the privacy of the diary while keeping the heart of the story.

Technical Summary: Private Seeds, Public LLMs (RPSG)

Problem Statement

The generation of synthetic replicas of private text is a critical need for NLP applications ranging from clinical analysis to social media research, particularly when original data cannot be shared due to privacy constraints (e.g., self-disclosures containing PII). Existing methods face a dichotomy:

DP Fine-Tuning: Approaches like DP-SGD apply differential privacy during model training. However, they are often impractical for modern API-only LLMs (e.g., GPT-4, Claude) and suffer from high computational costs and model memorization risks when applied to open-source models.
Prompt-Based Methods: Techniques like AUG-PE use prompt engineering to guide LLMs without fine-tuning. While model-access agnostic, they often fail to preserve the richness, utility, and specific semantic structures of the original private data.
One-to-One Rewriting: Methods like RUPTA rewrite individual texts but may not sufficiently disrupt statistical patterns required to resist membership inference attacks (MIAs).

The core challenge is balancing fidelity (producing data useful for downstream tasks) with privacy (preventing the leakage of sensitive information and resisting inference attacks) without relying on direct fine-tuning of private data.

Methodology: RPSG

The authors propose Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), a three-phase pipeline that uses private data as "seeds" to generate synthetic text while integrating formal and empirical privacy safeguards.

Phase 1: Abstraction and Formal DP Candidate Selection

PII Redaction: Private seeds are first processed by a regex-based filter to mask structured PII.
Abstraction: A pre-trained abstraction model (e.g., BART-large-cnn) generates $m$ abstracted candidates for each private seed. These candidates are guided by sentiment alignment to ensure the expressive intent and affective polarity of the original text are preserved, while breaking direct semantic correspondence.
Formal DP Selection: To select the final candidate from the abstracted set, RPSG employs the Gaussian Mechanism. A utility function (normalized cosine similarity) measures the alignment between the seed and each candidate. Gaussian noise is added to these utility scores based on a privacy budget ( $\epsilon$ ) and failure probability ( $\delta$ ). The candidate with the highest noisy score is selected. This step ensures the selection process satisfies $(\epsilon, \delta)$ -differential privacy.

Phase 2: Synthetic Data Generation

The DP-selected abstracted candidates are fed into a Large Language Model (LLM) via prompting to generate synthetic variants.
A second generation step uses these variants to produce the final synthetic data ( $S_{syn}$ ), with prompts designed to reflect the structure, semantics, and diversity of the original input.

Phase 3: Data Refinement

To mitigate memorization risks and further enhance privacy, the pipeline applies a rigorous filtering process:

Surrogate Model Training: A pre-trained model (e.g., BERT-small) is fine-tuned on the synthetic variants to capture their distributional properties.
Cosine Similarity Filtering: Synthetic samples with high embedding similarity to the original private seeds are removed to prevent direct leakage.
NLL-Based Filtering: The surrogate model computes the Negative Log-Likelihood (NLL) for remaining samples. Samples with low NLL (indicating high confidence and potential memorization) are filtered out based on an $\alpha$ -percentile threshold. Only samples with high NLL (less likely to be memorized) are retained.
Final PII Check: A final PII filter is applied to the refined dataset.

Key Contributions

Novel Method (RPSG): A framework that directly uses private data as seeds to produce one-to-one mapped synthetic text, integrating a formal DP mechanism for candidate selection and empirical safeguards (NLL filtering) to reduce memorization.
Comprehensive Evaluation: The method is evaluated against three distinct baselines: a gradient-based method (DP-SGD), a prompt-based method (AUG-PE), and a rewriting method (RUPTA) across varying privacy budgets ( $\epsilon$ ).
Empirical Privacy Proof: The authors demonstrate that RPSG provides strong resistance to Membership Inference Attacks (MIAs), achieving AUC scores near 50% (random chance) across multiple attack types (PPL, REFER, LIRA), significantly outperforming baselines in privacy protection while maintaining competitive utility.

Experimental Results

The authors evaluated RPSG on two datasets: a subset of PubMed abstracts and a custom Reddit dataset focused on financial hardship and poverty.

Utility (Downstream Performance): RPSG consistently outperformed DP-SGD and often AUG-PE in language modeling tasks (Next-word prediction accuracy). For instance, on the Reddit dataset with GPT-4o-mini, RPSG achieved 32.8% accuracy compared to 19.9% for AUG-PE. RPSG maintained stable accuracy even as privacy budgets tightened, whereas DP-SGD performance collapsed.
Diversity and Similarity: RPSG demonstrated higher lexical diversity (lower Self-BLEU, higher N-gram scores) and better global distributional alignment (lower FID, KLD, and higher Mauve scores) compared to baselines.
Privacy Resistance:
- MIAs: RPSG achieved AUCs close to 50% (e.g., 50.0% LIRA on Reddit with Phi-4), indicating strong resistance to membership inference. In contrast, RUPTA and AUG-PE showed significant deviations from 50%, indicating higher vulnerability.
- PII Leakage: The rate of successful PII extraction in RPSG-generated data was extremely low (0.5%–1.2%).
Efficiency: RPSG demonstrated computational efficiency gains, requiring 1.22x to 1.38x fewer GPU hours than AUG-PE for generating 1,000 samples.

Significance and Claims

The paper claims that RPSG successfully bridges the gap between the utility of private data and the necessity of privacy preservation. By integrating formal differential privacy into the candidate selection phase and employing empirical refinement (NLL filtering), RPSG generates synthetic data that is:

High-Fidelity: Capable of supporting downstream tasks like sentiment analysis and language modeling better than existing private generation methods.
Robust: Significantly more resistant to membership inference attacks than prompt-based or rewriting-based alternatives.
Practical: Applicable to both API-based and open-source models without requiring the computationally expensive fine-tuning of private data.

The authors emphasize that their evaluation rigorously avoids common methodological pitfalls in MIA research (e.g., temporal distribution shifts, artificial lexical filtering), ensuring the reported privacy guarantees are valid. They conclude that RPSG offers a viable path for rigorously integrating privacy into synthetic data generation, enabling the safe use of sensitive text data for research and AI development.

Limitations Acknowledged: The authors note that RPSG's optimal thresholds are sensitive to specific LLM generation patterns and that the current evaluation focuses on per-sample leakage, not potential re-profiling risks from aggregating many synthetic samples.