DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

1. Problem Statement

The development of Large Language Models (LLMs) typically requires centralized access to massive training datasets. However, in many real-world scenarios (e.g., healthcare, finance, government), data owners cannot provide "eyes-on" access to raw private data due to strict privacy regulations or security policies.

• The Trade-off: Existing methods for generating differentially private (DP) synthetic data face a dilemma:
  • DP Fine-tuning (e.g., DP-SGD): Provides strong privacy guarantees but requires the model to ingest raw private data during training, violating the "eyes-off" constraint.
  • Prompt-based Methods (e.g., Aug-PE): Avoid direct data access by using iterative prompting to steer a frozen, off-the-shelf LLM. However, because the model weights are not updated, the generated synthetic data often lacks domain fidelity (it does not accurately capture the specific distribution, style, or structure of the private corpus).

Goal: Can we train an LLM to generate high-quality, domain-faithful synthetic text that respects the "eyes-off" constraint (never seeing raw private examples) while providing formal differential privacy guarantees?

2. Methodology: DP-RFT

The authors propose Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm designed to train an LLM ($M_{gen}$) to generate synthetic data aligned with a private corpus ($D_{priv}$) without ever ingesting the raw private examples.

Core Workflow

1. Input Prompt Generation: The system generates diverse input prompts containing public information about the target domain (e.g., keywords, topics, length constraints) to encourage varied generation.
2. On-Policy Sampling: The LLM generates synthetic documents ($d$) based on these prompts.
3. DP-Protected Reward Calculation:
   • Instead of feeding private data to the LLM, the system calculates a reward signal inside the private data boundary.
   • Similarity as Reward: The system computes the similarity between the generated synthetic document $d$ and all private documents $D_{priv, i}$ using a similarity metric (e.g., text embedding similarity).
   • DP Voting: To ensure differential privacy, the sum of similarity scores is clipped to a threshold $c$ and perturbed with Gaussian noise. This creates a DP-protected nearest-neighbor vote (or average similarity score).
   • Reward Function ($R$): The final reward is defined as:
     $$R = \begin{cases} R_{sim} & \text{if } R_{prompt} > \tau \\ 0 & \text{otherwise} \end{cases}$$
     • $R_{sim}$: The DP-protected similarity score.
     • $R_{prompt}$: A secondary reward (via LLM-as-a-judge or rule-based checks) ensuring the output adheres to the input prompt (e.g., length, topic). This prevents reward hacking, where the model might optimize for similarity by ignoring the prompt instructions.
4. Policy Optimization: The LLM is updated using Proximal Policy Optimization (PPO) to maximize the expected DP reward. This allows the model to learn the distribution of the private data through the reward signal without ever seeing the raw data.

Privacy Guarantee

The privacy analysis follows the Private Evolution framework. By adding Gaussian noise to the aggregated similarity scores (the reward signal) before exposing them to the LLM, the system ensures that the presence or absence of any single private document cannot be inferred from the model's updates. The privacy budget ($\epsilon$) is controlled by the noise multiplier.

3. Key Contributions

1. Novel Training Paradigm: DP-RFT is the first method to combine Reinforcement Fine-Tuning (RFT) with Differential Privacy for synthetic data generation. It successfully adapts model weights to fit a private distribution without direct data ingestion, satisfying the "eyes-off" constraint.
2. Reward Mechanism Innovation: The paper introduces a DP-protected similarity reward mechanism that utilizes raw similarity scores (rather than binary votes) to provide richer gradient signals, while employing a dual-reward system ($R_{sim} + R_{prompt}$) to mitigate reward hacking.
3. Comprehensive Evaluation: The method was evaluated across four diverse datasets:
   • PubMed: Medical abstracts.
   • BBC News: News articles.
   • WildChat: LLM-user chat logs.
   • QMSum: Meeting transcripts.
   • Evaluations covered both intrinsic quality (similarity to private data) and extrinsic utility (performance when used to train downstream models).

4. Experimental Results

The experiments demonstrate that DP-RFT significantly outperforms existing "eyes-off" baselines and narrows the gap with "eyes-on" DP fine-tuning methods.

• Downstream Utility:
  • DP-RFT consistently achieved higher next-token prediction accuracy on downstream models (BERTSmall and GPT-2) compared to Aug-PE (the prompt-based baseline) across all datasets and privacy budgets.
  • Improvement Magnitude: On BERTSmall, DP-RFT showed relative improvements of up to 22% (PubMed), 35% (BBC), and 47% (QMSum) over Aug-PE.
  • Tight Privacy Budgets: Under strict privacy constraints (low $\epsilon$), DP-RFT outperformed DP-Finetuning (which violates the eyes-off constraint) on BERTSmall, particularly in low-data regimes (e.g., QMSum with only 700 examples).
• Fidelity and Similarity:
  • DP-RFT generated synthetic data with higher embedding similarity to the private corpus than Aug-PE.
  • Qualitative Analysis: Human/LLM evaluations showed DP-RFT better captured lexical overlap and structural properties (e.g., turn-taking in meetings, abstract structure in medical papers) compared to the frozen Aug-PE approach.
• Efficiency: While Aug-PE requires iterative generation (high inference cost), DP-RFT incurs a one-time training cost, after which synthetic data can be generated efficiently.

5. Significance and Impact

• Bridging the Gap: DP-RFT effectively bridges the performance gap between "eyes-off" methods (which are safe but low quality) and "eyes-on" DP fine-tuning (which is high quality but requires data access).
• Practical Privacy: It enables organizations to leverage LLMs for synthetic data generation in highly regulated environments where data cannot leave the secure boundary, facilitating downstream model development without compromising privacy.
• Scalability: The method is applicable to long-form, structured, and domain-specific text, making it a versatile tool for privacy-preserving AI development.

In conclusion, DP-RFT represents a significant advancement in privacy-preserving machine learning, proving that reinforcement learning guided by differentially private signals can train high-fidelity synthetic data generators without ever exposing individual private examples.