💬 NLP

DP-Fusion: Token-Level Differentially Private Inference for Large Language Models

The paper proposes DP-Fusion, a token-level differentially private inference mechanism for large language models that provably bounds the influence of sensitive context tokens on outputs to enable high-quality document privatization with significantly improved privacy and utility trade-offs compared to existing methods.

Original authors: Rushil Thareja, Preslav Nakov, Praneeth Vepakomma, Nils Lukas

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rushil Thareja, Preslav Nakov, Praneeth Vepakomma, Nils Lukas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, chatty robot (a Large Language Model, or LLM) that helps you summarize documents. Sometimes, those documents contain secret information, like a patient's name, a specific date, or a bank account number. You want the robot to summarize the story without accidentally spilling those secrets.

The problem is, current robots are bad at this. If you just tell them "don't say the name," they might get confused and write gibberish. If you tell them to "rewrite it nicely," they might accidentally slip up and reveal the secret anyway.

Enter DP-FUSION: The "Secret-Safe" Blender

The authors of this paper built a new method called DP-FUSION. Think of it as a special blender for text that guarantees secrets stay hidden while keeping the story readable. Here is how it works, using a simple analogy:

The Scenario: The Two Versions of a Story

Imagine you have a medical report about "Mrs. Smith" who had a bill of $345.25.

The Original Story: Contains the name "Mrs. Smith" and the exact bill amount.
The Redacted Story: You take a marker and black out "Mrs. Smith" and "$345.25," replacing them with blanks like [NAME] and [AMOUNT].

How DP-FUSION Works (The Blender Process)

Instead of just picking one version or the other, DP-FUSION does a clever dance:

Run the Robot Twice:
- First, it asks the robot to predict the next word using the Redacted Story (the safe version). This gives a "safe" guess.
- Second, it asks the robot to predict the next word using the Original Story (the secret version). This gives a "risky" guess that might leak the secret.
The "Privacy Dial" (The Blender Knob):
- The system has a dial called $\epsilon$ (epsilon).
- Dial turned all the way down (Low $\epsilon$ ): The blender mixes the two guesses so heavily that the "risky" secret is almost completely drowned out by the "safe" guess. The robot might say "a patient" instead of "Mrs. Smith." This is very private, but the story might feel a bit generic.
- Dial turned up (High $\epsilon$ ): The blender lets more of the "risky" guess through. The robot might say "Mrs. Smith" if it fits the context well. The story sounds better, but there's a slightly higher chance a secret could slip out.
The Guarantee:
- The magic of DP-FUSION is that it mathematically proves that no matter how hard a hacker tries to guess the secret, their chances of success are strictly limited by the setting on that dial. Even if the hacker knows exactly how the blender works, they can't reverse-engineer the secret.

Why Is This Better Than What We Have Now?

The paper compares DP-FUSION to other methods:

The "Scrubber" (NER): This is like using a black marker to cross out secrets. It's safe, but it leaves ugly holes in the text, making it hard to read (low utility).
The "Paraphraser" (Prompt Engineering): This is like asking the robot to "rewrite this nicely." It sounds good, but the robot often accidentally reveals the secret anyway because it wasn't forced to be careful.
DP-FUSION: This is the sweet spot. It keeps the text flowing smoothly (high quality) while mathematically guaranteeing the secrets stay hidden.

The Results

The researchers tested this on real legal and medical documents. They found that:

Quality: The text generated by DP-FUSION was much more natural and readable than other privacy methods. In fact, it was 6 times better (in terms of "perplexity," a measure of how confusing the text is) than the next best privacy method.
Security: Even when hackers tried to guess the hidden names or dates, they failed almost as often as if they were just guessing randomly.
Bonus Safety: The method also helps stop "jailbreak" attacks, where hackers try to trick the robot into ignoring its rules. By treating suspicious inputs as "sensitive tokens," the blender can dilute their influence, keeping the robot safe.

In a Nutshell

DP-FUSION is a new way to use AI to summarize sensitive documents. It acts like a smart blender that mixes a "safe" version of the text with the "real" version, controlled by a privacy dial. This ensures that secrets (like names or dates) are mathematically guaranteed to stay hidden, while the resulting story remains high-quality and easy to read.

Technical Summary: DP-FUSION

Problem Statement

Large Language Models (LLMs) lack inherent privacy guarantees during inference. When LLMs are augmented with tools or external databases containing sensitive information (e.g., Personally Identifiable Information or PII), the model's outputs can inadvertently reveal this context. Existing privacy-preserving inference methods suffer from significant limitations:

Lack of Provable Guarantees: Methods like prompt engineering or simple scrubbing (redaction) offer no formal privacy assurances and are vulnerable to reconstruction or membership inference attacks.
Poor Utility/Privacy Trade-off: Differentially Private (DP) inference methods like DP-Decoding (interpolating with uniform noise) or DP-Prompt (clipping logits) often result in severely degraded text quality (high perplexity) to achieve privacy.

The specific challenge addressed is document privatization: paraphrasing a document containing sensitive tokens such that an attacker cannot reliably infer the original sensitive tokens from the paraphrased output, while maintaining high text quality.

Methodology: DP-FUSION

The authors propose DP-FUSION, a token-level Differentially Private Inference (DPI) mechanism that provably bounds the influence of sensitive context tokens on the generated output. The method operates under a threat model where an adaptive gray-box attacker knows the inference method, model architecture, and weights, and observes the privatized output and the redacted input.

Core Mechanism

DP-FUSION adapts the concept of ensemble-based private prediction (similar to PMixED) to the inference setting. The process involves four main steps:

Token Labeling: A subset of sensitive tokens in the input document is identified and grouped into privacy groups ( $G_1, \dots, G_m$ ) using a tagger (e.g., NER).
Baseline Inference (Public): The LLM is run on the document with all sensitive tokens redacted (replaced with placeholders of equal length) to obtain a "public" next-token distribution ( $p_{pub}$ ).
Private Inference: The LLM is run $m$ times (one for each privacy group), where only the tokens of a specific group $G_i$ are revealed alongside the redacted context. This yields $m$ "private" next-token distributions ( $p_{priv, i}$ ).
Distribution Blending (Mollification): For each group $i$ $i$ , a coefficient $\lambda_i \in [0, 1]$ $λ_{i} \in [0, 1]$ is calculated via bisection search. This $\lambda_i$ $λ_{i}$ is the maximum value such that the Rényi divergence between the mixed distribution ( $\lambda_i p_{priv, i} + (1-\lambda_i) p_{pub}$ $λ_{i} p_{p r i v, i} + (1 - λ_{i}) p_{p u b}$ ) and the baseline $p_{pub}$ $p_{p u b}$ satisfies a pre-defined privacy budget $\beta_i$ $β_{i}$ .
- The final next-token distribution is an average of these blended distributions.
- A token is sampled from this mixed distribution.

Privacy Guarantees

The method provides Per-group $(\alpha, \beta_i)$ -Rényi Differential Privacy (RDP).

Neighborhood Definition: Two documents are considered adjacent if they differ only by the presence/absence of all tokens in a single privacy group.
Theoretical Bound: The algorithm ensures that the symmetric Rényi divergence between the output distributions of adjacent documents is bounded. This translates to $(\epsilon, \delta)$ -DP guarantees for the entire generated transcript, where $\epsilon$ is controlled by the number of generated tokens, the Rényi order $\alpha$ , and the group budgets $\beta_i$ .
Robustness: The mechanism bounds the attacker's advantage even if the query is chosen adversarially (e.g., jailbreak attempts).

Key Contributions

Token-Level DP Inference: Introduction of a method that provides provable DP guarantees at the token level during LLM inference, specifically for paraphrasing tasks.
Improved Utility/Privacy Trade-off: Unlike prior methods that add noise to the output distribution or clip logits, DP-FUSION blends distributions based on the difference between "sensitive" and "non-sensitive" contexts. This allows for the retention of more semantic information while strictly bounding the leakage of specific tokens.
Mitigation of Prompt Injection: The authors demonstrate that the mechanism can also be applied to security, where "sensitive tokens" are defined as chunks retrieved from untrustworthy sources (e.g., in RAG pipelines). By bounding the influence of these chunks, DP-FUSION mitigates prompt injection attacks.
Open Source Implementation: The authors release a PyPI package and GitHub repository enabling the application of DP-FUSION to arbitrary LLMs and datasets.

Experimental Results

Experiments were conducted using the Qwen 2.5 7B-Instruct model on the TAB-ECHR dataset (European Court of Human Rights cases with hand-annotated PII).

Utility (Perplexity): DP-FUSION achieved a perplexity (PPL) of 1.426 at a relaxed privacy setting ( $\alpha\beta = 0.10$ $α β = 0.10$ ) and 1.459 at a strict setting ( $\alpha\beta = 0.01$ $α β = 0.01$ ).
- This is significantly better than DP-Decoding (PPL > 3.9) and DP-Prompt (PPL > 4.26 for width 50, and >100 for width 5).
- DP-FUSION achieves 6 $\times$ lower perplexity than the best-performing baseline (DP-Prompt) at comparable privacy levels.
Utility (LLM-as-a-Judge): In pairwise comparisons against other methods, DP-FUSION paraphrases won >95% of the time against DP-Decoding and DP-Prompt. It also outperformed the "No DPI - NER" (pure redaction) baseline 45% of the time in strict privacy settings.
Privacy (Attack Success Rate): The authors evaluated token recovery attacks (Min-K% and LOSS attacks).
- DP-FUSION maintained an Attack Success Rate (ASR) close to random guessing (approx. 26-29% for $|C|=5$ ) even at high utility.
- In contrast, DP-Prompt (width 50) showed an ASR of ~50%, indicating a failure to provide empirical privacy despite lower perplexity.
- DP-FUSION offered slightly better privacy than the No-DPI NER baseline in strict settings due to the randomness introduced during distribution mixing.

Significance and Claims

The paper claims that DP-FUSION represents a significant advancement in deploying LLMs with sensitive data by offering provable privacy guarantees without the severe utility degradation seen in existing DP inference methods.

Paradigm Shift: The authors argue that instead of treating privacy as a post-processing step or a global noise injection, privacy can be managed by explicitly modeling the influence of specific token groups and bounding that influence via distribution blending.
Practicality: The method is computationally feasible (parallelizable) and works with existing LLMs without requiring retraining.
Dual Utility: Beyond privacy, the mechanism is claimed to be effective against security threats like prompt injection in RAG systems, providing a unified approach to handling untrusted context.

The authors remain modest regarding the dependency on the quality of the initial token tagger (NER), acknowledging that false negatives in tagging fall outside the theoretical guarantees, though empirical results suggest off-the-shelf taggers are sufficient for the demonstrated use cases.