💬 NLP

Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation

This paper introduces Privacy-Aware Decoding (PAD), a lightweight, model-agnostic inference-time defense that adaptively injects calibrated Gaussian noise into token logits to mitigate privacy leakage in Retrieval-Augmented Generation systems while providing rigorous differential privacy guarantees and preserving response utility.

Original authors: Haoran Wang, Xiongxiao Xu, Baixiang Huang, Kai Shu

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoran Wang, Xiongxiao Xu, Baixiang Huang, Kai Shu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Over-Sharing" Librarian

Imagine you have a super-smart librarian (the AI) who has read millions of books. Sometimes, people ask the librarian questions that aren't in the books, so the librarian goes to a special, restricted archive (the Retrieval part) to find the answer.

This archive contains sensitive, private stories—like a patient's medical history or a private email. The goal is for the librarian to use these stories to give a helpful answer without accidentally reading the private parts out loud to the whole room.

The problem is that sometimes, the librarian gets too eager or too confident and accidentally repeats the private story word-for-word. This is called a privacy leak. Previous attempts to fix this were like trying to rewrite the entire library's catalog or training the librarian to forget things, which is slow, expensive, and often ruins the quality of the answers.

The Solution: The "Smart Noise" Filter

The authors propose a new method called Privacy-Aware Decoding (PAD). Instead of changing the librarian's training or the library's books, they put a smart filter on the librarian's mouth while they are speaking.

Think of the AI's thought process as a chef choosing ingredients for a soup. The AI looks at all possible words (ingredients) and picks the best one. PAD acts like a seasoning master who adds a pinch of "confusion" (noise) to the chef's decision-making process, but only when necessary.

Here is how the "seasoning" works in three simple steps:

1. The "Confidence Check" (Screening)

The filter first asks: "Is the librarian 100% sure about this next word?"

If the answer is YES (High Confidence): The librarian is likely saying something general and safe (like "The sky is blue"). The filter says, "Great, no need to mess with that." It leaves the word alone so the answer stays clear and useful.
If the answer is NO (Low Confidence/Uncertain): The librarian is hesitating. This is dangerous territory because they might be about to pull a specific, private detail from the archive to fill the gap. The filter says, "Stop! We need to scramble this."

2. The "Smart Noise" Injection

When the filter detects a risky moment, it doesn't just add random static. It uses Adaptive Noise.

Analogy: Imagine you are trying to whisper a secret. If you are whispering a generic phrase, you speak clearly. But if you are about to whisper a specific name that shouldn't be heard, you suddenly start mumbling or speaking in a code that only the listener (the AI) can decode, but an eavesdropper (the attacker) cannot understand.
The filter adds just enough "static" to the risky words to blur the private details, but not so much that the sentence becomes nonsense.

3. The "Privacy Scorecard" (RDP Accounting)

How do we know the filter is actually working? The system keeps a Privacy Scorecard (called Rényi Differential Privacy).

Think of this like a bank statement. Every time the filter adds "noise" to protect a secret, it deducts a tiny amount from a "privacy budget."
At the end of the conversation, the system tells you exactly how much privacy was spent and guarantees that the private information is safe within a specific mathematical limit. It proves, "We spent X amount of privacy budget to ensure your secret wasn't leaked."

Why This is Better Than Old Methods

Old Way (Retraining): Trying to teach the librarian to never remember private things. This takes years and makes the librarian forget useful things too.
Old Way (Static Noise): Adding static to every word the librarian says. This makes the whole conversation sound like a bad radio connection, even when talking about safe topics.
PAD (The New Way): It's like a spotlight. It only shines the "confusion light" on the specific words that are risky. The rest of the conversation remains clear, natural, and helpful.

The Results

The authors tested this on real-world scenarios, like medical advice and email archives. They found that:

Less Leaking: The AI stopped repeating private details (like specific medical conditions or email addresses) much more often than before.
Still Useful: The answers the AI gave were still high-quality and easy to understand. The "noise" didn't ruin the soup; it just removed the poison.
Fast & Easy: It works instantly while the AI is talking. You don't need to retrain the model or change the database.

Summary

Privacy-Aware Decoding is a safety switch that turns on only when the AI is about to say something risky. It adds just enough "fuzz" to hide private secrets while keeping the rest of the conversation crystal clear, ensuring the AI remains helpful without becoming a privacy hazard.

Technical Summary: Privacy-Aware Decoding (PAD)

Problem Definition

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by conditioning outputs on external knowledge sources. However, when the retrieval corpus contains private or sensitive data, RAG systems are vulnerable to extraction attacks. Adversaries can craft prompts to force the model to emit verbatim text or leak personally identifiable information (PII) directly from the retrieved documents.

Existing mitigation strategies face two primary limitations:

Decoding Vulnerability: Even with private retrieval mechanisms, the decoding stage (generation) can act as a leakage channel. Low-temperature or greedy decoding strategies often cause LLMs to emit sensitive content verbatim, bypassing upstream protections.
Scalability and Practicality: Strong differential privacy (DP) methods like Differentially Private Stochastic Gradient Descent (DP-SGD) require significant retraining, making them impractical for large, pre-trained LLMs. Other approaches often require modifications to the retrieval infrastructure or corpus filtering.

There is a need for a model-agnostic, inference-time defense that mitigates privacy leakage without retraining or altering the retrieval pipeline.

Methodology: Privacy-Aware Decoding (PAD)

The authors propose Privacy-Aware Decoding (PAD), a lightweight, inference-time mechanism that adaptively injects calibrated Gaussian noise into token logits during generation. PAD operates entirely at the decoding stage and is model-agnostic.

The framework consists of four core components:

1. Logit Clipping and Sensitivity Bounding

To establish a deterministic sensitivity bound required for differential privacy, PAD first applies $L_2$ logit clipping. For a fixed radius $C$ , the logits $\mathbf{s}_t$ are clipped to $\bar{\mathbf{s}}_t$ :
$\bar{\mathbf{s}}_t = \mathbf{s}_t \cdot \min\left(1, \frac{C}{\|\mathbf{s}_t\|_2}\right)$
This ensures that for any two neighboring retrieval contexts, the difference in clipped logits is bounded by $\Delta = 2C$ , providing a fixed sensitivity for the Gaussian mechanism.

2. Confidence-Based Screening

PAD does not apply noise uniformly. Instead, it employs a screening mechanism to identify high-risk tokens. A token is considered high-risk if:

The model's confidence (maximum softmax probability) is below a threshold $\tau_{conf}$ .
The logit margin (difference between the top two logits) is below a threshold $\tau_{margin}$ .

Tokens satisfying these criteria are flagged for enhanced protection; confident predictions receive minimal noise to preserve utility.

3. Adaptive Noise Calibration

For high-risk tokens, PAD injects Gaussian noise $\mathcal{N}(0, \sigma_t^2 \mathbf{I})$ where the scale $\sigma_t$ is dynamically calibrated based on three factors:

Risk-Adaptive Scale ( $r_t$ ): Modulated by the logit margin. Smaller margins (higher uncertainty) result in larger noise scales.
Context-Aware Calibration: The noise scale is further adjusted by:
- Token Entropy: Higher entropy (uncertainty) increases noise.
- Position Factor: Early tokens in the sequence receive higher noise to prevent error propagation.
- Confidence Factor: Lower confidence increases noise.
Amplification Factor ( $\lambda_{amp}$ ): A global parameter to tune the privacy-utility trade-off.

The final noise scale is computed as:
$\sigma_t = \sigma_{base} \cdot \text{calibrate}(\mathbf{p}_t, t) \cdot r_t \cdot \lambda_{amp}$

4. Rényi Differential Privacy (RDP) Accounting

PAD utilizes a Rényi Differential Privacy accountant to track cumulative privacy loss. Because the noise scale $\sigma_t$ is data-dependent (adaptive), the reported $(\varepsilon, \delta)$ guarantee is specific to the realized noise schedule. The framework employs a $\gamma$ -relaxation, providing formal DP guarantees only for the fraction $\gamma$ of tokens identified as high-risk, while leaving low-risk tokens unprotected to maintain generation quality.

Key Contributions

PAD Framework: The first decoding-time privacy protection mechanism specifically designed for RAG, requiring no retraining or retrieval infrastructure changes.
Adaptive Noise Mechanism: A novel approach that targets high-risk tokens using confidence-based screening and context-aware calibration, balancing privacy protection with generation utility.
Explicit Privacy Accounting: An RDP-based accountant that reports response-level $(\varepsilon, \delta)$ privacy guarantees for protected tokens under a data-dependent noise schedule.
Empirical Validation: Comprehensive evaluation demonstrating that PAD reduces privacy leakage while preserving response utility across diverse datasets and models.

Experimental Results

The authors evaluated PAD on three real-world datasets: HealthCareMagic, ChatDoctor-iCliniq (medical dialogues), and Enron Mail. Experiments were conducted using Pythia-6.9B and Llama2-7B models against black-box extraction attacks.

Key Findings:

Privacy Mitigation: PAD substantially reduced private information leakage. On the HealthCareMagic dataset with Pythia-6.9B, PAD reduced "Repeat Prompt" leakage (verbatim reproduction of retrieved text) from 264 to 127 (a 51.9% reduction) and ROUGE-based leakage by 49.6%.
Utility Preservation: Unlike baselines that degrade generation quality, PAD maintained or improved perplexity (PPL). For instance, on HealthCareMagic, PAD achieved a PPL of 9.88 compared to 10.94 for the leakage baseline.
Comparison to Baselines: PAD outperformed existing retrieval-based defenses (Summarization, Set Distance Threshold) and post-processing methods (Static Noise Injection) in the privacy-utility trade-off.
Ablation Studies: Removing the risk-adaptive or context-aware calibration components resulted in measurable increases in privacy leakage, confirming the necessity of the adaptive components.
Parameter Sensitivity: The study identified an optimal amplification factor range (3.0–5.0) for balancing privacy and utility. Very low $\varepsilon$ values (e.g., 0.01) eliminated leakage but rendered outputs unusable due to high perplexity.

Significance and Claims

The paper positions PAD as a significant step toward mitigating privacy risks in RAG via decoding strategies. The authors claim that PAD offers a universal and scalable solution for sensitive domains (such as healthcare) because it:

Operates entirely at inference time without retraining.
Is model-agnostic and compatible with existing retrieval pipelines.
Provides explicit, per-response privacy accounting.

The work acknowledges limitations, noting that the current $\gamma$ -relaxed guarantee protects only a subset of tokens and that the reported privacy accounting is based on the realized noise schedule rather than a worst-case, input-independent bound. The authors suggest future work should explore joint optimization of retrieval and training defenses and extend the framework to multimodal RAG settings.