💻 computer science

LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests

This paper presents LLM-Redactor, an open-source framework that empirically evaluates eight privacy-preserving techniques for LLM requests and demonstrates that a hybrid strategy combining local inference, redaction, and semantic rephrasing effectively minimizes data leaks while providing a decision rule for selecting optimal methods based on threat models and workloads.

Original authors: Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum

Published 2026-04-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef (the developer) who wants to cook a complex recipe using a famous, high-tech kitchen (the Cloud LLM). You have a secret family sauce (your proprietary code or private data) that you don't want the kitchen staff to see, taste, or write down in their logbooks.

The problem is that every time you send a request to the cloud kitchen, you are handing over your ingredients. Even if the kitchen is secure, the staff might accidentally leave your notes on the counter, use them to train their own recipes, or hand them over to a judge if asked.

This paper, "LLM-Redactor," is like a new, super-smart sous-chef that stands between you and the cloud kitchen. The researchers tested eight different ways to protect your secret ingredients before they leave your house. They didn't just guess; they cooked up 1,300 fake scenarios (like fake emails, secret passwords, and code snippets) to see which method actually kept the secrets safe without ruining the meal.

Here is the breakdown of their "Eight Techniques" using simple analogies:

The Eight Privacy Strategies

A. The "Cook at Home" Method (Local Inference):
- The Idea: Don't send the recipe to the cloud kitchen at all. Cook it right there on your own stove using a smaller, local oven.
- Pros: 100% safe. Nothing leaves your house.
- Cons: Your local oven might not be powerful enough to cook the most complex dishes (like advanced coding tasks).
B. The "Black Marker" Method (Redaction):
- The Idea: Before sending the recipe, you take a black marker and scribble out the sensitive parts (names, addresses, passwords) and replace them with generic labels like [EMAIL_1].
- Pros: Fast and easy.
- Cons: If you miss a spot, the secret leaks. Also, the cloud kitchen might get confused if you scribble out too much context.
C. The "Rewriter" Method (Semantic Rephrasing):
- The Idea: Instead of just scribbling, you rewrite the whole paragraph. You say, "I need a sauce for a client in a big city" instead of "I need a sauce for John Smith at Acme Corp."
- Pros: Removes the specific identity while keeping the meaning.
- Cons: Takes time to rewrite, and sometimes the AI might accidentally keep a clue that reveals who the client is.
D. The "Glass-Box Room" (Trusted Execution Environment - TEE):
- The Idea: You send the recipe to a special, locked glass room inside the cloud kitchen. Only the machine inside can see it. The staff outside can't peek, and even if they try to break in, the glass is unbreakable.
- Pros: Very secure; the kitchen staff never sees the raw ingredients.
- Cons: Requires special hardware and is harder to set up.
E. The "Assembly Line" (Split Inference):
- The Idea: You do the first few steps of the cooking at home, then send only the half-cooked dish to the cloud kitchen to finish.
- Pros: The cloud never sees the raw ingredients.
- Cons: A smart chef might be able to reverse-engineer the half-cooked dish to guess what the original ingredients were.
F. The "Magic Code" (Fully Homomorphic Encryption - FHE):
- The Idea: You lock your recipe in a magical box. The cloud kitchen cooks the food inside the locked box without ever opening it. They only give you the cooked food back, still in the box.
- Pros: Theoretically perfect privacy.
- Cons: It's incredibly slow. Cooking a simple egg might take 10,000 years with this method right now.
G. The "Puzzle Pieces" (Secret Sharing/MPC):
- The Idea: You cut your recipe into 100 tiny puzzle pieces and send them to 100 different cloud kitchens. No single kitchen has the whole picture, so they can't figure out the secret.
- Pros: Very secure.
- Cons: Requires complex coordination and is currently too slow for everyday use.
H. The "Static Noise" (Differential Privacy):
- The Idea: You add a little bit of "static" or random noise to the recipe. Maybe you swap a word for a synonym. It's still readable, but if someone tries to analyze it later, the noise makes it impossible to pinpoint the exact original details.
- Pros: Good for statistical safety.
- Cons: Can make the recipe slightly less precise.

The Big Discovery: "No One Hero"

The researchers found that no single technique is perfect.

If you just use the Black Marker (B), you might miss a secret name.
If you just rewrite (C), you might leave a clue.
If you just cook at home (A), you can't do the hard jobs.

The Winning Strategy (The "A+B+C" Combo):
The best approach is a hybrid team:

Try to cook at home first (A). If the local oven can handle it, do it there. Zero leaks.
If you must go to the cloud, use the Black Marker (B) AND the Rewriter (C). Scribble out the obvious secrets, then rewrite the rest to hide the context.

The Results:

PII (Personal Info): This combo was amazing. Out of 500 test cases, there were zero exact leaks of personal info.
Secret Code: It was good, but not perfect (about 31% leak rate on complex code).
The "Hidden Identity" Problem: The hardest thing to hide was implicit identity.
- Example: "The CFO of a company whose wife works at a competitor."
- Even if you remove the names, the story still reveals who it is. The researchers found that no amount of rewriting could hide this without making the prompt useless. For this specific case, you either have to cook at home (A) or use the Glass-Box Room (D).

The Practical Advice (The Decision Rule)

The paper gives a simple guide for developers based on how much risk they can tolerate:

If you need 100% security: Cook at home (A). If you can't, use the Glass-Box Room (D).
If you want a good balance (The Sweet Spot): Use the A+B+C combo. Route easy stuff to your local machine, and for the rest, redact and rewrite before sending to the cloud. This is fast enough for most people and very safe.
If you are in a hurry: Just use the Black Marker (B). It's fast, though slightly less secure.

The Bottom Line

You can't just rely on one trick to protect your data in the age of AI. You need a layered defense. By combining local processing, smart redaction, and rewriting, you can send your requests to the cloud without handing over your secrets—unless your secret is a story that reveals itself by its very nature. In that case, keep it to yourself!

The researchers have released their "sous-chef" tool (the code) for free so anyone can start protecting their AI requests today.

1. Problem Statement

Large Language Model (LLM) APIs have become critical infrastructure for developer workflows, yet they pose significant privacy risks. Prompts sent to cloud vendors often contain sensitive data, including Personally Identifiable Information (PII), organizational secrets, proprietary code, and credentials. Once transmitted, this data may be logged, retained for training, subpoenaed, or leaked via third-party telemetry.

Existing privacy solutions are insufficient because they operate at the margins:

Network-level encryption (TLS) protects against passive observers but not the vendor.
Data Loss Prevention (DLP) focuses on data at rest or in version control, not transient API requests.
Secret scanners only catch known patterns at commit time, not live traffic.

There is a lack of a systematic framework that operates directly within the LLM request pipeline to protect the content of prompts before they leave the user's device.

2. Methodology

The authors propose LLM-Redactor, an open-source "shim" (middleware) compatible with the Model Context Protocol (MCP) and OpenAI-compatible APIs. The system implements and evaluates eight distinct privacy-preserving techniques, ranging from local inference to advanced cryptography.

The Eight Techniques Evaluated

Local-only Inference (A): Trivial requests are answered by a local model (e.g., Ollama); only complex requests go to the cloud.
Redaction with Placeholder Restoration (B): A local NER detector (Presidio + spaCy) and regex replace sensitive spans with typed placeholders (e.g., ⟨EMAIL_1⟩). The mapping is kept in memory and restored in the response.
Semantic Rephrasing (C): A local model rewrites the prompt to remove identifying details while preserving technical intent.
Trusted Execution Environment (TEE) (D): Inference runs inside a hardware-isolated enclave (e.g., AWS Nitro) where the vendor cannot see the plaintext.
Split Inference (E): The first $N$ layers of a model run locally; only intermediate activations (not tokens) are sent to the cloud.
Fully Homomorphic Encryption (FHE) (F): Inference runs on encrypted data (demonstrated on a small classifier; full LLMs are currently impractical).
Multi-Party Computation (MPC) (G): Inputs are secret-shared across non-colluding servers.
Differential Privacy (DP) Noise (H): Calibrated word-level noise is injected to blur residual signals.

Evaluation Setup

Benchmark: A ground-truth labeled dataset of 1,300 synthetic samples across four workload classes:
- WL1: PII-heavy prose (names, emails, SSNs).
- WL2: Secret-heavy configuration (API keys, tokens).
- WL3: Implicit identity (contextual descriptions like "the CFO whose wife works at a competitor").
- WL4: Proprietary code (internal function names, schemas).
Metrics:
- Privacy: Exact leak rate (verbatim text in cloud request), partial leak rate, and semantic leak rate (for implicit identity).
- Utility: Quality delta via judge-model A/B testing.
- Performance: Latency overhead and token cost.

3. Key Contributions

Taxonomy & Threat Model: A comprehensive classification of eight privacy techniques organized by privacy property, utility cost, and practicality, alongside a defined threat model covering six attack scenarios (e.g., vendor log exfiltration, training contamination).
Reference Implementation: A modular, open-source shim (llm-redactor) that allows independent toggling of all eight options and supports both MCP and HTTP interfaces.
Comprehensive Benchmark: The first empirical comparison of all eight technique classes on a common dataset with 4,014 annotated sensitive spans.
Decision Framework: A practical decision rule for practitioners to select the optimal privacy strategy based on their threat-model budget (acceptable leak rate) and workload characteristics.

4. Key Results

Privacy Performance

No Single Dominant Solution: No single technique works best for all workloads.
Best Practical Combination (A+B+C): Routing locally where possible, then redacting and rephrasing the rest, achieved the best balance:
- 0.6% combined leak on PII (WL1).
- 0 exact leaks on PII across 500 samples.
- 31.3% combined leak on proprietary code (WL4).
The "Implicit Identity" Problem: Techniques relying on content transformation (B, C, H) failed significantly on WL3 (Implicit Identity). Even with rephrasing, the semantic leak rate remained ≥95% because the structural relationships defining the identity (e.g., "CFO of X") are necessary for the prompt's utility and cannot be removed without breaking the task.
- Solution: Only Local Inference (A) or TEE (D) effectively mitigates implicit identity leaks.
Redaction (B) Limitations: Regex/NER detectors missed 25.9% of organization names and 79.8% of employee IDs (unless custom patterns were added).

Utility and Performance

Latency:
- Redaction (B): Adds <50ms overhead.
- Rephrasing (C): Adds ~1–2 seconds (due to local model inference).
- Local Routing (A): Adds ~300ms for classification.
Token Cost: Redaction (B) actually reduced token counts by 4–12% because placeholders are shorter than the original sensitive data (e.g., ⟨EMAIL_1⟩ vs. a full email address).
Quality: Redacted prompts resulted in a modest quality drop; a judge model preferred the original (unredacted) response ~75–80% of the time, indicating a trade-off between privacy and answer specificity.

Cryptographic Options

FHE, MPC, Split Inference: While theoretically offering strong privacy (0% token leak), they remain research-stage with prohibitive latency (10,000x–100,000x slower for FHE) or complex infrastructure requirements.

5. Significance and Recommendations

The paper establishes that content-level transformations alone are insufficient for high-stakes privacy, particularly against implicit identity leaks.

Proposed Decision Rule for Practitioners:

Zero Tolerance (λ = 0): Use Local Inference (A) for all possible requests. For the remainder, use TEE (D) or refuse the request.
Low Tolerance (λ ≤ 0.05): Use A + B + C. Route locally, then redact and rephrase. This is the strongest practical configuration for most organizations.
Moderate Tolerance (λ ≤ 0.25): Use B (Redaction) alone. It is the cheapest option with meaningful protection.
Latency Constraints: Use B (Redaction) only, as it adds minimal latency.

Conclusion:
The authors conclude that a hybrid approach is necessary. While A+B+C offers the best practical trade-off for general use, the fundamental limitation of removing "meaning" without destroying "utility" means that Trusted Execution Environments (TEE) are required for scenarios involving implicit identity or high-value secrets where local inference is not feasible. The paper provides the tools (code, benchmarks, and decision logic) to implement these strategies immediately.