💬 NLP

Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning

The paper introduces Privacy-R1, a reinforcement learning framework that dynamically routes text chunks between local and remote LLMs to optimally balance privacy protection and task performance by distinguishing between replaceable and task-critical sensitive information, achieving state-of-the-art results on a new high-density medical dataset.

Original authors: Zheng Hui, Yijiang River Dong, Sanhanat Sivapiromrat, Ehsan Shareghi, Nigel Collier

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zheng Hui, Yijiang River Dong, Sanhanat Sivapiromrat, Ehsan Shareghi, Nigel Collier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very important, complex question to ask. You want the smartest possible answer, but your question contains sensitive secrets—like your home address, your medical history, or your bank account number.

You face a tough choice:

The "Super-Brain": Send your question to a massive, cloud-based AI (like a genius consultant). It will give you a perfect answer, but it might see and remember your secrets.
The "Local Notebook": Ask a smaller AI running on your own computer. It keeps your secrets safe, but it's not very smart, so the answer might be vague or wrong.

For a long time, people tried to solve this by using a "redaction pen." They would try to cross out every secret in your question before sending it to the Super-Brain. But this often backfired. If you ask, "My name is John and I have a broken leg," and you cross out "John," the Super-Brain doesn't know who has the broken leg. If you cross out "broken leg," it doesn't know what's wrong. The answer becomes useless.

Enter Privacy-R1: The Smart Diplomat

This paper introduces a new system called Privacy-R1. Instead of using a blunt redaction pen, it uses a Smart Diplomat (an AI agent) that learns how to negotiate the best deal for you.

Here is how it works, using a simple analogy:

1. The "Sentence-by-Sentence" Strategy

Imagine your question is a long letter made of many paragraphs. The old way was to read the whole letter, cross out all the secrets, and send the messy result.

Privacy-R1 breaks the letter into individual sentences. It looks at each sentence one by one and asks: "Do I need to send this specific sentence to the Super-Brain, or can I handle it myself?"

2. Learning the Rules (Reinforcement Learning)

How does the Diplomat know what to do? It learns through trial and error, just like a video game character.

The Goal: Get a perfect answer (High Score) while keeping secrets hidden (Low Risk).
The Penalty: If the Diplomat accidentally sends a secret to the Super-Brain, it loses points. If it keeps a secret but the answer becomes bad, it also loses points.
The Reward: If it sends a non-secret sentence to the Super-Brain and gets a great answer, it gains points.

Over time, the Diplomat learns a policy:

"Oh, this sentence mentions a specific date and a hospital name. If I hide that, the Super-Brain won't understand the diagnosis. I'll send it, but I'll make sure the patient's name stays hidden."
"This sentence just says 'I feel tired.' I can answer that myself. No need to send it to the Super-Brain."

3. The "Medical" Test

To prove this works in real life, the researchers created a special test using medical records. Medical data is tricky because it's full of secrets (names, dates, locations) that are also critical for the doctor to give a diagnosis.

They found that the old "redaction pen" method often ruined the medical advice because it deleted too much context. But Privacy-R1 learned to be clever. It kept the critical medical details (so the diagnosis was accurate) while hiding the personal names and addresses (so privacy was safe).

The Big Takeaway

Think of Privacy-R1 as a traffic controller for your data.

Old Way: A static wall that blocks everything, even the cars that need to pass.
Privacy-R1: A smart traffic light that knows exactly which cars (sentences) are safe to let through to the big city (the cloud) and which ones should stay in the neighborhood (your local device).

Why does this matter?
It means we can finally use powerful, cloud-based AI for sensitive jobs like healthcare, law, or finance without having to choose between "getting a good answer" and "keeping our secrets safe." The system learns to find the perfect balance, giving us the best of both worlds.

1. Problem Definition: The Privacy-Utility Trade-off

The paper addresses the critical dilemma faced when deploying Large Language Models (LLMs) in sensitive domains (e.g., healthcare, finance):

The Dilemma: Users must choose between sending queries to powerful, proprietary remote LLMs (high performance, high privacy risk) or processing them on smaller, local models (high privacy, low performance).
Limitations of Prior Work: Existing approaches, such as Privacy-Conscious Delegation, rely on static, monolithic prompt rewriting. These systems attempt to paraphrase an entire query to remove all Personally Identifiable Information (PII) before sending it to the remote model.
- Failure Mode 1 (Coherence): Removing entities breaks linguistic links.
- Failure Mode 2 (Utility Collapse): Crucial PII (e.g., a specific patient name or ID required for diagnosis) is removed, rendering the query unanswerable or the task impossible.
Core Challenge: The task requires a system that can distinguish between replaceable PII (which should be shielded) and task-critical PII (which must be revealed to achieve high utility), a decision that depends on the global context of the query.

2. Methodology: The Privacy-R1 Framework

The authors reframe the problem as a sequential decision-making task and introduce Privacy-R1, a Reinforcement Learning (RL) framework.

A. Architecture Overview

The framework operates in three stages:

Semantic Chunking: The user query $q$ is segmented into a sequence of coherent sentences $C = (c_1, \dots, c_n)$ using SpaCy. This ensures the agent processes linguistically complete units.
Stateful Policy Agent: A lightweight Transformer-based agent processes the entire sequence of chunks. Unlike static rewriters, this agent maintains a stateful context, allowing it to understand dependencies (e.g., a pronoun in chunk $n$ referring to a PII entity in chunk 1).
Dynamic Routing: For each chunk $c_t$ $c_{t}$ , the agent selects an action $A_t \in \{ \text{Local}, \text{Remote} \}$ $A_{t} \in {Local, Remote}$ .
- Local ( $M_{LOCAL}$ ): Trusted, secure, but weaker model.
- Remote ( $M_{REMOTE}$ ): Untrusted, powerful API.
- The outputs are concatenated and passed to the local model to form the final response.

B. Policy Agent Design

Input: Sentence chunks are embedded using a frozen MiniLM model and augmented with sinusoidal positional encodings.
Contextualization: A multi-layer Transformer encoder uses self-attention to generate contextualized representations ( $h_t$ ) for each chunk, considering the entire query history.
Action Selection: A linear layer with Softmax outputs the probability distribution for routing each chunk locally or remotely.

C. Reward Function Design

The agent is trained to optimize a composite reward function $R$ that balances task performance and privacy leakage:
$R = \text{TaskGain} - \lambda \cdot (\text{PrivacyLeak})^2$

TaskGain: A binary signal (1 or 0) determined by an "LLM-as-a-judge." It equals 1 if the final response is semantically equivalent or better than the gold-standard response (generated by the remote model with full PII access).
PrivacyLeak: The fraction of PII units exposed to the remote model.
Non-Linear Penalty: The authors use a quadratic penalty $(\text{PrivacyLeak})^2$ rather than a linear one. This imposes a severe penalty on catastrophic leaks (high exposure) while tolerating minor, incidental exposures, encouraging safer policies.
Hyperparameter $\lambda$ : Controls the agent's risk aversion.

D. Training Procedure

Supervised Pre-training (Warm-up): The agent is initialized as a binary classifier using heuristic labels (route local if PII exists, else remote) to provide a conservative starting point.
Reinforcement Learning (Fine-tuning): The agent is fine-tuned using Proximal Policy Optimization (PPO). The agent interacts with batches of queries, receives the episodic reward, and updates its policy to maximize expected reward while maintaining exploration via an entropy bonus.

3. Key Contributions

Novel RL Framework (Privacy-R1): The first work to use RL to learn a stateful, dynamic delegation policy at the sub-prompt (chunk) level. It implicitly learns to differentiate between replaceable and task-critical PII.
Med-PCD Dataset: A new, challenging medical dataset with high PII density. It is constructed by synthetically injecting realistic PII (names, dates, MRNs) into anonymized doctor-patient dialogues, specifically designed to stress-test contextual reasoning in privacy-preserving systems.
State-of-the-Art Performance: Demonstrates that learned, adaptive policies significantly outperform static rewriting baselines, establishing a new Pareto frontier for the privacy-utility trade-off.

4. Experimental Results

The framework was evaluated on the PUPA (general domain) and Med-PCD (medical domain) datasets across various local models (ranging from 1B to 8B parameters).

Performance Gains:
- On Med-PCD, Privacy-R1 significantly outperformed the static baseline (PAPILLON). For example, with a Llama-3.1-8B local model, it achieved +7.5% higher Quality Preservation while reducing Privacy Leakage by 4.1%.
- The improvement was most pronounced with weaker local models (e.g., 1B), where Privacy-R1 provided a +30.2% quality boost over baselines by strategically offloading critical chunks to the remote model.
Ablation Studies:
- Statefulness: Removing the Transformer's state (using an MLP) caused a >13% drop in quality, proving that cross-chunk context is essential for handling anaphora and dependencies.
- Reward Design: Replacing the quadratic penalty with a linear one resulted in a 16.2% rate of catastrophic leaks (>80% PII exposure), compared to 1.1% for the quadratic version.
Controllability: The hyperparameter $\lambda$ allows developers to tune the policy. Increasing $\lambda$ monotonically reduces leakage (from 15.5% to 1.2%) at the cost of quality, providing an intuitive control knob for risk management.

5. Significance and Conclusion

Paradigm Shift: The paper moves the field away from static, "one-shot" text manipulation toward dynamic, learned policies. It proves that effective privacy preservation requires understanding the intent and context of the data, not just the presence of entities.
Practical Deployment: Privacy-R1 offers a pragmatic solution for deploying powerful AI in sensitive sectors. By allowing a "calculated risk" for task-critical data while shielding replaceable data, it enables the use of state-of-the-art models without compromising core privacy requirements.
Future Directions: While currently limited to single-turn dialogues and binary routing, the framework lays the groundwork for multi-turn privacy preservation and delegation across heterogeneous model fleets.

In summary, Privacy-R1 demonstrates that Reinforcement Learning is the optimal approach for navigating the complex trade-off between data privacy and AI utility, achieving a balance that static heuristics cannot match.