💬 NLP

MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering

This paper introduces MedPriv-Bench, the first benchmark designed to jointly evaluate the privacy-utility trade-off in medical large language models by synthesizing realistic privacy threats and demonstrating a pervasive tension between clinical accuracy and patient data protection.

Original authors: Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant, super-smart medical assistant (an AI) to help you understand your health records. You want this assistant to be incredibly helpful, giving you the best possible advice based on your specific history. But there's a catch: you don't want it to accidentally spill your deepest secrets to the world while doing so.

This paper, MedPriv-Bench, is about building a "stress test" to see if these AI assistants can walk that tightrope between being helpful and being discreet.

Here is the breakdown in simple terms:

1. The Problem: The "Mosaic" Leak

Usually, we think of privacy leaks as the AI accidentally saying your name or your address. But this paper points out a sneakier danger called "Contextual Leakage."

The Analogy: Imagine you are in a crowded room. If someone shouts your name, everyone knows who you are (that's a standard leak). But what if no one says your name? Instead, someone describes you as: "A 55-year-old deep-sea welder who just had a rare stem cell transplant for a specific syndrome called VEXAS."

Even though they didn't say your name, that specific combination of details is like a unique fingerprint. Anyone who knows you could instantly figure out, "Oh, that's Bob!" The AI might not mean to reveal your identity, but by trying to give a perfectly detailed answer, it accidentally paints a picture that is too specific.

2. The Solution: A New "Stress Test" (MedPriv-Bench)

Before this paper, we had tests to see if AI doctors were smart enough to answer questions correctly. We didn't have a test to see if they were safe enough.

The authors built MedPriv-Bench, which is like a fire drill for medical AI.

The Setup: They created a fake medical database.
The Trap: They secretly injected "secret ingredients" into the patient files (like "the patient is homeless," "the patient is depressed," or "the patient is a celebrity").
The Test: They asked the AI questions that needed those secrets to answer well, but didn't ask for the secrets directly.
- Example Question: "How should we plan this patient's discharge?"
- The Trap: To answer well, the AI needs to know the patient is homeless. But if it says "The patient is homeless," it has leaked the secret. If it says nothing, the advice might be bad.

3. The "Referee" (The NLI Judge)

How do you know if the AI leaked a secret? You can't ask a human to read thousands of answers. So, the authors created a digital referee.

The Analogy: Think of the referee as a detective with a magnifying glass.

The AI gives an answer.
The detective (a specialized computer program) asks: "Does this answer logically prove the secret exists?"
If the AI says, "We need to find a shelter," the detective knows, "Aha! You figured out the patient is homeless, even though you didn't say the word 'homeless'."
This referee is surprisingly good, agreeing with human experts about 86% of the time.

4. The Big Discovery: The "Helpful vs. Safe" Trade-off

The researchers tested 9 different AI models (some famous, some medical-specific). They found a universal rule: The smarter and more helpful the AI tries to be, the more likely it is to leak secrets.

The Analogy: Imagine a butler.

The "Over-Reasoning" Butler: This butler is so eager to help that if you ask, "What should I wear?", he says, "Well, since you have a rash on your left arm and you're allergic to wool, and you're going to a wedding in July, you should wear a linen shirt." He is very helpful, but he also revealed you have a rash and an allergy (secrets you didn't ask to share).
The "Safe" Butler: This butler says, "I'm not sure what to recommend without more info." He is safe, but not very helpful.

The paper found that medical-specific AIs (trained only on medical books) were better at keeping secrets than general AIs (like the ones that write emails and code). However, even the best ones struggled. When they tried to protect privacy, their answers became a bit less useful.

5. The Takeaway

The paper concludes that we can't just trust AI to "behave" in hospitals. We need a standard way to check them.

The Final Metaphor:
Think of medical AI like a new car. Before you let it drive your family, you don't just check if the engine is fast (Utility); you also check if the brakes work (Privacy).

MedPriv-Bench is the new crash-test dummy and brake-test track specifically designed for medical cars.
It shows us that right now, many of these "cars" are fast but have weak brakes. We need to fix the brakes (privacy) without slowing the engine down too much (utility), or we risk a crash.

In short: This paper gives us the tools to stop AI doctors from accidentally "telling on" their patients while trying to save them.

1. Problem Statement

The integration of Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) in healthcare offers significant potential for clinical decision support by grounding outputs in verifiable evidence (e.g., Electronic Health Records). However, this introduces a critical, often overlooked privacy threat known as Contextual Leakage.

The Core Issue: While standard de-identification techniques (like Named Entity Recognition) remove explicit Protected Health Information (PHI) such as names and IDs, they fail to address semantic relationships. An LLM might synthesize a response that, while devoid of explicit identifiers, contains a unique constellation of medical details (e.g., a rare diagnosis combined with specific demographics, occupation, or temporal markers).
The Risk: This unique combination allows for patient re-identification, violating regulations like HIPAA's "Minimum Necessary Standard" and GDPR.
The Gap: Existing benchmarks focus heavily on clinical accuracy or general-domain privacy. There is no standardized framework to simultaneously evaluate the trade-off between clinical utility (answering the question correctly) and privacy preservation (preventing re-identification) in medical open-ended question answering (QA).

2. Methodology: MedPriv-Bench Framework

The authors propose MedPriv-Bench, the first benchmark designed to jointly evaluate privacy and utility. The framework utilizes a multi-agent, human-in-the-loop pipeline to generate high-fidelity test cases.

A. Data Source & Construction

Base Data: The pipeline uses the PMC-Patients dataset (de-identified clinical case reports).
PHI Taxonomy: A specific taxonomy of PHI categories was defined based on HIPAA and expert input, including:
- Stigmatized Behaviors (e.g., substance abuse).
- Mental Health & Cognitive Status.
- Socioeconomic Determinants of Health (SDOH).
- Quasi-Identifiers / VIP Status.
Four-Phase Generation Pipeline:
1. PHI Injection Agent: Takes de-identified clinical notes and injects realistic, contextually coherent sensitive PHI entities (e.g., adding "homelessness" or "suicidal ideation" into a narrative) to create a "sensitive context."
2. Question Agent (White-box Attacker): Generates a clinically meaningful query that implicitly requires the injected PHI to answer optimally, creating realistic "privacy pressure" without explicitly asking for the secret.
3. Answer Agent: Synthesizes a "Gold Standard" response that answers the question using the PHI but attempts to generalize the sensitive details to protect privacy (e.g., mentioning "substance use concerns" instead of naming the specific drug).
4. Auto-Validation & Filtering: A GPT-based agent and human experts (20% sample) filter the data to ensure clinical correctness (no hallucinations) and minimum viable utility (no evasive refusals).

B. Evaluation Metrics

The benchmark measures two competing dimensions:

Response Utility (0–5): A rubric-based score measuring how well the model answers the clinical question, derived from the Gold Standard.
Privacy Leakage Rate:
- Fact-Level ( $LR_{fact}$ ): The proportion of injected PHI facts revealed in the output.
- Instance-Level ( $LR_{inst}$ ): A strict binary metric; an instance is "leaked" if any single PHI fact is revealed.
- Detection Mechanism: Uses a pre-trained RoBERTa-NLI (Natural Language Inference) model as an automated judge. It treats leakage detection as an entailment problem: Premise = Model Response, Hypothesis = Injected PHI. If the model entails the hypothesis, it is a leak. This approach achieved 85.9% alignment with human expert judgment.

3. Key Contributions

MedPriv-Bench: A novel benchmark specifically for medical open-ended QA that quantifies the privacy-utility trade-off.
Automated Evaluation Protocol: A scalable, NLI-based automated judge that replaces expensive human evaluation for leakage detection while maintaining high accuracy (81.7% instance-level alignment with humans).
Empirical Analysis: A comprehensive evaluation of 9 representative LLMs (Closed-source, Open-source, and Medical-specific) revealing pervasive trade-offs.

4. Key Results

The evaluation of 9 models (including GPT-5-mini, Claude Haiku, Llama3, and Med42) yielded several critical findings:

The Privacy-Utility Trade-off: There is a clear inverse relationship. Models that achieved higher utility scores often exhibited higher leakage rates.
- Example: GPT-5-mini achieved the highest utility (4.91/5) but had a high leakage rate (58.4% instance-level) without defenses.
- Defense Impact: Applying a simple prompt-based defense ("Protect patient privacy...") reduced leakage significantly across all models but often degraded utility scores.
Model Performance:
- Med42-v2-70B (a medical-specific model) emerged as the most robust, achieving the lowest fact-level leakage (20.2%) while maintaining high utility (4.59/5). This suggests domain-specific training inherently improves privacy awareness.
- Open-source models (e.g., Deepseek-V3.2) showed high vulnerability, with leakage rates exceeding 79% without defenses.
The "Over-Reasoning" Trap: Advanced reasoning capabilities in powerful models inadvertently increase leakage. The ability to synthesize complex details to be "helpful" often leads to over-exposure of sensitive context.
Leakage Source Analysis:
- Leakage is primarily a generation-stage issue, not a retrieval issue.
- Brevity as a Defense: Constraining response length (e.g., to 100 words) drastically reduced leakage (from 39.4% to 5.7%) while preserving utility. This suggests enforcing "brevity" is an effective proxy for the HIPAA "Minimum Necessary Standard."
Question Type Sensitivity: Queries related to Infection Control & Wound Care had the highest leakage rates (40%), as they require specific environmental or physical details, whereas Medication Adherence queries had the lowest (24.1%).

5. Significance and Conclusion

Regulatory Compliance: The paper highlights that current medical AI deployments may violate HIPAA/GDPR not through explicit data theft, but through subtle contextual leakage that enables re-identification.
Standardization: MedPriv-Bench provides the first standardized methodology to validate the safety of medical LLMs, moving beyond "Can it answer?" to "Can it answer safely?"
Practical Implications:
- Prompt Engineering: Simple privacy constraints in prompts are effective but insufficient for optimal balance; they often degrade utility.
- Model Selection: Domain-specific models (like Med42) may offer a better baseline for privacy than general-purpose models.
- System Design: Enforcing response length constraints is a lightweight, effective mechanism to mitigate privacy risks in RAG systems.

The study concludes that rigorous, domain-specific benchmarks are essential for the safe deployment of medical AI, as the trade-off between clinical efficacy and patient privacy is a fundamental constraint that cannot be ignored.