💬 NLP

PII-Bench: Evaluating Query-Aware Privacy Protection Systems

This paper introduces PII-Bench, a comprehensive evaluation framework with 2,842 samples across 55 categories, to assess privacy protection systems and reveals that while current models detect PII well, they struggle significantly with determining query-relevant PII, especially in complex multi-subject scenarios.

Original authors: Hao Shen, Zhouhong Gu, Haokai Hong, Weili Han

Published 2026-02-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hao Shen, Zhouhong Gu, Haokai Hong, Weili Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, helpful robot assistant (like a Large Language Model, or LLM). You ask it for advice, but in your question, you accidentally slip up and include your home address, your phone number, and your boss's name.

If the robot just repeats your question back to the internet or stores it, your private info is now exposed. This is the Privacy Problem.

For a long time, the solution was simple but clumsy: The "Scissors" Approach.
If you asked the robot anything, the safety system would take a giant pair of scissors and cut out every single piece of personal information it could find.

The Problem: If you asked, "Is my resume good for this job?" and the system cut out your name, your university, and your past jobs, the robot would just say, "I can't help you because I don't know who you are." It protected your privacy, but it broke the conversation.

The New Idea: The "Smart Filter"

This paper introduces a new, smarter way to handle this called Query-Aware Privacy.

Think of it like a VIP Security Guard at a high-end club.

The Old Way (All PII Masking): The guard stops everyone at the door and strips them of their clothes, wallets, and ID cards before letting them in. No one gets hurt, but no one can enjoy the party because they have nothing to wear or pay with.
The New Way (Query-Unrelated PII Masking): The guard looks at why you are there.
- If you are there to buy a drink, the guard checks your ID (to prove you are 21) but lets you keep your wallet and home address.
- If you are there to meet a friend, the guard lets you keep your ID and wallet but hides your home address so no one else sees it.
- The Goal: Keep the information the robot needs to answer your question, but hide the information it doesn't need.

The Challenge: The "PII-Bench" Test

The authors realized that while we have good tools to find personal info (like a metal detector), we are terrible at deciding what to keep and what to hide based on the context.

To fix this, they built a massive training ground called PII-Bench.

What is it? Imagine a giant gym with 2,842 different practice scenarios.
The Scenarios: Some are simple (one person talking about themselves), and some are chaotic (a whole family reunion conversation with 7 different people, 50 different secrets, and a confusing mix of topics).
The Test: They ask the AI: "Here is a story about a person. Here is a question they are asking. Which parts of the story do you need to answer the question, and which parts should you blur out?"

What They Found

They tested the smartest robots on the planet (like GPT-4, Claude, and others) using this gym. Here is the verdict:

Good at Finding, Bad at Deciding: The robots are excellent at spotting personal info (like finding a phone number in a text). They are like a metal detector that beeps at every piece of metal.
Struggling with Context: They are terrible at figuring out relevance. If you ask, "How is my credit score?" the robot often hides the credit score because it thinks, "Oh, that's personal!" even though that's the only thing you asked for.
The "Big Brain" vs. "Small Brain": The biggest, most powerful robots did okay, but they still made mistakes. The smaller, cheaper robots (which you might want to run on your own phone for privacy) performed very poorly. They often got confused in complex situations, like a family dinner conversation.

Why This Matters

This paper is a wake-up call. We can't just rely on "blurring everything out" anymore because it makes our AI assistants useless. We need them to be intelligent editors.

They need to learn the difference between:

Relevant: "I need to know your salary to help you file taxes." (Keep this!)
Irrelevant: "I need to know your salary to help you file taxes." (But I don't need to know your home address or your mother's maiden name.) (Hide these!)

The Bottom Line

The authors have built the first "driver's license test" for privacy systems. They showed us that while our AI is getting smarter, it still needs to learn how to be a thoughtful librarian rather than a blind censor. It needs to know exactly which books to show you and which pages to keep closed, so you get the help you need without losing your secrets.

1. Problem Statement

The widespread adoption of Large Language Models (LLMs) has created a significant privacy risk: users frequently submit prompts containing Personally Identifiable Information (PII). While traditional privacy protection strategies involve masking all PII, this "blunt" approach often destroys the utility of the LLM response. For example, masking a candidate's job title or education history when a user asks for a suitability assessment renders the model unable to answer the query.

The core challenge is Query-Aware Privacy Protection: The system must distinguish between PII that is relevant to the user's specific query (which must be preserved for functionality) and PII that is irrelevant (which should be masked to protect privacy). Current research lacks a comprehensive framework to evaluate models on this specific capability, particularly in complex, multi-subject scenarios.

2. Methodology

A. The Proposed Strategy: Query-Unrelated PII Masking

The authors propose a novel strategy where the system performs two steps:

PII Detection: Identify all PII entities in the prompt and associate them with specific subjects (individuals).
Relevance Assessment: Determine which PII entities are necessary to answer the specific user query ( $E_q$ ).
Masking: Mask only the entities in $E \setminus E_q$ (the set of all PII minus the query-relevant PII), preserving the text coherence and query utility.

B. PII-Bench: The Evaluation Framework

To evaluate this capability, the authors constructed PII-Bench, a comprehensive dataset and benchmark.

Scale: 2,842 test samples.
Structure: Each sample contains a User Description ( $d$ ), a User Query ( $q$ ), and a Standard Answer indicating the query-relevant PII ( $E_q$ ).
Categories: Covers 7 main PII types (PER, CODE, LOC, ORG, DEM, DATETIME, QUANTITY) with 55 fine-grained subcategories.
Dataset Variants:
- PII-Single: 1,214 samples with single-subject descriptions.
- PII-Multi: 1,228 samples with multi-subject descriptions (e.g., colleagues, family) to test relationship reasoning.
- PII-Hard: 200 samples with high PII density and long text lengths.
- PII-Distract: 200 samples simulating complex multi-user interactions with significant information interference.
Construction Process: An automated pipeline using LLMs for entity generation and consistency optimization (ensuring logical coherence between age, occupation, etc.), followed by rigorous human verification by 25 privacy experts.

C. Evaluation Tasks

The benchmark defines three fundamental tasks:

PII Detection: Identify entity spans, associate them with subjects, and classify types.
Query-Related PII Detection: Select the minimal subset of PII necessary to answer the query.
Query-Unrelated PII Masking: Generate a modified prompt where irrelevant PII is replaced with tags (e.g., <Name>, <Phone>) while keeping relevant PII intact.

3. Key Contributions

Novel Strategy: First proposal of a query-unrelated PII masking strategy that balances privacy protection with LLM utility, moving beyond "all-or-nothing" masking.
PII-Bench Benchmark: The first comprehensive evaluation framework specifically designed to assess query relevance in privacy protection, featuring diverse scenarios from single-subject to complex multi-party interactions.
Empirical Insights: A systematic evaluation revealing the current limitations of both traditional models and advanced LLMs in handling query-aware privacy tasks.

4. Experimental Results

The authors evaluated various models, including traditional baselines (BiLSTM-CRF), API-based LLMs (GPT-4o, Claude-3.5, DeepSeek-V3), Open-Source LLMs (Llama-3.1, Qwen-2.5), and Small Language Models (SLMs).

PII Detection Performance:
- Advanced LLMs perform well in basic PII detection (Strict-F1 scores around 0.88–0.90).
- However, Entity Type Classification remains a bottleneck, with a consistent gap between entity boundary detection and type classification accuracy.
Query-Relevance Detection Performance:
- Significant Limitations: Even the best-performing models (GPT-4o) struggle with determining query relevance. With naive prompting, GPT-4o achieved only 0.627 F1, far below the human baseline of 0.951 F1.
- Prompting Impact: Chain-of-Thought (CoT) strategies (Self-Consistency, Auto-CoT) improved performance slightly (up to ~0.72 F1 for GPT-4o), but the gains were marginal compared to the difficulty of the task.
- Model Scale: Small Language Models (SLMs) showed drastic performance drops, often failing to identify relevant PII correctly.
- Complexity Sensitivity: Performance degrades sharply as the number of subjects increases (dropping from F1 0.85 to 0.52 with >5 subjects) and with longer text lengths.
Privacy-Utility Tradeoff:
- The Query-Unrelated PII Masking strategy achieved the optimal balance, with a Privacy Score ( $P$ ) of 0.83 and a Utility Score ( $U$ ) of 0.89.
- This outperformed "No Masking" (High utility, 0 privacy) and "All PII Masking" (High privacy, low utility ~0.52).

5. Significance and Conclusion

Research Gap: The paper highlights that while LLMs are good at finding PII, they are currently poor at reasoning about which PII is necessary for a specific context. This is a critical gap for deploying privacy-preserving AI in real-world applications.
Practical Implication: The "Query-Unrelated" strategy is proven to be the most viable path forward, allowing organizations to protect user data without rendering AI assistants useless.
Future Directions: The results suggest that future research must focus on improving the reasoning capabilities of models regarding context and relevance, rather than just entity extraction. The benchmark also indicates a need for better performance in small, on-device models, which currently lack the sophistication for reliable privacy management.

In summary, PII-Bench establishes a new standard for evaluating privacy systems, demonstrating that achieving "intelligent" privacy protection—where data is masked only when irrelevant to the task—remains a significant challenge for current AI technologies.