The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques

This paper argues that current evaluations of attacks on PII removal techniques are flawed due to unmitigated data leakage and contamination, creating a paradox where trustworthy research requires access to private data that is inherently restricted from public scrutiny.

The Gist

Imagine you have a very important, private diary. You want to share it with the world so researchers can learn from your experiences, but you don't want anyone to know who you are.

So, you take a red marker and cross out your name, your address, your phone number, and your birthday. You think, "Perfect! Now it's anonymous. My secrets are safe."

This paper is about a group of researchers who are trying to figure out if that red marker is actually strong enough to protect you. They have a big problem: They suspect that the tests used to prove the red marker is weak are actually rigged.

Here is the breakdown of their argument, using some simple analogies.

1. The "Rigged Test" Problem

Imagine a security guard (the researcher) trying to see if a new lock (the PII removal tool) is good. To test it, they hire a thief (the attack model) to try to break in.

The researchers in this paper looked at many recent "thief vs. lock" tests and found a major flaw: The thief was often cheating.

• The "News Leak" Cheat: In some tests, the thief was given the diary and the local newspaper. The newspaper had already reported on the diary's owner. So, when the thief guessed the name, they weren't breaking the lock; they were just reading the newspaper. The test made the lock look weak, but it was actually the newspaper that was the problem.
• The "Memory Leak" Cheat: In other tests, the thief was a super-smart AI that had already read the original diary before it was redacted. When the thief saw the redacted version, they didn't have to guess; they just remembered what they had read before. It's like asking someone to solve a math problem they already memorized the answer to. The test concluded the lock was weak, but really, the thief just had the answer key.

The Conclusion: The researchers argue that we have been overestimating how easy it is to hack these privacy tools because the "thieves" in the experiments had unfair advantages (like access to the original data or public news).

2. The "Impossible Lab" Problem

So, the researchers asked: "Okay, let's do a fair test. Let's make a new lock, give it to a thief who has NEVER seen the original diary, and see if they can break it."

But here is the catch: They can't do this test.

To test the lock properly, you need a "real" private diary that no one has ever seen before. But:

• Real private data (like real medical records or court files) is locked away by strict laws. Researchers aren't allowed to touch it.
• Fake data (made by computers) sounds like a good idea, but computers are trained on the internet. If you ask a computer to write a "fake" private diary, it might accidentally copy real secrets from the internet it learned from. So, the "fake" diary isn't actually fake; it's just a mix of public secrets.

The Paradox:

• If you use real data, you can't publish the results because that would violate privacy laws.
• If you use fake or public data, the test is flawed because the "thief" might already know the answers.

The researchers call this a "conundrum." It's like trying to test a bomb-proof safe, but you aren't allowed to put a real bomb inside it, and if you use a fake bomb, the test doesn't count.

3. The "Leaky Bucket" Reality

To prove their point, the researchers did a small, safe experiment. They took two types of data that were unlikely to be in an AI's memory:

1. Old Czech Court Announcements: These were posted online for a month and then deleted.
2. New YouTube Travel Vlogs: Videos uploaded after the AI was trained.

They redacted the names and places, then asked a powerful AI to guess what was missing.

• The Result: The AI guessed correctly about 19% of the time for the videos and 5% for the court papers.
• Why? Not because the AI was a genius detective, but because the "red marker" missed some clues. For example, if the AI saw "I Love NY gift shop" left unmasked, it could guess the location was New York. Once it knew the location, it could guess the names of famous people associated with that place.

This showed that even with "new" data, the tools aren't perfect. But the researchers emphasize that we can't know for sure if the tools are truly broken without being able to test them on massive amounts of real, private data that we aren't allowed to touch.

The Big Takeaway

The paper ends with a sad but honest conclusion:

We are currently stuck.
We cannot scientifically prove whether our privacy tools (redacting names from text) are safe or unsafe in the real world.

• The tests we can do are flawed because they use data the AI might already know.
• The tests we should do (using real private data) are illegal or unethical to publish.

The Solution?
The researchers suggest we need a new way of thinking. Instead of just trying to "break" the locks with hackers, we need to build a mathematical theory of privacy that accounts for how AI learns and how information flows. We need to define the rules of the game before we start playing, so we know exactly what "safe" means in a world where super-smart computers can read between the lines.

Until then, we are flying blind, hoping our red markers are strong enough to keep our secrets safe.

Technical Summary

Here is a detailed technical summary of the position paper "The Conundrum of Trustworthy Research on Attacking Personally Identifiable Information Removal Techniques" by Ochs and Habernal.

1. Problem Statement

The paper addresses a critical paradox in Natural Language Processing (NLP) and data privacy: while Personally Identifiable Information (PII) removal is the standard method for complying with regulations (e.g., GDPR, HIPAA) and enabling data sharing, recent studies claim these sanitized documents are vulnerable to reconstruction attacks by Large Language Models (LLMs).

The authors argue that the reported success of these attacks is likely overestimated due to fundamental flaws in experimental design. Specifically, existing evaluations often fail to rule out data leakage and data contamination. If an attack model has already "seen" the original data during pre-training or fine-tuning, or if the "private" information was already public (e.g., news articles about a court case), the attack does not prove the PII removal technique is weak; it only proves the model has prior knowledge.

The core research questions are:

1. Are the evaluations of existing PII reconstruction attacks fundamentally flawed?
2. Can public researchers address these flaws without access to real, sensitive private data?

2. Methodology

The authors employ a multi-faceted approach combining critical literature review, theoretical analysis, and small-scale empirical experiments.

A. Critical Analysis of Existing Literature

The authors systematically reviewed recent works (e.g., Nyffenegger et al., Patsakis & Lykousas, Lukas et al.) that claim to reconstruct PII from sanitized texts. They categorized the sources of "success" in these attacks into three types of leakage:

• Media/Public Knowledge Leakage: Attacks where the "private" data was already public (e.g., court rulings linked to news articles).
• Memorization Leakage: Attacks where the model was fine-tuned on the original (un-sanitized) data before the attack, allowing it to recall the text rather than infer it.
• Benchmark Bias: Attacks using famous public figures (e.g., celebrities) where the model simply matches patterns from its pre-training data rather than inferring privacy breaches.

B. Theoretical Framework for Valid Attacks

The paper outlines the necessary conditions for a valid adversarial attack setup:

• Defense: Must use state-of-the-art, open-source PII removal tools (e.g., Presidio) to ensure transparency.
• Threat Model: Must assume an attacker with strong reasoning capabilities (LLMs) but zero prior knowledge of the specific document content.
• Data Requirement: The evaluation data must be private (not in the LLM's pre-training or fine-tuning data) to ensure any successful reconstruction is due to the weakness of the removal technique, not data leakage.

C. Empirical Case Studies

To demonstrate the difficulty of conducting such research, the authors conducted two small-scale experiments using data unlikely to be in LLM training corpora:

1. Czech Court Announcements: 28 PDF documents from 2018, translated to English. These were chosen because the URLs were deleted shortly after publication, making them unlikely to be in web-scale crawls.
2. YouTube Travel Vlogs: 41 transcripts from videos uploaded after the training cutoff of the chosen open-weight model (gpt-oss-120b, trained up to August 2025).

Attack Setup:

• Sanitization: PII was masked using Presidio (NER and rule-based).
• Attack Model: gpt-oss-120b (an open-weight reasoning model).
• Prompt: The model was tasked with identifying contextual hints and proposing 3 candidates for each masked span.
• Metric: Exact Match@3 (EM@3) – success if the correct PII was in the top 3 guesses.

3. Key Contributions

• Identification of Evaluation Flaws: The paper provides the first comprehensive critique showing that most PII reconstruction attacks suffer from data leakage (memorization or public knowledge), rendering their success rates misleading.
• The "Conundrum" of Private Data: The authors demonstrate that to truly evaluate PII removal, one needs private data that has never been seen by the attacker. However, accessing such data is legally and ethically restricted for public researchers, creating a deadlock where trustworthy, reproducible research is currently impossible.
• Rejection of Synthetic Data: The paper argues that synthetic data generated by LLMs is also flawed because LLMs may regurgitate real training data or introduce biases that do not reflect real-world privacy risks.
• Call for Formalization: The authors advocate for a shift from empirical "attack-and-defend" cycles to formal privacy frameworks (similar to cryptography or differential privacy) that mathematically define threat models, data assumptions, and attacker capabilities.

4. Results

The empirical experiments yielded the following findings:

• Partial Success: The attack model achieved an EM@3 of 19.1% on YouTube transcripts and 5.5% on Czech court documents.
• Root Cause of Success: Analysis revealed that the attacks succeeded primarily due to incomplete PII detection by the sanitization tool (e.g., Presidio missed a name in one sentence, allowing the model to infer it in another) and contextual leakage (e.g., mentioning "I Love NY gift shop" allowed the model to guess "New York" even if the city name was masked).
• Generic Guessing: In cases with low context, the model defaulted to generic guesses (e.g., "Jan Novák" for Czech names), which artificially inflated success rates for common names but failed for unique private individuals.
• Limitation of Public Data: Even with carefully selected "private" data, the authors could not fully eliminate the risk of leakage (e.g., travel destinations are common knowledge in pre-training data), highlighting the extreme difficulty of the task.

5. Significance and Implications

• Paradigm Shift: The paper challenges the NLP community to stop relying on flawed benchmarks that conflate "model knowledge" with "privacy breach." It suggests that current claims of PII removal being "broken" may be exaggerated.
• Research Ethics & Policy: It highlights a systemic failure in the research ecosystem: public researchers cannot ethically or legally access the "ground truth" private data required to validate privacy tools. This creates a barrier to transparent, reproducible science.
• Future Directions: The authors conclude that the field needs a new axiomatic theory of privacy for the LLM era. Current frameworks (GDPR, Differential Privacy) are insufficient because they do not account for the complex data flows, proprietary model training data, and the specific nature of textual inference. They call for formal threat models that can reason about privacy risks without requiring access to sensitive ground-truth data.

In summary, Ochs and Habernal argue that while PII removal is imperfect, the current evidence of its failure is scientifically unsound due to experimental design flaws. Solving this requires a move toward formal privacy guarantees and a restructuring of how privacy research is conducted and funded.