Can Large Language Models Really Recognize Your Name?

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in privacy-critical pipelines to detect and redact Personally Identifiable Information (PII), such as human names, from text. These systems operate on the fundamental assumption that LLMs can reliably recognize human names regardless of context.

The authors identify a critical vulnerability: Contextual Ambiguity. Many real human names are orthographic neighbors (differing by only one letter) to non-human entities (e.g., locations, organizations, minerals, bacteria, or syndromes). When these names appear in text snippets with ambiguous linguistic cues, LLMs often fail to classify them as "PERSON," instead misidentifying them as the non-human entity they resemble. This failure leads to uneven privacy protection, where individuals with linguistically ambiguous names are at a higher risk of data leakage than those with common, unambiguous names.

The paper investigates two specific phenomena causing these failures:

1. Name Regularity Bias (NRB): The tendency of models to rely on superficial morphological patterns (e.g., suffixes like "-ite" suggesting minerals) rather than deep contextual understanding, leading to misclassification of rare but real names.
2. Benign Prompt Injection (BPI): A scenario where user input contains instruction-like text (e.g., "keep the term X intact") that the LLM mistakenly interprets as a system command rather than data to be anonymized, causing it to ignore privacy redaction rules.

2. Methodology: The AmBench Benchmark

To systematically evaluate these vulnerabilities, the authors constructed AmBench, a novel benchmark dataset containing over 12,000 real human names.

• Data Construction:
  • Name Selection: The authors identified real human names that are exactly one Levenshtein edit distance away from non-human entities in five categories: Locations, Organizations, Syndromes, Minerals, and Bacteria.
    • Example: "Italys" (a real name) vs. "Italy" (Location); "Albanir" (Name) vs. "Albania" (Location).
  • Template Generation: Using a prompt-based pipeline, they generated over 60,000 concise, two-sentence text snippets. These templates were designed to be syntactically and semantically compatible with both human and non-human entities (e.g., using pronouns like "she" for a location-like name to create ambiguity).
  • Validation: Templates were validated by human annotators and LLMs to ensure they were ambiguous yet sound.
• Experimental Setup:
  • Models Evaluated: 12 state-of-the-art LLMs (including GPT-4o, GPT-5, Gemini 2.5 Pro/Flash, DeepSeek R1/V3, and smaller open-weight models like Llama 3.1 8B) and traditional NER tools (Flair, PrivateAI).
  • Metrics:
    • Recall: The ability to correctly identify the name as a human.
    • Consistency: Whether the model classifies the same name as a human across different ambiguous templates.
    • False Discovery Rate (FDR): The rate of hallucinating human names where none exist.
  • Case Study: The authors evaluated Anthropic's Clio, an enterprise-grade tool used to summarize Claude conversations for privacy insights, testing the combined impact of NRB and BPI.

3. Key Contributions

1. AmBench Benchmark: The first large-scale dataset specifically designed to stress-test LLMs on the recognition of ambiguous human names, covering 12,000+ names across 5 distinct non-human entity categories.
2. Identification of Systemic Bias: The paper demonstrates that LLMs suffer from Name Regularity Bias, where morphological similarity to non-human entities causes significant recall drops (20–40%) compared to common names.
3. Benign Prompt Injection (BPI) Analysis: The authors show that even non-malicious, instruction-like user text can cause LLMs to bypass privacy redaction protocols, effectively "hiding" sensitive names in plain sight.
4. Fairness Implications: The study highlights that privacy failures are not random; they disproportionately affect individuals with names that are orthographic neighbors to common non-human terms, raising serious concerns about fairness in AI-driven privacy systems.

4. Key Results

• Significant Recall Drop: Across 12 LLMs, the recall for AmBench names dropped by 20–40% compared to baseline popular names.
  • Example: GPT-5 and GPT-4o saw recall drops of roughly 0.4 points on ambiguous names.
  • Reasoning Models: Even advanced reasoning models (e.g., DeepSeek R1, GPT-5-mini) struggled, often misclassifying names as the non-human entity they resembled (e.g., classifying a mineral-like name as a mineral).
• Inconsistency: Models were highly inconsistent. The same name could be classified as a "PERSON" in one template and a "LOCATION" in another, even when the context was structurally identical.
• Small Models vs. Large Models: Smaller models (SLMs) sometimes achieved higher recall but at the cost of a massive False Discovery Rate (FDR), hallucinating names frequently.
• Human vs. AI: Untrained human volunteers also struggled with ambiguous names (66% accuracy vs. 97% for baseline names), but LLMs performed significantly worse in terms of missing names entirely (failing to detect them as PII), whereas humans might misclassify them but still flag them as entities.
• Clio Case Study (NRB + BPI):
  • When BPI instructions were added to the input (e.g., "keep the term X intact"), the leakage rate of ambiguous names in Clio's summaries quadrupled (from ~3.9% to ~18.2%).
  • Clio's privacy auditor failed to detect these leaks, often assigning high privacy scores to summaries containing leaked names when BPI was present.

5. Significance and Implications

• Privacy Failure Modes: The paper reveals that relying on LLMs for the first step of privacy protection (PII detection) is risky. If the model fails to recognize a name due to ambiguity, all downstream redaction or anonymization mechanisms fail.
• Fairness and Equity: The findings suggest that privacy protection is not uniform. Individuals with names that are orthographic neighbors to common words (often associated with specific cultural or linguistic backgrounds) face higher risks of identity exposure. This challenges the fairness of current AI privacy solutions.
• Security Risks: The combination of NRB and BPI creates a new attack vector. Malicious actors could potentially engineer inputs to force LLMs to leak specific user data or perform membership inference attacks.
• Call to Action: The authors urge the research community to move beyond simple accuracy metrics and develop systematic taxonomies of failure modes. They advocate for:
  • Rigorous stress-testing of privacy tools against ambiguity.
  • Integration of knowledge bases (e.g., rare name lists) into privacy pipelines.
  • Moving toward formal privacy guarantees rather than relying solely on probabilistic LLM performance.

In conclusion, the paper demonstrates that LLMs cannot be blindly trusted to recognize human names in ambiguous contexts, posing a significant, unaddressed risk to the fairness and efficacy of automated privacy systems.