Evaluating LLM-based Personal Information Extraction and Countermeasures

1. Problem Statement

Personal Information Extraction (PIE) involves automatically extracting sensitive data (e.g., names, emails, phone numbers, work history) from publicly available profiles. Traditionally, this was done using regular expressions, keyword searches, and Named Entity Recognition (NER). However, these methods suffer from limited success due to a lack of semantic understanding of complex profile structures.

The paper addresses a critical security gap: Large Language Models (LLMs) possess advanced semantic understanding and can be misused by attackers to automate PIE at scale. This capability significantly lowers the barrier for launching sophisticated attacks like spear phishing, telecommunication fraud, and credential stuffing. The authors aim to systematically benchmark the effectiveness of LLM-based PIE attacks and evaluate existing and novel countermeasures.

2. Methodology

A. Attack Framework

The authors propose a structured framework for LLM-based PIE attacks consisting of two configurable components:

1. Prompt Design: Includes selecting prompt styles (direct, persona, contextual, pseudocode), using in-context learning (few-shot examples), and adding instructions to bypass potential defenses.
2. Profile Processing: Involves either direct utilization of the raw profile (e.g., HTML source) or filtering redundant information (e.g., removing CSS/HTML tags) to fit LLM context limits.

B. Datasets

To address the lack of ground-truth benchmarks, the authors constructed five datasets containing 8 categories of personal information (email, phone, address, name, work experience, education, affiliation, occupation):

• Synthetic Dataset: 100 profiles generated by GPT-4 in various styles (neat, geeky, formal, etc.) to ensure controlled ground truth.
• Real-world Datasets:
  • Celebrity: Profiles of top 100 famous people.
  • Physician: 19th-century American physicians from Wikipedia.
  • Professor: Profiles of EECS professors from top universities (MIT, CMU, Stanford, Berkeley).
  • Court: European Court of Human Rights cases (adapted from a text anonymization dataset).

C. Evaluation Setup

• Models: Evaluated 10 LLMs (including GPT-4, GPT-3.5, PaLM 2, Gemini, Vicuna, Llama-2, InternLM) ranging from 7B to >1.5T parameters.
• Baselines: Compared against traditional methods: Regular Expressions, Keyword Search, NER (spaCy, BERT-based), and mlscraper.
• Metrics:
  • Accuracy: For structured data (email, phone) requiring exact matches.
  • Rouge-1 & BERT Score: For unstructured data (work experience, education) to measure word overlap and semantic similarity.

D. Countermeasures Evaluated

The study evaluated existing defenses and proposed a novel one:

1. Symbol Replacement (SR): Replacing @ with AT, . with DOT.
2. Keyword Replacement (KR): Replacing names with placeholders (e.g., <first_name>).
3. Hyperlink (HL): Hiding text behind clickable links.
4. Text-to-Image (TI): Converting text to images.
5. Prompt Injection (PI - Novel): Injecting invisible text into the profile that instructs the LLM to ignore previous context and output false information. This is designed to be invisible to human users (e.g., same font color as background) but visible to the LLM.

3. Key Contributions

1. First Systematic Benchmark: The first comprehensive measurement study quantifying LLM-based PIE attacks and defenses across multiple models and datasets.
2. Dataset Creation: Curated 4 datasets (1 synthetic, 3 real-world) with manually labeled ground truth for 8 categories of personal information.
3. Novel Defense Strategy: Proposed Prompt Injection as a defensive mechanism. Unlike traditional defenses that obscure data, PI actively misleads the LLM into generating incorrect data while remaining invisible to human users.
4. Adaptive Attack Analysis: Evaluated whether adaptive attacks (e.g., paraphrasing, retokenization, instruction isolation) could bypass the proposed prompt injection defense.

4. Key Results

A. Effectiveness of LLM-based Attacks

• Superior Performance: LLMs significantly outperform traditional methods. For example, on the synthetic dataset, GPT-4 achieved 100% accuracy for email and 98% for phone numbers, whereas traditional methods (like Regex) struggled with non-standard formats and failed completely on semantic categories like "work experience."
• Model Size Correlation: Larger models generally perform better. GPT-4 (100% email accuracy) vastly outperformed smaller open-source models like Vicuna-7b (65% accuracy).
• Category Variance: LLMs excel at extracting structured data (email, phone, name) but show lower accuracy for complex, unstructured data (work experience, affiliation).

B. Evaluation of Defenses

• Failure of Traditional Defenses: Existing methods like Symbol Replacement, Hyperlinks, and Keyword Replacement are largely ineffective against LLMs. For instance, GPT-4 achieved 97-100% accuracy even when email addresses were obfuscated with "AT/DOT" or hidden in hyperlinks.
• Success of Prompt Injection: Prompt injection proved highly effective. When applied, it reduced the extraction accuracy of GPT-4 for emails from 100% to 0%. It successfully misled LLMs into outputting fake data (e.g., "321@gmail.com" instead of "123@gmail.com").
• Generality: Unlike other defenses limited to specific formats (e.g., SR only works for emails), Prompt Injection is applicable to all categories of personal information.
• Robustness Against Adaptive Attacks: The study tested various adaptive attacks (e.g., retokenization, data isolation) designed to bypass prompt injection. These attacks showed limited success, often failing to recover high accuracy or sacrificing the utility of the extraction process.

C. Cost and Usability

• Computational Cost: LLM-based extraction is slower (approx. 1 second per profile) and more expensive than traditional regex but remains feasible for large-scale attacks.
• Financial Cost: Using GPT-4 APIs costs roughly $0.009 per profile, which is significantly cheaper than crowdsourcing (e.g., Amazon MTurk).
• Human Impact: Prompt injection, when implemented correctly (invisible text), has zero impact on normal human users, unlike symbol replacement which requires manual decoding.

5. Significance and Implications

• Security Risk: The study demonstrates that LLMs have democratized high-precision personal information extraction, making large-scale spear phishing and fraud significantly easier and more automated than ever before.
• Paradigm Shift in Defense: The paper challenges the conventional view of prompt injection as purely an offensive tool. It establishes that prompt injection can be weaponized defensively to protect privacy by poisoning the data seen by malicious LLMs.
• Future Directions: The authors highlight that defending against prompt injection remains an open challenge. They suggest future work on multimodal defenses (e.g., image-based prompt injection) to counter attackers who might use screenshots to bypass text-based injections.

In conclusion, the paper provides a critical warning that current privacy protections are insufficient against modern LLMs and proposes a novel, effective, and user-transparent defense mechanism based on the very technology (prompt injection) that attackers exploit.