Inferential Privacy Leakage in Anonymized Conversational AI Logs

Technical Summary: Inferential Privacy Leakage in Anonymized Conversational AI Logs

Problem Statement
The paper addresses a critical gap in conversational AI privacy: the assumption that removing explicit Personally Identifiable Information (PII) from user logs is sufficient to protect user identity. While traditional sanitization removes names, locations, and direct demographic statements, the authors investigate whether a user's demographic identity (age, gender, country) remains recoverable from the style, topic mixture, and cultural markers inherent in their conversation history. The study posits that Large Language Models (LLMs) can infer these attributes through stereotyped reasoning patterns, even when the input data has been rigorously filtered to exclude explicit self-disclosures.

Methodology
The research utilizes two datasets collected via a data-donation pipeline from consenting participants recruited through Clickworker, under IRB approval.

1. Datasets:

   • Dataset 1 (Multi-country): 1,057 users from Brazil, India, Nigeria, and Pakistan. It includes full ChatGPT conversation histories and basic demographics (age, gender, country).
   • Dataset 2 (Indian Sub-cohort): A subset of 212 Indian users from Dataset 1 who also provided Google Search, YouTube Search, and YouTube Watch histories, along with expanded demographics (religion, caste, income, education, voting preference).

2. Filtering and Cohort Selection:

   • To test inference beyond explicit disclosure, the authors constructed a "conservative analytic cohort."
   • Filtering: They applied a two-stage filter to user messages:
     • A deterministic Named Entity Recognition (NER) pass (SpaCy) to flag standard PII.
     • An LLM-based filter (Llama-3.3-70B-Instruct) to detect implicit demographic self-disclosures (e.g., "As a single mother...").
   • Cohort: Only users whose entire conversation history passed both filters (no flagged messages) were retained. This ensures that any successful inference relies solely on stylistic and topical signals, not explicit statements.

3. Inference Protocol:

   • Adversary: The same Llama-3.3-70B-Instruct model was used as the adversary to predict demographics.
   • Task: The model predicted age bracket, gender, and country (and additional attributes for Dataset 2) using progressively larger prefixes of the sanitized conversation history (5% increments).
   • Metrics: Performance was measured using weighted F1, precision, and recall against ground-truth survey data. The "context-needed" metric tracked the minimum prefix size required for a correct prediction.
   • Qualitative Analysis: The model's chain-of-thought rationales were analyzed to identify the reasoning patterns driving predictions.

4. Cross-Platform Comparison:

   • For Dataset 2, the same inference protocol was applied to Google Search, YouTube Search, and YouTube Watch histories to compare ChatGPT logs against established behavioral profiling substrates.

Key Contributions

1. Dataset Assembly: Creation of a donated corpus of 1,057 ChatGPT histories from four Global South countries, paired with self-reported demographics, and a sub-cohort with parallel search/watch histories.
2. Disclosure Audit: A granular analysis of 1.24 million messages revealing that 34.5% contain personal information, with the median user revealing identifying content within the first 14% of their history.
3. Inference on Sanitized Data: Demonstration that demographics remain highly recoverable from conversations that have passed strict filters for explicit self-disclosure.
4. Bias Mechanism Identification: Qualitative analysis revealing that inference relies on four recurring stereotype patterns, leading to asymmetric error distributions.
5. Cross-Platform Benchmarking: A direct comparison showing ChatGPT logs are a comparable, though distinct, inference surface to traditional search and browsing logs.

Results

• Explicit Disclosure: 34.5% of user messages contained personal information across a 20-category taxonomy. The median user revealed identifying content within the first 14% of their history, with a spike in first-message disclosures. Disclosure rates did not attenuate as conversations lengthened.
• Inference Performance (Sanitized Cohort): Even with no explicit demographic flags, the LLM achieved high accuracy:
  • Gender: Weighted F1 of 0.90 (Baseline: 0.52).
  • Country: Weighted F1 of 0.88 (Baseline: 0.26).
  • Age: Weighted F1 of 0.84 (Baseline: 0.23).
  • Speed: For over half the users, the model correctly predicted demographics using only the first 5% of the conversation history.
• Reasoning Patterns (Bias): Analysis of the model's rationales revealed four stereotyped cues driving inference:
  1. Tech ≡ Male: Technical content (programming, Linux, finance) was almost exclusively read as male. 26% of women in technical fields were misclassified as men, compared to 1% of men misclassified as women.
  2. Tech ≡ Western: Technical proficiency in Global South users (Nigeria, Pakistan) was often interpreted as "Western-style education," leading to misclassification as American or British.
  3. English Fluency ≡ US: Fluent English lacking local markers (currency, slang) was defaulted to US residence.
  4. Contemporary Content ≡ Young: Older users discussing modern software or career transitions were frequently misclassified into younger age brackets (25–34).
• Cross-Platform Comparison:
  • ChatGPT logs were the strongest signal for age, education, and voting preference.
  • Search and YouTube logs were stronger for gender, religion, and income.
  • YouTube Watch history was consistently the weakest signal.
  • ChatGPT does not subsume older substrates but offers a qualitatively different, narrative-rich profiling surface.

Significance and Claims
The paper claims that message-level PII removal is insufficient as a privacy intervention for conversational AI. The study demonstrates that:

1. Inference is distinct from memorization: The attack does not rely on the model having seen the specific user's data during training. Instead, it relies on the model's general pretraining-derived priors and stereotypes applied to new text.
2. Privacy harm is unevenly distributed: The error distribution is not random; it systematically misclassifies specific groups (women in tech, older users with contemporary skills, Global South professionals). This creates a dual harm: stereotype-conforming users are accurately profiled, while stereotype-violating users are misassigned, leading to representational harm and incorrect downstream targeting.
3. ChatGPT is a new, comparable profiling substrate: While not uniformly more invasive than search logs, ChatGPT logs capture a different class of data (narrative, emotional, reflective) that complements existing behavioral advertising substrates.
4. Policy Implications: Current anonymization practices (redacting PII) fail to address the inferential risks posed by stylistic and topical features. The authors argue for conversation-level privacy interventions (e.g., stylistic obfuscation, differential privacy) rather than isolated message redaction.

The authors explicitly state they do not claim ChatGPT is uniformly more invasive than search, nor do they measure downstream commercial outcomes. Their contribution is the structural observation that identity is recoverable from sanitized logs via stereotype-mediated reasoning, and that this risk is distributed unevenly across populations.