Automated Profile Inference with Language Model Agents

1. Problem Definition: Automated Profile Inference

The paper identifies a new privacy threat to online pseudonymity called Automated Profile Inference.

• The Threat: An adversary can instruct Large Language Models (LLMs) to autonomously collect, analyze, and extract sensitive personal attributes from a user's publicly available activities (posts, comments) on pseudonymous platforms (e.g., Reddit, Twitter).
• The Gap: While traditional profiling requires significant manual effort and expertise, LLMs can automate this process at scale. Existing LLM-based privacy studies often focus on extracting predefined Personally Identifiable Information (PII) via classification. This paper addresses a more practical and dangerous scenario: autonomous inference where agents discover implicit, unstructured, and sensitive information (SPI) without predefined targets, potentially leading to de-anonymization.
• The Challenge: Real-world user activities are noisy, inconsistent, and often contain only indirect clues (e.g., mentioning a car model implies height). Naive LLM prompting fails to extract these implicit details effectively.

2. Methodology: AutoProfiler

The authors propose AutoProfiler, a multi-agent framework inspired by criminal offender profiling methodologies. It decomposes the complex task of profiling into four specialized, collaborative LLM agents working in an iterative workflow:

1. Strategist: The coordinator. It analyzes the current state, decides the next action (Retrieve, Infer, Refine, or Finish), and directs other agents. It maintains a high-level view of the profiling progress.
2. Retriever: Responsible for data collection. It uses platform APIs to fetch user activities (e.g., comments) in reverse chronological order.
3. Extractor: The analyst. It examines specific batches of retrieved text to infer personal attributes (e.g., occupation, health status, family situation). It outputs structured JSON with the attribute type, value, confidence score (1-5), and evidence.
4. Summarizer: The quality controller. It reviews inferred attributes to resolve inconsistencies, remove duplicates, merge conflicting data, and refine vague guesses into specific facts.

Key Technical Features:

• Iterative Workflow: The system processes data in batches (e.g., 10 comments at a time). The Strategist loops through retrieval and inference until no new data is available or the profile is complete.
• Structured Communication: Agents communicate via JSON schemas rather than natural language to ensure consistency and reduce parsing errors.
• Memory Management: To handle long context windows, the system uses short-term memory for immediate tasks and long-term memory (structured summaries) for the Strategist and Summarizer to retain critical profile data without exceeding token limits.
• Zero-Shot Learning: Agents are guided by detailed role descriptions rather than hand-crafted few-shot examples, allowing them to adapt to diverse and unpredictable user content.

3. Key Contributions

1. Definition of a New Threat: Formalized "Automated Profile Inference" as a distinct privacy risk where LLM agents autonomously construct detailed user profiles from public data, bypassing the need for human expertise.
2. AutoProfiler Framework: Developed a multi-agent system that significantly outperforms single-model approaches and traditional PII extraction tools in accuracy and scope.
3. Comprehensive Evaluation: Evaluated the framework on:
   • Real-world Datasets: 250 Reddit users and 220 Twitter users (including verified figures and researchers).
   • Synthetic Dataset: The SynthPAI benchmark.
4. De-anonymization Case Study: Demonstrated that inferred attributes (e.g., location, occupation, education) can be cross-referenced with public data (LinkedIn) to uniquely identify pseudonymous users.
5. Mitigation Analysis: Explored the limitations of current text anonymizers and proposed platform-side and LLM-side mitigation strategies.

4. Experimental Results

• Effectiveness: AutoProfiler achieved high accuracy in inferring both PII (e.g., gender, occupation) and SPI (e.g., health conditions, family trauma).
  • On the Twitter dataset, the full framework achieved 92% identity-level accuracy (correctly guessing the name of verified public figures) using GPT-5, significantly outperforming baselines.
  • On the SynthPAI dataset, AutoProfiler outperformed state-of-the-art baselines (PII-Extractor, FTI, PIE) across all attribute categories (e.g., 95.4% accuracy for Occupation vs. 73.9% for the best baseline).
• Efficiency & Cost:
  • Speed: The system is 120x faster than human profilers (approx. 30 seconds vs. 1 hour per user).
  • Cost: It is 50x cheaper than human labeling (approx. $0.59 vs. $30 per user).
• De-anonymization: In a case study on Reddit, the system successfully narrowed down 10 high-risk users to fewer than 5 LinkedIn candidates each, with one user being uniquely identified using only four attributes (Location, Occupation, Education, Gender).
• Robustness: The framework remained effective even when 10% of user comments were replaced with noise (random comments from other users), demonstrating the Summarizer's ability to filter irrelevant data.

5. Significance and Implications

• Privacy Crisis for Pseudonymity: The study proves that online pseudonymity is no longer a reliable shield against privacy breaches. Even without direct identifiers, the accumulation of "harmless" public posts can be synthesized by LLMs to reveal sensitive identities and private details.
• Limitations of Current Defenses: Standard text anonymization tools (which mask specific entities like names) are ineffective against this threat because LLMs infer sensitive info from context and nuance, not just explicit keywords.
• Malicious Potential: The inferred profiles can be used for:
  • De-anonymization: Linking pseudonyms to real identities.
  • Targeted Attacks: Enabling "Pig Butchering" scams (financial fraud) or child exploitation by identifying vulnerable users based on inferred emotional states or family situations.
• Call to Action: The authors advocate for increased public awareness, stricter API rate limits by platforms, and the development of new privacy-enhancing technologies specifically designed to counter LLM-based inference attacks. They have responsibly disclosed findings to major LLM providers and platform operators.

In conclusion, AutoProfiler demonstrates that the combination of autonomous agents and LLMs creates a powerful, low-cost, and highly effective tool for privacy invasion, fundamentally challenging the security of online pseudonymous platforms.