Technical Summary: PRIVACYSIM: Evaluating LLM Simulation of User Privacy Behavior

Problem Statement

Large Language Models (LLMs) are increasingly employed to simulate human behavior for evaluating data-sharing systems, potentially replacing costly and slow controlled user studies. However, it remains unclear whether LLMs can faithfully simulate individual-level privacy decisions. Existing work often relies on generalized personas or population-level alignment, failing to capture the nuances of individual behavior, particularly the "privacy paradox" where users' stated attitudes diverge from their actual actions. There is a lack of standardized evaluation suites to assess whether specific user persona attributes can drive LLMs to accurately mimic individual privacy decision-making in AI-mediated contexts.

Methodology

The authors introduce PRIVACYSIM, an evaluation suite designed to benchmark LLM simulation of user privacy behavior against ground-truth human responses.

Dataset Construction

The evaluation is grounded in real-world data rather than synthetic generation. The authors curated a dataset from five published user studies covering diverse domains:

1. LLM Healthcare Consultation
2. AI Agent Permissions
3. LLM Chatbot
4. Chatbot Group Chat
5. LLM Conversational Agents

From these studies, they sampled 1,000 users (200 per study) and extracted 9,400 user-question pairs. Each user is associated with a "privacy persona" ($P_u$) composed of three hypothesized facets:

• Demographics ($D_u$): Age, gender, education (optional, as some studies omitted them).
• Previous Experiences ($E_u$): Familiarity and usage frequency of AI/chatbots.
• Privacy Attitudes ($A_u$): Stated concerns and preferences regarding data usage.

Simulation Task

The task is defined as an LLM simulator $f$ taking a user's persona $P_u$ and a data-sharing question $q_i$ to produce a simulated response $\hat{r}_{u,i}$. The goal is to align $\hat{r}_{u,i}$ with the user's actual response $r_{u,i}$.

Evaluation Metric

Due to varying response scales across studies (e.g., 5-point Likert vs. 0–100 numeric), the authors employ a tolerance accuracy metric. A simulated response is considered a match if it falls within a tolerance bound ($\tau \cdot W_{Q_u}$) of the actual response, where $W_{Q_u}$ is the scale's upper bound. The authors set $\tau = 0.15$, which effectively acts as an exact match for 5-point scales but allows a $\pm 15$ point margin for 100-point scales.

Experimental Setup

The study evaluated nine frontier LLMs (both open-weight and closed-source), including GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Pro, and various Qwen and Gemma variants. The models were tested under eight prompt conditions:

1. No persona (baseline).
2. Subsets of the three persona facets ($D_u$, $E_u$, $A_u$) in all non-empty combinations.
3. Variants incorporating privacy theories (Privacy Calculus, Bounded Rationality, Protection Motivation Theory) as reasoning lenses.

Key Contributions

1. PRIVACYSIM Benchmark: The first standardized evaluation suite for assessing LLMs as individual-level privacy simulators, utilizing real user data from five distinct studies.
2. Persona Facet Analysis: A systematic investigation into how demographics, previous experiences, and stated attitudes contribute to simulation accuracy.
3. Privacy Paradox Quantification: Empirical evidence demonstrating that stated privacy attitudes alone are often poor predictors of actual behavior in simulation, highlighting the difficulty of modeling the privacy paradox.
4. Clustered Difficulty Analysis: Identification of specific user groups (based on stance and exposure) that are disproportionately difficult for LLMs to simulate.

Results

RQ1: Extent of Simulation Accuracy

• Limited Fidelity: Even the strongest model, Gemini 3.1 Pro, achieved only 40.4% accuracy when conditioned on the richest available persona combination (Previous Experiences + Privacy Attitudes) across all five studies. This indicates that current LLMs are far from faithfully simulating individual privacy decisions.
• Persona Value: Conditioning on persona facets consistently improved accuracy over the no-persona baseline (which achieved ~36–37%). The combination of Previous Experiences ($E_u$) and Privacy Attitudes ($A_u$) yielded the best results, while Demographics ($D_u$) alone provided negligible gains.
• Domain Variance: Simulation quality varied significantly by domain. "LLM Healthcare Consultation" was the most difficult to simulate (lowest accuracy), whereas "LLM Conversational Agents" and "Chatbot Group Chat" were easier.

RQ2: Hardest-to-Simulate Groups

Using a (stance, exposure) taxonomy, the authors identified distinct user clusters:

• Hardest Group: Users with low stated privacy concern but high AI/chatbot exposure ("Low-stance / High-exposure") were the most difficult to simulate, achieving the lowest accuracy (36.51%) and the smallest gain from persona conditioning (+1.07 percentage points). The authors hypothesize that LLMs incorrectly infer these users are less concerned than they actually are, failing to capture the divergence between their stated low concern and their actual behavior shaped by extensive usage.
• Easiest Group: Users with high stated concern and moderate exposure were the easiest to simulate, showing the largest improvement from persona conditioning (+8.53 percentage points).

RQ3: Model Characteristics and Prompting

• Reasoning Effort: Increasing reasoning effort (e.g., from low to medium) had a negligible impact on accuracy (<1 percentage point).
• Model Size: Scaling model size within a family (e.g., Mini vs. Full, 27B vs. 122B) yielded only modest gains (0.12 to 1.31 percentage points).
• Theory-Informed Prompts: Prompting models to reason through established privacy theories (Privacy Calculus, Bounded Rationality, Protection Motivation Theory) did not improve simulation quality beyond the signal provided by the persona facets alone.

Significance and Claims

The paper concludes that while LLMs possess a non-trivial baseline understanding of general privacy norms, they currently lack the fidelity to serve as reliable proxies for individual-level privacy decision-making. The gap between what users say (attitudes) and what they do (behavior) remains the central obstacle.

The authors position PRIVACYSIM as a first step toward understanding these limitations. They claim that LLM simulation is currently appropriate only for low-stakes, early-stage tasks (e.g., comparing design alternatives, surfacing obvious violations, generating hypotheses) but must be paired with human user studies before any deployment decision. The work suggests that bridging the gap to faithful simulation will likely require richer behavioral signals beyond the current set of persona facets, rather than simply scaling models or refining prompts.