Technical Summary: Security and Privacy Prompts in the Wild

Problem Statement

While Large Language Models (LLMs) are increasingly utilized as everyday information sources, their role in digital security and privacy (S&P) remains understudied. Prior research on LLM response quality in S&P domains relied on expert-authored misconceptions or FAQs rather than actual user queries. Consequently, it is unclear what specific S&P questions users pose to LLMs, how well models answer them, and whether responses are consistent across repeated queries. Inaccurate or contradictory S&P advice can lead to real-world consequences, such as compromised accounts or exposed digital identities, and conflicting guidance can undermine user protective behaviors.

Methodology

The study utilized WildChat, a dataset of 3.2 million real-user conversations, to construct a representative corpus of S&P interactions.

1. Data Curation and Classification:

   • From 1.7 million unique English prompts (after filtering for toxicity, length, and non-S&P content), the authors identified 14,727 S&P prompts using a three-LLM majority voting classifier (Qwen 3, Calme 3.2, and GPT 5.2).
   • These prompts were categorized into nine S&P topics (e.g., Account & Auth, Data Privacy, Social Engineering) based on a taxonomy adapted from prior literature.

2. Thematic Analysis (RQ1):

   • A stratified sample of 450 prompts (50 per category) underwent thematic analysis by two researchers. This process generated six primary themes and 22 sub-themes to characterize user intent.

3. Response Quality and Consistency Evaluation (RQ2 & RQ3):

   • The authors curated 270 "advice-seeking" prompts (requests for recommendations, guidance, or specific information), excluding coding or creative writing tasks.
   • Models Evaluated: Five LLMs were tested: three commercial (GPT 5.5, Gemini 3.1, Claude 4.7) and two open-weight (Qwen 3, Llama 4).
   • Procedure: Each prompt was queried 10 times per model to generate 2,700 responses per model.
   • Quality Metric: Responses were scored using an LLM-as-judge checklist method. Binary checklists specifying correct answer components were generated for each prompt. Scores (1–10) were averaged across three scorer models to mitigate self-preference bias.
   • Consistency Metric: Consistency was measured using bidirectional entailment on evidence quotes extracted by the scorer models. A prompt was deemed consistent if the entailment score between pairs of responses across the 10 runs exceeded 0.5.

Key Contributions

• First Empirical Characterization: This study provides the first analysis of real-world S&P prompts directed at LLMs, moving beyond synthetic or expert-curated datasets.
• Taxonomy of User Intent: The research identifies unique interaction patterns, including users treating LLMs as defensive agents, probing model internals, and making adversarial requests.
• Dual Evaluation Framework: The paper demonstrates that response quality and consistency are distinct dimensions of reliability, necessitating separate evaluation metrics in S&P contexts.

Results

1. Thematic Analysis of User Prompts

The analysis of 450 prompts revealed six themes:

• General Knowledge (33.3%): The most prevalent theme, including broad informational queries and exam-style questions.
• User-side Navigation (20.9%): Focused on incident response, such as drafting appeal letters for account bans.
• Defensive Action (11.8%): Users seeking protective assessments (e.g., "Is this app safe?") or counter-fraud strategies (e.g., scam-baiting).
• Inquiry about the LLM (10.2%): Users probing the model's own system, training data, and privacy practices (e.g., "Are our conversations recorded?").
• Harmful & Offensive Requests (6.9%): Requests to assist in attacks, including person-directed doxxing, system exploitation, and platform circumvention.
• S&P Task Production: Requests to generate security-related content.

2. Response Quality

Commercial models significantly outperformed open-weight models:

• GPT 5.5 achieved the highest mean score (8.67/10), with 98% of prompts receiving a "good" or better rating (score ≥ 7).
• Gemini 3.1 (8.52) and Claude 4.7 (8.47) followed closely.
• Qwen 3 scored 7.90, while Llama 4 scored lowest at 6.71, with only 47% of its responses meeting the "good" threshold.
• Qualitative Findings: High-quality responses were factually accurate and ethically appropriate. Low-quality responses often failed on factual accuracy (e.g., hallucinating features) or violated ethical criteria by providing harmful instructions without warnings.

3. Response Consistency

Consistency did not strictly correlate with quality:

• Llama 4 produced the most consistent responses (97.4% of prompts had non-contradictory runs), despite having the lowest average quality.
• GPT 5.5 was also highly consistent (97.0%).
• Qwen 3 was the least consistent (89.6%).
• Critical Observation: Even high-quality models (e.g., Gemini 3.1) occasionally produced contradictory advice on the same prompt across runs (e.g., conflicting file-system permission configurations), which poses a risk of confusing users.

Significance and Claims

The paper claims to fill a critical gap in understanding how users interact with LLMs for security and privacy. Its primary significance lies in:

1. Revealing Dual-Use Dynamics: The study highlights that LLMs function not just as information sources but as interactive agents for both defensive and offensive S&P tasks. This creates a "two-sided detection problem" where models must refuse malicious requests without over-refusing legitimate defensive inquiries.
2. Threat Surface Expansion: Users actively probe LLMs for internal information (API keys, training data), shifting part of the threat surface to the model itself.
3. Reliability Dimensions: The authors argue that response quality and consistency are complementary dimensions. In S&P contexts, a model that provides high-quality but contradictory advice can be as harmful as a low-quality model, as inconsistency erodes user trust and protective behaviors.
4. Evaluation Imperative: Future work and model deployment in S&P domains must evaluate and report both quality and consistency to fully characterize reliability.

The authors acknowledge limitations, including a potential bias toward tech-savvy users in the WildChat dataset, a focus on English prompts, and the analysis of single-turn prompts rather than multi-turn dialogues.