Technical Summary: CIDER - A Dataset of Contextual Disclosure Boundaries for Privacy Preference Alignment

Problem Statement

While Large Language Models (LLMs) are increasingly integrated into daily communication, their ability to align with human privacy preferences remains inconsistent. Existing research primarily relies on the Contextual Integrity (CI) framework to evaluate models against general, group-level privacy norms derived from regulations and literature. However, these norms are inherently coarse-grained and fail to capture the dynamic, nuanced, and individualized privacy behaviors people exhibit in real-life scenarios.

The core challenge lies in eliciting and modeling individual privacy boundaries. Unlike public norms, personal boundaries are formed implicitly through lived experience and situational risk perceptions, making them difficult for users to articulate explicitly. Current evaluation methods lack a systematic operationalization for assessing an LLM's capacity to adapt to these specific, context-dependent disclosure preferences, particularly in scenarios involving AI-mediated communication.

Methodology

The CIDER Dataset

To address this gap, the authors introduce CIDER (Contextual Information Disclosure Boundaries Elicited from Real Users), a dataset comprising 14,850 human annotations from 169 participants across 60 interpersonal communication scenarios.

• Scenario Construction: Scenarios were derived from the PrivacyLens dataset, covering diverse data-sharing practices that violate privacy norms. Each scenario was instantiated into multiple contexts by varying the communication role (sender, subject, recipient) and the AI-mediated condition (human vs. AI assistant).
• Disclosure Variants: For each context, the authors generated 9 disclosure variants systematically varying along two dimensions:
  1. Granularity: General (G1), Moderately Detailed (G2), and Very Detailed (G3).
  2. Identifiability: Not Identifiable (I1), Partially Identifiable (I2), and Fully Identifiable (I3).
• Data Collection: Participants rated the acceptability (Yes/No) of each variant within their assigned context. The dataset forms 1,650 contextual disclosure boundaries, representing individualized decision patterns.

Task Formulation

The paper formulates a privacy preference prediction task. Given a user's historical disclosure boundaries ($B$) across $N$ contexts, a model must infer the user's latent privacy preference ($p$) and predict their disclosure decisions ($\hat{y}$) for a new, unseen context ($c_{new}$).

The evaluation focuses on inference-time personalization (in-context learning) without parameter updates. The authors compare four conditions:

1. HC (High Context): Historical boundaries provided with full semantic context for scenarios and variants.
2. HL (Low Context): Historical boundaries provided with structural labels (granularity/identifiability) but redacted semantic content.
3. H (History Only): Historical boundaries provided as binary decisions without context or labels.
4. No-History Baseline: Prediction based solely on the target scenario without user history.

Experiments were conducted across 12 open and proprietary LLMs (including GPT-5.4, Claude Sonnet 4.6, Llama 4, and Qwen3 families) with varying history lengths ($k \in \{1, 4, 5, 6\}$).

Key Contributions

1. CIDER Dataset: The first personalized dataset capturing individualized contextual disclosure behavior, moving beyond group-level norms to model specific user boundaries.
2. Operationalization of Personal Boundaries: A structured framework for representing privacy preferences as binary vectors over disclosure variants defined by granularity and identifiability.
3. Evaluation Benchmark: A rigorous protocol for evaluating LLMs on personalized privacy alignment using in-context learning, highlighting the trade-offs between model scale, reasoning capabilities, and personalization fidelity.
4. Study Toolkit: A reusable pipeline for generating disclosure variants and eliciting decisions from real users, supporting future longitudinal studies.

Results

Prediction Accuracy and Personalization

• General Improvement: In-context personalization generally improves prediction accuracy. Using only 6 historical examples ($k=6$), accuracy improved by up to 11.41 percentage points compared to the no-history baseline.
• Model Scale and Reasoning: Larger, reasoning-capable models (e.g., GPT-5.4 with medium reasoning effort, Claude Sonnet 4.6) significantly outperform smaller models. These models effectively leverage semantic context (HC condition) to understand user-specific preferences.
  • GPT-5.4 achieved 72.45% accuracy and Claude Sonnet 4.6 achieved 71.98% under HC with $k=6$.
  • Smaller models (e.g., Llama 3.1 8B, Qwen3-8B) often plateau at lower accuracies (e.g., ~59.70%) and tend to rely on structural heuristics (granularity and identifiability labels) rather than semantic understanding. In some cases, providing full semantic context (HC) did not help, or even hindered, smaller models compared to the HL condition.

Error Analysis and Trade-offs

• Imbalanced Error Shifts: While personalization improves overall accuracy, it often leads to imbalanced shifts in False Positives (FP) and False Negatives (FN).
  • Most models reduce FN for "easy" variants (low granularity/identifiability) but increase FP.
  • Conversely, for "hard" variants (high granularity/identifiability), models often reduce FP but increase FN.
• Balanced Performance: Claude Sonnet 4.6 was the only model to achieve balanced improvements, reducing both FP and FN rates across all variants.
• Failure Modes: Analysis of incorrect predictions revealed four error types:
  1. Self-Disclosure Bias: Models assume self-disclosure is always acceptable.
  2. Pattern Extrapolation: Models blindly extrapolate historical "Yes" patterns without semantic reasoning.
  3. Norm-Based Override: Models prioritize general ethical norms over specific user preferences.
  4. Rule-Based Generalization: Models derive simplified rules (e.g., "trusted sender = Yes") that fail in nuanced new contexts.

Personalization vs. Group/Norm Alignment

Comparative analysis showed that individual-level predictions tied or outperformed group-based boundaries in 67.92% of cases and norm-based boundaries in 60.69% of cases, validating the necessity of personalized modeling.

Significance and Claims

The paper positions CIDER as a critical resource for advancing personalized privacy alignment. The findings reveal a dual reality in current LLM capabilities:

1. Promise: Frontier models with strong reasoning capabilities can effectively learn and adapt to individual privacy boundaries using in-context history, significantly outperforming static, norm-based approaches.
2. Limitations: Current models, particularly smaller on-device models, struggle to leverage semantic context for personalization and often exhibit imbalanced error rates (trading off false positives for false negatives). Furthermore, the reliance on centralized infrastructure for large models presents a privacy paradox, as transmitting user data for personalization creates its own privacy risks.

The authors conclude that aligning LLMs with personal privacy preferences requires not only richer behavioral history but also improved model capabilities to interpret and utilize such information, especially for small-scale, on-device deployment. They call for future work focusing on targeted strategies (e.g., fine-tuning, better prompting) to enhance personalization fidelity in smaller models.