Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes domains (e.g., healthcare, personal assistants), yet they frequently violate contextual privacy. Unlike static privacy definitions (e.g., preventing data memorization), contextual privacy relies on Contextual Integrity (CI) theory, which posits that privacy is determined by the appropriateness of information flow based on specific social contexts.

The core problem addressed is the Privacy Awareness Gap: LLMs often fail to adhere to privacy norms in practice (leaking private information) despite potentially encoding these norms internally. Existing approaches treat privacy as a monolithic behavioral objective (via fine-tuning or reinforcement learning) or a static property, failing to account for the structured, multi-dimensional nature of privacy norms within the model's latent space. The authors ask: Do LLMs internally encode contextual privacy norms as structured representations, and if so, why do they fail to act on them?

2. Methodology

The authors propose a three-stage framework involving Probing (diagnosis) and Steering (intervention), grounded in CI theory.

A. Probing Framework

The study probes four models (Llama-3.1, Qwen-2.5, Mistral, Llama-2) at three distinct levels to understand how privacy is represented:

1. Concept Level: Tests linear separability of "appropriate" vs. "inappropriate" information flows using contrastive pairs.
2. Behavioral Level: Evaluates whether internal representations translate into compliant behavior in social role-play scenarios (measuring leakage vs. refusal).
3. CI-Parametric Level: Decomposes privacy into specific CI parameters to test if they occupy independent subspaces. The three norm-determining parameters analyzed are:
   • Information Type
   • Recipient
   • Transmission Principle
     (Note: Data Subject and Sender are treated as actors, not norm-determining parameters.)

Techniques:

• Linear Probing: Using logistic regression on hidden states to classify appropriateness.
• PCA (Principal Component Analysis): To identify dominant directions in the residual stream.
• Subspace Selectivity Test: Using Linear Discriminant Analysis (LDA) to verify if steering directions for one parameter (e.g., Recipient) are orthogonal to others (e.g., Information Type).

B. CI-Parametric Steering

Based on probing findings, the authors introduce CI-Parametric Steering, an inference-time intervention method.

• Mechanism: Instead of using a single "privacy vector," the method identifies unit-normalized direction vectors ($v^{(p)}_l$) for each of the three CI parameters.
• Intervention: The hidden state $h_l$ is modified by summing the contributions of all three independent directions:
  $$h'_l = h_l + \alpha \sum_{p \in P} v^{(p)}_l$$
  where $P = \{\text{info type, recipient, transmission principle}\}$ and $\alpha$ is the steering strength.
• Target: Applied to the top-$k$ layers where privacy signals are strongest.

3. Key Contributions & Findings

Finding 1: Privacy is Multi-Dimensional, Not Monolithic

• Linear Separability: Contextual privacy norms are linearly separable in the residual stream, primarily in upper layers.
• Multi-Axial Structure: Privacy cannot be captured by a single direction (k=1 PCA performs poorly, AUROC ~0.43). It requires at least three principal components (k=3, AUROC ~0.77) to be accurately represented.
• Implication: Treating privacy as a single vector (monolithic steering) fundamentally misrepresents the internal structure of the concept.

Finding 2: The Privacy Awareness Gap

• Internal vs. External: Models achieve near-perfect accuracy in classifying privacy norms internally (concept-level probing) but leak private information in up to 42.5% of behavioral scenarios.
• Conclusion: The failure is not a lack of "awareness" or encoding, but a misalignment between representation and behavior generation.

Finding 3: CI Parameters Occupy Independent Subspaces

• Subspace Selectivity: The representations for Information Type, Recipient, and Transmission Principle occupy functionally independent subspaces.
• Evidence: A classifier trained on the "Recipient" subspace achieves ~100% accuracy on Recipient data but drops to ~20% (random chance) on Information Type or Transmission Principle data. This confirms the compositional nature of privacy.

Contribution: CI-Parametric Steering

• The paper introduces a steering method that exploits this compositional structure by intervening along each CI dimension independently.
• This approach allows for robust generalization across different datasets and violation types, unlike monolithic methods which fail to transfer.

4. Experimental Results

The method was evaluated on synthetic data and two benchmarks: CONFAIDE (multi-party confidentiality) and PrivaCI-Bench (legal/regulatory privacy).

Key Observations:

1. Superiority over Monolithic Steering: Monolithic steering (using a single vector) often increased leakage when transferred to new datasets (e.g., CONFAIDE), likely because it disrupted the model's partial compliance with specific dimensions.
2. Robust Transfer: CI-parametric steering, trained only on synthetic data, successfully generalized to complex real-world benchmarks, reducing leakage to near-zero.
3. Stability: CI-parametric steering is less sensitive to the steering strength hyperparameter ($\alpha$) compared to monolithic methods, which showed high volatility.
4. Ablation: Steering along a single CI axis (e.g., only "Recipient") actually increased leakage compared to the baseline, confirming that all three dimensions are necessary to capture the full privacy subspace.

5. Significance and Impact

• Theoretical Insight: The paper challenges the view that privacy failures are due to a lack of internal knowledge. Instead, it demonstrates that LLMs possess a rich, structured, multi-dimensional representation of privacy that is simply misaligned with their output generation.
• Methodological Advancement: It moves beyond "black-box" fine-tuning and monolithic steering. By decomposing complex concepts into their constituent theoretical dimensions (CI parameters), it enables compositional control that is more interpretable and effective.
• Practical Application: CI-parametric steering offers a reliable, inference-time mechanism to enforce privacy in LLMs without retraining, showing that leveraging the internal geometry of the model is more effective than external behavioral constraints.
• Future Direction: The work suggests that for other complex, multi-faceted concepts (e.g., fairness, safety), decomposing them into theoretically grounded subspaces may be the key to effective representation engineering.

In summary, the paper establishes that contextual privacy in LLMs is a structured latent subspace, and that leveraging this structure via multi-axis steering is the most effective way to bridge the gap between internal privacy awareness and external compliant behavior.