Improving User Privacy in Personalized Generation: Client-Side Retrieval-Augmented Modification of Server-Side Generated Speculations

Technical Summary: Improving User Privacy in Personalized Generation

Problem Statement
Personalization in Large Language Models (LLMs) typically relies on Retrieval-Augmented Generation (RAG), where relevant context from a user's private profile is retrieved and fed to the model to tailor responses. However, this creates a fundamental trade-off:

1. Server-Side RAG: Offers high-quality personalization using powerful cloud models but requires disclosing private user profiles to the provider, posing significant privacy risks.
2. Client-Side RAG: Preserves privacy by keeping the profile on the user's device but is limited by the computational constraints of edge hardware, forcing the use of smaller, less capable models that often underperform.

Existing privacy-preserving alternatives, such as textual/vector obfuscation, are vulnerable to inversion attacks and allow providers to probe obfuscated data. Homomorphic encryption is currently impractical due to prohibitive computational overhead. The paper addresses the need for a method that leverages high-capacity server-side models for personalization without directly exposing the user's raw private context.

Methodology: The $P^3$ Framework
The authors propose $P^3$ (Client-Side Retrieval-Augmented Modification of Server-Side Generated Speculations), an interactive framework that decomposes generation into a collaborative process between a large server-side model ($M_{server}$) and a small client-side model ($M_{client}$).

• Architecture:

  • Server-Side: A large, high-capacity model generates a sequence of $k$ "draft" tokens based only on the user query, without access to the user profile.
  • Client-Side: A smaller model, deployed on the user's device, has access to the user's private profile. It retrieves relevant context ($C_q$) from the profile based on the query.
  • Verification & Correction Loop:
    1. The server proposes $k$ tokens.
    2. The client evaluates these tokens conditioned on the retrieved personal context.
    3. For each draft token $T_i$, the client calculates its probability ($P_{T_i}$) and compares it to the probability of the most likely alternative token ($T^*_i$) according to the client model.
    4. Acceptance: If the ratio $P_{T_i} / P_{T^*_i} \geq \tau$ (where $\tau$ is a rejection threshold), the token is accepted.
    5. Correction: If the ratio falls below $\tau$, the token is rejected and replaced with $T^*_i$. The remaining draft tokens in that batch are discarded, and the server generates a new batch conditioned on the corrected prefix.
  • Privacy Filter: Before sending the corrected prefix back to the server, the client filters out any Personally Identifiable Information (PII) found in the retrieved context, replacing it with a placeholder (e.g., [PII]). This ensures the server never sees the raw profile or PII-laden corrections.

• Distinction from Speculative Decoding: Unlike standard speculative decoding where a small model drafts and a large model verifies to reduce latency, $P^3$ reverses roles (large drafts, small verifies) to prioritize privacy and personalization. The client model intentionally deviates from the server's distribution to steer the output toward user preferences.

Key Contributions

1. Framework Design: Introduction of $P^3$, the first retrieval-augmented method for LLM personalization that avoids direct disclosure of user profiles to the server.
2. Privacy-Utility Balance: The method achieves high personalization quality by leveraging server-side capabilities while restricting information leakage to only the minimal information implicit in the accepted or corrected tokens.
3. Open Source: The authors open-sourced the implementation to foster further research in privacy-aware retrieval-augmented generation.

Experimental Results
Experiments were conducted on the LaMP-QA benchmark, comprising three datasets: Art & Entertainment, Lifestyle & Personal Development, and Society & Culture. The system used Qwen 2.5 (14B/3B) and Gemma 3 (12B/4B) families.

• Performance:
  • $P^3$ consistently outperformed both non-personalized server-side baselines and fully client-side personalized baselines across all datasets.
  • It achieved statistically significant improvements of 7.4% to 9% on average over the best non-leaky baselines.
  • $P^3$ recovered 90.3% to 95.7% of the utility of an "upper-bound" scenario where the full profile was exposed to the server (a "leaky" setting).
• Privacy Analysis:
  • Linkability Attack: An attacker attempting to identify the user profile from a set of candidates achieved only 24% accuracy with $P^3$, compared to 22.5% for a query-only baseline and 20% for a random baseline. This represents a marginal increase of 1.5% in leakage over the non-personalized baseline.
  • Attribute Inference Attack: The attacker's accuracy in inferring user attributes (e.g., occupation, location) was 58.5% with $P^3$, only 3.5% higher than the query-only baseline (55%). This is significantly lower than the 100% accuracy achieved when the full profile is shared.
• Efficiency:
  • The client-side model generates only 9.2% of the total tokens in the final response.
  • Client-side runtime accounts for only a fraction of the total latency (e.g., ~4.9s out of 23.6s total), with the majority of computation occurring on the server.
  • $P^3$ with a 3B client model significantly outperformed a fully client-side RAG baseline using a 3B model while using less user-side runtime and VRAM.

Significance and Claims
The paper claims that $P^3$ provides a practical and effective solution for personalized generation that improves privacy without sacrificing performance. It demonstrates that it is possible to utilize powerful cloud-hosted LLMs for personalization while retaining control over personal data. The authors position this work as a step toward principled risk reduction in privacy-sensitive applications, offering a middle ground between the absolute isolation of local models and the unrestricted disclosure of traditional server-side RAG. The results suggest that $P^3$ introduces only marginal additional leakage compared to submitting a query without any personal context, making it a viable alternative for scenarios where strict privacy guarantees are required but high model utility is also desired.