Technical Summary: Learning to Adapt Cross-Domain Preferences via Meta-LoRA for LLM Personalization

1. Problem Definition

The paper addresses the challenge of cross-domain zero- or few-shot personalization for Large Language Models (LLMs). The goal is to generate user-preferred responses in an unseen conversational domain (target domain) using only a handful of interactions (or none) from that specific domain, while leveraging a user's historical interactions from other domains (source domains).

The authors identify two primary obstacles in existing approaches:

1. Sparse Evidence Overfitting: Under zero- or few-shot conditions, limited target-domain data provides incomplete or ambiguous evidence. Aggressive adaptation often leads to overfitting on incidental patterns, while conservative updates fail to capture user-specific nuances.
2. Cross-Domain Heterogeneity and Negative Transfer: User preferences are high-dimensional and context-dependent. Existing methods often entangle stable user tendencies with source-domain artifacts (e.g., specific topics or discourse styles). This causes "negative transfer," where preferences valid in one domain degrade performance in another due to mismatched conventions.

2. Methodology

The authors propose a unified framework combining PAC-Bayes-regularized Meta-LoRA with dual-channel prior conditioning.

A. PAC-Bayes-Regularized Meta-LoRA

This component controls the magnitude and geometry of parameter updates during target adaptation to prevent overfitting.

• Meta-Learned Initialization: Through episodic meta-learning on source domains, the model learns a LoRA initialization ($\theta_0$). This serves as both the starting point for adaptation and the center of a prior distribution.
• PAC-Bayes Constraint: The method treats $\theta_0$ as the mean of a Gaussian prior. It minimizes a bound derived from PAC-Bayes theory, which relates empirical risk to the complexity of the posterior relative to the prior.
• Evidence-Calibrated Adaptation: The strength of the anchor (regularization) is dynamically adjusted based on the quality of the target evidence:
  • Support Set Size ($K$): As the number of target interactions decreases, the penalty for deviating from $\theta_0$ increases (scaling with $K^{-1}$).
  • Predictive Entropy: If the meta-initialized model is uncertain (high entropy) on the target support set, the anchor strength increases to prevent the model from over-correcting based on sparse data.
  • Directional Sensitivity: An empirical Fisher proxy is used to weight the regularization, constraining parameter updates more strictly in directions where the support loss is sensitive, while allowing flexibility elsewhere.

B. Dual-Channel Personalization Priors

To address the issue of entangling user preferences with domain artifacts, the framework functionally decomposes personalization into two distinct conditioning channels:

1. User-Side Prior (Textual Prompt): A human-readable prompt ($P_u$) is generated by summarizing the user's cross-domain history. This captures stable, transferable preferences (e.g., "prefers detailed explanations") while suppressing transient or source-specific topics.
2. Domain-Side Prior (Soft Tokens): A set of compact soft tokens ($T_t$) encodes the target domain's conventions (discourse structure, response style, information organization).
   • Graph-Guided Construction: Source domains are represented as nodes in a graph based on semantic, pragmatic, and structural views. For an unseen target domain, the system retrieves and composes relevant source domain representations using a reliability-weighted graph mechanism.
   • Topology-Preserving Projection: The composed domain representation is projected into soft tokens. A topology-preserving loss ensures that the relational geometry between source domains is maintained in the language model's hidden space, preventing the injection of arbitrary tokens that might disrupt the model's internal structure.

C. Unified Inference

The framework supports both zero-shot ($K=0$) and few-shot ($K>0$) settings using the same conditioning interface ($[P_u; T_t; x_t]$).

• Zero-Shot: The model generates directly using the meta-initialized parameters $\theta_0$ conditioned on the priors.
• Few-Shot: The model performs inner-loop updates on the target support set using the evidence-calibrated objective, then generates the final response using the adapted parameters $\theta^*$.

3. Key Contributions

1. PAC-Bayes-Regularized Meta-LoRA: A novel adaptation strategy that uses a meta-learned LoRA initialization as a prior center, calibrating update strength based on support set size and predictive uncertainty to mitigate overfitting in sparse regimes.
2. Functional User-Domain Factorization: A dual-channel conditioning mechanism that separates stable user preferences (injected via readable prompts) from domain-specific realization conditions (injected via topology-preserving soft tokens), enabling reliable transfer across heterogeneous domains.
3. Comprehensive Empirical Validation: Extensive experiments demonstrating consistent improvements across multiple benchmarks (HiCUPID, LaMP-QA), backbone models (LLaMA-3, Qwen3), and settings (zero-shot, few-shot, unseen-user cold start).

4. Experimental Results

The method was evaluated on HiCUPID and LaMP-QA benchmarks, comparing against strong baselines including retrieval-based methods, continuous prompting, parameter-efficient fine-tuning, and meta-learning approaches.

• Cross-Domain Performance: On HiCUPID, the proposed method achieved the highest Win Rate (WR) across all five held-out target domains. Crucially, it reduced cross-domain win-rate degradation ($\Delta WR$) by 47.9% relative to the best competing baseline, indicating significantly more stable transfer.
• Few-Shot Adaptation: The method demonstrated high sample efficiency, reaching 90% of its 10-shot performance with only 3 target interactions.
• Unseen User Cold Start: In scenarios where the target user has no interaction history, the method improved the win rate by 110.2% over the strongest baseline, leveraging graph-guided domain priors to avoid generic responses.
• Generalization: Consistent gains were observed on the Qwen3-8B backbone, confirming the framework is not tied to a specific model architecture.
• Ablation Studies: Removing components such as the $K^{-1}$ scaling, entropy calibration, or topology preservation led to significant performance drops, validating the necessity of each module.

5. Significance and Claims

The paper claims that its primary contribution lies in solving the tension between adaptation speed and generalization stability in cross-domain personalization. By grounding the adaptation process in PAC-Bayes theory, the method provides a principled way to limit parameter drift when evidence is scarce. Furthermore, by decoupling user preferences from domain realizations, it avoids the negative transfer common in history-based methods.

The authors emphasize that their approach does not merely fit the source data better but facilitates the reliable transfer of personalized behavior to unseen domains. The framework is presented as a unified solution that handles both zero-shot and few-shot scenarios without requiring separate training pipelines, making it practical for real-world deployment where target-domain data is often limited or non-existent.