🤖 machine learning

Sharpness-aware Model Merging with Salience Recovery for LLM-based Cross-Domain Sequential Recommendation

This paper proposes SharpRec, a novel framework for LLM-based cross-domain sequential recommendation that addresses knowledge conflicts and performance saturation in model merging through sharpness-aware geometric alignment and preference salience activation to significantly outperform existing state-of-the-art baselines.

Original authors: Huwei Ji, Jiajie Su, Yuyuan Li, Xiaohua Feng, Chaochao Chen

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huwei Ji, Jiajie Su, Yuyuan Li, Xiaohua Feng, Chaochao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to recommend the perfect book to a visitor. If you only know they love sci-fi, you might suggest a space opera. But what if you also knew they secretly love cooking? A truly smart librarian could connect those dots, suggesting a sci-fi novel set in a futuristic kitchen. This is the dream of "Cross-Domain Sequential Recommendation": using what a person likes in one area (like movies) to guess what they might enjoy in another (like books), even if they haven't explicitly told you about their new interests.

To do this, modern computers use "Large Language Models" (LLMs). Think of these as super-smart, well-read brains that understand the deep meaning of words, not just a list of keywords. They can figure out that "Star Wars" and "cooking" might share a theme of "epic journeys," even if the user never bought a cookbook. However, teaching these brains to juggle multiple interests at once is tricky. If you try to mix their knowledge of sports and toys, they might get confused, mixing up the rules of soccer with the rules of building blocks. This paper tackles the problem of how to combine these different "expert brains" into one super-brain without them tripping over each other or forgetting their unique skills.

The researchers, Huwei Ji and their team from Zhejiang University and Hangzhou Dianzi University, discovered that simply smashing different recommendation models together doesn't work well. They found two main problems. First, the models often have "geometric incompatibility." Imagine trying to merge two different puzzle pieces that were carved to fit into very different, jagged holes; if you force them together, the picture gets distorted. Second, they noticed a "saturation" effect. If you keep adding more and more sources of knowledge (like adding books, movies, sports, and toys), the model's performance hits a wall and stops improving. It's like mixing too many paints together; eventually, you just get a muddy brown instead of a vibrant new color. The team calls this "statistical homogenization," where the unique, sharp details of each interest get smoothed out into a boring average.

To fix this, the team built a new framework called SharpRec. They didn't just mash the models together; they gave them a makeover. First, they used a technique called "Sharpness-aware Geometric Alignment." Think of this as sanding down the jagged edges of the puzzle pieces before gluing them. By gently nudging the models to find "flat" spots in their learning landscape (places where small changes don't ruin the result), they made sure the different domains could fit together smoothly without crashing.

Second, they introduced "Preference Salience Activation." When the models were merged, the unique, high-value details (like a specific love for vintage toys) were getting lost in the crowd. SharpRec acts like a highlighter, finding those important, rare details that got smoothed out and boosting them back up. It's like taking a blurry photo and using software to sharpen the most important parts, ensuring the model remembers the specific things that make a user unique, rather than just a generic average.

The team tested SharpRec on real-world data from Amazon, looking at pairs like Books and Movies, or Sports and Toys. They found that their method consistently outperformed all the other top methods. In fact, while other methods hit a performance wall after combining just three or four different domains, SharpRec kept getting better and better as they added more sources. They also showed that it works even when there are very few users who overlap between the two domains, proving it doesn't need a massive crowd of people who like both things to make the connection.

In short, the paper suggests that by carefully aligning the "shape" of the models and actively protecting their most unique features, we can build recommendation systems that are not only smarter but also capable of learning from a vast array of different interests without getting confused or losing their edge. The results, measured by how often the system correctly guessed the next item a user would like, show that this approach is a significant step forward in making AI recommenders that truly understand our complex, multi-faceted lives.

Technical Summary: SharpRec for LLM-based Cross-Domain Sequential Recommendation

1. Problem Statement

Cross-Domain Sequential Recommendation (CDSR) aims to improve recommendation performance in a target domain by leveraging interaction data from auxiliary domains. While Large Language Models (LLMs) have emerged as a powerful tool for CDSR due to their ability to perform deep semantic reasoning and bridge gaps between non-overlapping users, existing paradigms face significant limitations:

LLM-enhanced approaches often suffer from semantic-collaborative misalignment, where forcing items into a unified semantic space distorts dynamic preference modeling.
LLM-based approaches (e.g., unified instruction tuning) struggle with data distribution conflicts across domains and lack flexibility, as domain updates require costly full-model retraining.
Model Merging, a promising alternative that integrates domain-specific adapters (e.g., LoRA) without retraining, encounters two critical empirical bottlenecks when applied to realistic, heterogeneous CDSR scenarios:
1. Cross-Domain Knowledge Conflict: Merging models from divergent domains (e.g., Sports and Toys) leads to severe parameter interference. Empirical analysis reveals that independent fine-tuning traps models in "sharp minima" that are geometrically incompatible, causing negative transfer that cannot be resolved by simple linear weight tuning.
2. Performance Saturation in Multi-Domain Fusion: As the number of source domains increases, performance gains diminish rapidly. This is attributed to "statistical homogenization," where linear aggregation acts as a mean filter, smoothing out salient, high-magnitude parameters (outliers) into a generic, feature-poor Gaussian distribution, thereby erasing the high-order signals essential for capturing complex cross-domain preferences.

2. Methodology: SharpRec

To address these bottlenecks, the authors propose SharpRec (Sharpness-aware Model Merging with Salience Recovery), a framework designed to lift the performance upper bound of merged LLM-based CDSR models. The framework consists of two synergistic modules:

A. Sharpness-aware Geometric Alignment (SGA)

SGA addresses the geometric incompatibility (Challenge 1) by ensuring that domain-specific models reside in connected, low-loss basins prior to merging.

Mechanism: Instead of standard fine-tuning, SGA employs a sharpness-aware optimization objective. It treats the merging process as a worst-case stochastic perturbation, guiding the model toward "flat minima" rather than "sharp minima."
Objective: The optimization minimizes the loss difference between the original parameters and perturbed parameters within a radius $\rho$ . This ensures that the aggregated parameters remain stable under the shifts induced by merging.
Theoretical Basis: The authors prove that the merging interference error is bounded by the product of domain divergence and loss sharpness. By minimizing sharpness ( $\sigma(\theta)$ ), SGA theoretically guarantees that the merging path stays within a low-loss region, mitigating negative transfer.

B. Preference Salience Activation (PSA)

PSA addresses the statistical homogenization (Challenge 2) by reconstructing the parameter distribution post-fusion.

Mechanism: After the initial linear aggregation of adapters, PSA applies a non-linear reparameterization strategy. It first introduces stochastic disentanglement by subtracting Gaussian noise to shift parameters away from the high-density mean.
Transformation: It then applies an element-wise non-linear projection function $T(\cdot)$ that amplifies large-magnitude outliers while suppressing near-zero noise. This transforms the distribution from a narrow Gaussian back into a heavy-tailed distribution.
Goal: This process recovers the "salient preference signals" (high-magnitude weights) that encode domain-specific expertise, which are typically eroded by linear averaging.
Theoretical Basis: The authors demonstrate that this heavy-tailed distribution expands the model's functional space coverage (measured by the Jacobian determinant), allowing the merged model to capture intricate, long-tail user interests that linear fusion suppresses.

3. Key Contributions

Novel Framework: The proposal of SharpRec, a new paradigm for LLM-based CDSR that integrates sharpness-aware optimization and distributional recovery to achieve robust and scalable knowledge transfer.
Technical Solutions:
- SGA: A method to guide models into flat minima, resolving geometric incompatibility and mitigating cross-domain parameter interference.
- PSA: A non-linear reparameterization technique to reactivate salient preference signals, lifting the performance ceiling and alleviating saturation in multi-domain fusion.
Empirical Validation: Extensive experiments across dual-domain and multi-domain scenarios demonstrating that SharpRec consistently outperforms state-of-the-art baselines, including traditional CDSR methods, LLM-enhanced approaches, and existing model merging techniques.

4. Experimental Results

The authors evaluated SharpRec on the Amazon Review 2023 dataset across seven domains (Book, Movie, Food, Kitchen, Sport, Toy, Clothing).

Overall Performance (RQ1): SharpRec achieved state-of-the-art results across all six cross-domain tasks (e.g., Book $\to$ Movie, Sport $\to$ Toy), significantly outperforming baselines like WeaveRec, X-Cross, and LLM4CDSR. For instance, in the Book $\to$ Movie task, SharpRec achieved an NDCG@5 of 79.93%, compared to 40.77% for WeaveRec.
Scalability (RQ2): In multi-domain settings (integrating up to 7 source domains), baseline methods exhibited rapid performance saturation after 3–4 domains. In contrast, SharpRec maintained a robust upward trajectory, achieving continuous gains (e.g., +51.4% improvement over WeaveRec in the Food task with 7 domains).
Ablation Studies (RQ3): Removing SGA led to severe performance degradation due to parameter interference, while removing PSA caused performance saturation in multi-domain settings, confirming the necessity of both modules.
Robustness (RQ5): SharpRec demonstrated stable performance across varying user overlap ratios (20% to 80%), effectively circumventing the "overlap dilemma" that plagues traditional CDSR methods.

5. Significance and Claims

The paper claims that the effectiveness of existing model merging methods in CDSR is fundamentally constrained by geometric parameter interference and the statistical homogenization of preference signals.

The significance of SharpRec lies in its ability to:

Break the Scalability Ceiling: By addressing the root causes of saturation and conflict, SharpRec enables the effective integration of a large number of diverse auxiliary domains, a capability previously limited in CDSR.
Decouple Performance from Data Density: By leveraging semantic alignment and geometric recovery, the framework ensures effective knowledge transfer even in scenarios with minimal user overlap, reducing reliance on explicit co-occurrence signals.
Provide a Robust Paradigm: The work offers a scalable and efficient solution for utilizing open-world knowledge in recommender systems, moving beyond the limitations of linear aggregation and rigid representation alignment.

The authors conclude that SharpRec paves the way for more efficient utilization of LLMs in cross-domain scenarios by ensuring that merged models retain the distinct, high-order features necessary for complex reasoning and preference modeling.