🤖 machine learning

GOD: Enhancing Generalization via Deep Grafting for Sequential Recommendation

This paper proposes GOD (Graft-Oriented Distillation), a component-level knowledge distillation framework that enhances sequential recommendation generalization by constructing hybrid models through grafting to disentangle and provide precise feedback on student embeddings and encoders, thereby outperforming state-of-the-art baselines without incurring additional inference costs.

Original authors: WooJoo Kim, JunYoung Kim, JaeHyung Lim, HwanJo Yu

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: WooJoo Kim, JunYoung Kim, JaeHyung Lim, HwanJo Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, algorithms constantly try to guess what we want next. Whether it is a song, a movie, or a product, these systems learn from our past actions to predict our future choices. This field, known as sequential recommendation, relies on the idea that our history holds the key to our next move. However, real human behavior is messy. Our interaction histories are often short, filled with gaps, and sometimes contain random clicks that don't reflect our true interests. When an algorithm tries to learn from such sparse and noisy data, it often struggles to generalize, meaning it fails to make good predictions when it encounters new or unusual situations. To fix this, researchers have turned to a technique called knowledge distillation. Imagine a master chef teaching an apprentice; the master knows the complex flavors and techniques, and the goal is to transfer that deep understanding to the student so the student can cook well on their own. In machine learning, a large, powerful model acts as the teacher, and a smaller, faster model acts as the student. The student learns by watching the teacher's final answers or internal thoughts, hoping to mimic that expertise.

Despite the promise of this teaching method, a new study suggests that the traditional way of doing it has a blind spot. In standard practice, the teacher and the student work separately. The teacher processes the data and produces a result, and the student tries to copy that result. The problem is that when the data is messy, it is hard to tell where the student is failing. Is the student misunderstanding the raw ingredients? Is it failing to combine them correctly? Or is it simply trying too hard to copy a specific answer that happened to be wrong? Because the teacher and student operate on their own separate tracks, their mistakes get tangled together, making it difficult to provide precise guidance. Researchers at Pohang University of Science and Technology in South Korea have proposed a different approach to untangle these errors. They call their method GOD, which stands for Graft-Oriented Distillation. Instead of keeping the teacher and student entirely separate, this method temporarily stitches parts of the student into the teacher's brain during the training process.

The researchers tested this idea by taking a standard recommendation system and creating hybrid models. They would take a frozen, pre-trained teacher model and swap out one of its core components with the corresponding part from the student. For instance, they might replace the teacher's memory bank of item names with the student's version, while keeping the teacher's logic engine intact. This created a "grafted" model that could process data using the student's memory but the teacher's reasoning. They did the reverse as well, using the teacher's memory with the student's logic engine. By running these hybrid models, the researchers could see exactly how well the student's memory worked when guided by the teacher's logic, and how well the student's logic worked when fed the teacher's data. This provided a much clearer view of where the student was struggling, allowing for targeted corrections that a standard, separate teaching method could not offer.

To make this process work smoothly, the researchers introduced a technique where the teacher and student components could look at each other while processing information. This mutual attention helped stabilize the learning, especially when the student's parts were not yet fully trained. They then used a specialized learning rule that compared the relationships between all these different hybrid views, rather than just matching final scores. This rule ensured that the student learned the underlying structure of the data, not just the surface-level answers. The result was a system that could learn robust patterns even from very short or noisy histories. When tested on three real-world datasets involving online shopping, restaurant reviews, and movie ratings, the new method consistently outperformed existing state-of-the-art techniques. In some cases, it improved the accuracy of recommendations by nearly fourteen percent compared to the best previous methods.

One of the most significant findings was that this approach worked particularly well when the data was sparse. In situations where users had very few interactions, the new method helped the student model generalize much better than before. It also proved to be more robust when the input data was noisy or corrupted, maintaining high performance even when a significant portion of the history was replaced with random items. The researchers found that the method remained effective even when the teacher was not perfectly trained, suggesting that the grafting technique could extract useful knowledge from imperfect sources. Crucially, all these complex hybrid models were only used during the training phase. Once the student was fully trained, the system discarded the teacher and the grafts, using only the compact student model for actual predictions. This means that while the training process was more sophisticated, the final application required no extra computing power or time, making it a practical solution for real-world recommendation systems.

The study also explored how this method held up when the student and teacher were of the same size, a scenario often used to test self-improvement techniques. Even without a large teacher to guide a small student, the grafting method continued to outperform other approaches, suggesting that the benefit came from the way the components were coupled and examined, not just from the size difference between the models. The researchers concluded that by isolating and supervising specific components of the learning process, they could overcome the limitations of traditional distillation. This component-level feedback allowed the system to learn more reliable patterns from imperfect data, offering a clearer path for machines to understand human preferences in a complex and often messy world. The work demonstrates that sometimes, to teach a machine better, you have to let it think with your brain for a moment, before it learns to think on its own.

Technical Summary: GOD: Enhancing Generalization via Deep Grafting for Sequential Recommendation

Problem Statement

Sequential Recommendation (SR) aims to predict a user's next interaction by modeling patterns in their historical interaction sequences. While deep sequential models, particularly Transformer-based architectures, have achieved strong performance, they struggle with real-world data characterized by sparsity and noise. In such environments, models often overfit to observed co-occurrences or spurious transitions rather than learning robust patterns that generalize to unseen interactions.

Existing Knowledge Distillation (KD) methods attempt to mitigate data sparsity by transferring dense supervision from a large, pretrained "teacher" model to a compact "student" model. However, conventional KD approaches typically run the teacher and student through independent inference paths. The student is then trained to match the teacher's outputs or representations. The authors argue that this separation entangles the effects of the student's components (e.g., embedding tables and sequence encoders). Consequently, it becomes difficult to determine whether poor generalization stems from unreliable embeddings, overfitted encoders, or the co-adaptation of both to sparse histories. Furthermore, standard KD provides only output-level or intermediate-representation-level feedback, failing to offer component-specific guidance necessary for robust generalization.

Methodology: Graft-Oriented Distillation (GOD)

To address these limitations, the paper proposes GOD, a component-level KD framework that utilizes grafting. Grafting involves replacing selected frozen components of the teacher model with trainable student counterparts to create "hybrid source models." This allows the system to evaluate specific student components under the teacher's computation path, providing granular feedback.

1. Hybrid Source Models

GOD constructs three distinct source models to facilitate component-level distillation:

Non-Grafted Teacher: The original frozen teacher model, serving as a stable knowledge anchor.
Embed-Grafted Teacher: The teacher's embedding table is replaced with the student's trainable embedding table, while the teacher's sequence encoder remains frozen. This evaluates the student's embeddings under the teacher's encoding logic.
Encoder-Grafted Teacher: The teacher's sequence encoder is replaced with the student's trainable encoder, while the teacher's embedding table remains frozen. This evaluates the student's encoder under the teacher's embedding representations.

The Target Model is the Non-Grafted Student, which receives distilled representations from all three source models.

2. Grafted Encoding (GE)

To stabilize representation generation in hybrid models, particularly during early training when student components are not yet optimized, GOD introduces Grafted Encoding.

Mechanism: For Transformer-based models, GE concatenates teacher-side and student-side embeddings and processes them through a shared sequence encoder.
Mutual Attention: Unlike standard causal attention, GE enables mutual attention between teacher and student tokens within the concatenated sequence. This allows stable teacher-side tokens to provide context for student-side tokens early in training, while improved student embeddings later refine the representations without updating the frozen teacher.
Efficiency: Although GE increases sequence length during training, it is not used during inference, incurring no additional inference cost.

3. Graft-aware Contrastive Learning (GCL)

GOD employs contrastive learning to distill relational structures among the four representations generated by the source and target models:

Pairwise Relations: The framework considers all pairwise relations among the representations: $(T,T)$ , $(S,T)$ , $(T,S)$ , and $(S,S)$ .
Adaptive Weighting: Not all pairs are equally informative. GCL dynamically weights pairs based on their similarity. Highly similar relations are down-weighted to reduce redundant supervision, while less-aligned (complementary) pairs are emphasized.
Stop-Gradient: The weighting mechanism uses a stop-gradient operation to ensure weights act as adaptive coefficients rather than direct optimization targets.

4. Inference and Optimization

During inference, GOD uses only the Non-Grafted Student, ensuring no additional computational overhead compared to standard models. The training objective jointly optimizes the recommendation loss (predicting the next item) and the distillation loss (GCL) using a coefficient $\lambda$ .

Key Contributions

Component-Level Perspective: The paper re-examines KD for SR, highlighting how separated teacher-student paths can weaken generalization under sparse histories and proposing a shift toward component-level feedback.
GOD Framework: The introduction of a KD framework that constructs hybrid source models via grafting. This allows for the evaluation of student embeddings and encoders within teacher-conditioned computation paths.
Empirical Validation: Extensive experiments on three real-world datasets (Amazon Beauty, Yelp, MovieLens 1M) demonstrate that GOD consistently outperforms state-of-the-art KD and self-supervised baselines.

Experimental Results

The authors evaluated GOD across three datasets using three backbone architectures (GRU4Rec, FMLPRec, and SASRec).

Performance: GOD achieved the best results across all datasets and metrics. It outperformed the strongest baseline by up to 13.92% (specifically in NDCG@20 on the Yelp dataset with SASRec).
Generalization: GOD showed particularly large improvements on sparse datasets (Amazon Beauty, Yelp) compared to denser ones (MovieLens 1M), validating its effectiveness in handling sparse interactions.
Robustness:
- Sparsity: GOD maintained superior performance even for users with very short interaction histories.
- Noise: The model demonstrated robustness against injected noise in input sequences.
- Teacher Quality: GOD performed well even when the teacher was only partially trained, outperforming bidirectional KD methods that rely on teacher updates.
Efficiency: Despite the added complexity of hybrid models during training, GOD reached its best validation performance faster (in wall-clock time) than strong baselines like BD and EMKD, and incurred lower memory usage than some competitors.
Self-Distillation: Even in a self-distillation setting where the teacher and student have identical capacity (no compression), GOD outperformed both conventional KD and self-supervised contrastive baselines, suggesting the gains stem from the grafting mechanism itself rather than just model compression.

Significance and Claims

The paper claims that GOD addresses a fundamental limitation in sequential recommendation: the entanglement of student components in standard KD. By using grafting to isolate and supervise specific components (embeddings vs. encoders) within a teacher-conditioned path, GOD provides more direct and fine-grained guidance.

The authors emphasize that GOD does not rely on external modalities (like Large Language Models) or massive teacher updates. Instead, it enhances the generalization of distilled knowledge through structural coupling. The framework is presented as a practical solution that improves recommendation performance without prohibitive training overhead and incurs zero additional cost during inference, as it relies solely on the non-grafted student model at deployment.