Measuring the stability and plasticity of recommender systems

1. Problem Statement

Traditional offline evaluation protocols for recommender systems rely on static datasets, splitting data into training and test sets to measure snapshot performance. However, real-world systems are dynamic; they evolve over time, requiring frequent retraining with recent data. This creates a critical gap in current evaluation methods:

• The Stability-Plasticity Dilemma: Systems face a trade-off between stability (retaining previously learned patterns) and plasticity (adapting quickly to new patterns).
• Limitations of Current Methods: Existing continual learning frameworks (often used in supervised classification) rely on distinct "tasks," a concept that is ambiguous in recommendation systems where data flows continuously without clear task boundaries. Furthermore, most research focuses on "catastrophic forgetting" (stability) while neglecting plasticity.
• The Need: There is a lack of a unified, agnostic framework to measure how different recommendation algorithms behave when retrained on evolving data, specifically regarding their ability to balance retaining old knowledge versus assimilating new patterns.

2. Methodology

The authors propose an algorithm- and metric-agnostic offline evaluation protocol designed to simulate real-world data evolution and measure stability and plasticity.

3.1 Data Dynamics (Artificial Shift)

To control for the timing and nature of change, the authors introduce an artificial shift in the dataset:

1. Temporal Split: The dataset $D$ is split chronologically into two equal intervals, $D_1$ (older) and $D_2$ (newer).
2. Label Manipulation: 50% of the items in $D_2$ are randomly selected and their IDs are changed to new, unique labels.
   • This forces the model to treat these items as entirely new entities in $D_2$.
   • The remaining 50% of items retain their original IDs, preserving some natural interaction patterns.
3. Result: This creates a scenario where the underlying distribution changes, forcing models to adapt (plasticity) while potentially forgetting old patterns (stability).

3.2 Experimental Protocol

Two models are trained to simulate a "legacy" vs. "retrained" scenario:

• Model $M_1$ (Legacy): Trained only on $D_1$.
• Model $M_2$ (Retrained): Trained on both $D_1$ and $D_2$.
• Evaluation: Both models are tested on the held-out test sets from both time periods ($D_{Test}^1$ and $D_{Test}^2$).

3.3 Metrics

The framework defines two core metrics based on the performance scores ($S$) of the models on the test sets:

• Stability: Measures how well the retrained model ($M_2$) retains performance on old data compared to the legacy model ($M_1$).
  $$Stability = 1 - (S_{1,1} - S_{2,1})$$
  Where $S_{1,1}$ is $M_1$ on $D_{Test}^1$ and $S_{2,1}$ is $M_2$ on $D_{Test}^1$.
  • Interpretation: A value near 1 indicates high stability (no performance drop on old data). Values $>1$ imply the retrained model actually improved on old data.
• Plasticity: Measures the ability of the retrained model ($M_2$) to adapt to new data compared to the legacy model ($M_1$).
  $$Plasticity = S_{2,2} - S_{1,2}$$
  Where $S_{2,2}$ is $M_2$ on $D_{Test}^2$ and $S_{1,2}$ is $M_1$ on $D_{Test}^2$.
  • Interpretation: Higher values indicate better adaptation to the new regime.

3. Key Contributions

1. A Novel Framework: An evaluation protocol specifically designed for recommendation systems that does not require predefined "tasks," making it applicable to continuous data streams.
2. Agnostic Design: The methodology is independent of the specific algorithm (e.g., Neural vs. Matrix Factorization), the dataset, or the evaluation metric (accuracy, diversity, etc.).
3. Empirical Insights: Preliminary evidence of a trade-off between stability and plasticity across different algorithmic paradigms.

4. Experimental Results

The authors evaluated three representative algorithms from the RecBole library on the GoodReads dataset (book reviews):

• UKNN: User-based K-Nearest Neighbors (Neighborhood-based).
• BPRMF: Bayesian Personalized Ranking Matrix Factorization (Latent Factor).
• NeuMF: Neural Matrix Factorization (Deep Learning).

Key Findings:

• UKNN (Neighborhood-based):
  • High Stability (1.038): It retained or even improved performance on old data after retraining.
  • Low Plasticity (0.180): It struggled significantly to adapt to the new items in $D_2$.
  • Analysis: Neighborhood methods act as "memorization" of interactions; adding new data does not overwrite old patterns but fails to integrate new ones effectively.
• BPRMF (Matrix Factorization):
  • Lower Stability (0.989): Suffered a slight performance drop on old data.
  • Highest Plasticity (0.283): Showed the best ability to adapt to new patterns.
• NeuMF (Neural):
  • Balanced Profile: Stability (1.008) and Plasticity (0.276) were between UKNN and BPRMF, showing strong retention and good adaptability.

Conclusion on Trade-offs: The results suggest a distinct trade-off. Neighborhood-based models (UKNN) are rigid and stable but inflexible. Model-based approaches (BPRMF, NeuMF) are more sensitive to new patterns (higher plasticity) but risk slight degradation on historical data (lower stability).

5. Significance and Future Work

• Practical Impact: This framework allows system architects to select algorithms based on the domain's dynamics. For example, highly dynamic domains (news, social media) might favor high-plasticity models, while static domains (books, music) might benefit from high-stability models.
• Regulatory Compliance: Understanding these profiles aids in risk assessment for regulations like the EU's Digital Services Act, predicting how systems react to data shifts.
• Limitations & Future Work:
  • The 50% artificial shift might underestimate the impact of real-world drift.
  • The current protocol has a potential "temporal leakage" bias where $M_2$ has future knowledge relative to $D_1$.
  • Future work includes expanding to more datasets, testing metrics beyond accuracy (diversity, fairness), and releasing an open-source software package for reproducibility.

In summary, this paper moves beyond static accuracy metrics to provide a dynamic lens for evaluating recommender systems, highlighting that the "best" algorithm depends on the specific balance of stability and plasticity required by the application context.