ERASE -- A Real-World Aligned Benchmark for Unlearning in Recommender Systems

This paper introduces ERASE, a large-scale benchmark designed to align machine unlearning evaluation with real-world recommender system constraints by covering diverse tasks, datasets, and algorithms to systematically analyze the effectiveness and robustness of current unlearning methods.

The Gist

Imagine you have a very smart, personalized recipe book (a Recommender System) that learns what you like to eat based on everything you've ever cooked or ordered. Over time, it gets really good at suggesting meals just for you.

But what happens if you decide to go vegan, or if you realize you accidentally added a toxic ingredient to your history? You want the book to completely forget those specific ingredients and the meals you made with them, as if they never happened. This is the problem of "Machine Unlearning."

The paper introduces ERASE, a new, massive testing ground designed to see which "eraser" tools work best in the real world. Here's the breakdown using simple analogies:

1. The Problem: The "Big Eraser" vs. The "Tiny Eraser"

Previous tests for machine unlearning were like trying to clean a house by knocking down the whole wall just to remove one picture. They assumed you wanted to delete huge chunks of data (like 5% of all your history) all at once.

ERASE realizes that in real life, you don't delete your whole history at once. You might delete one sensitive photo today, block a spammer tomorrow, and remove a bad review next week.

• The Old Way: "Delete the whole kitchen!"
• The ERASE Way: "Just wipe that one sticky spot on the counter, quickly, without making a mess."

2. The Three Kitchen Scenarios

To make sure their test is realistic, ERASE checks three different types of "cooking" (recommendation tasks):

• Collaborative Filtering (CF): Like a friend saying, "You liked this movie, so you'll probably like that one." (Based on what others like).
• Session-Based (SBR): Like a music playlist that changes based on what you are listening to right now.
• Next-Basket (NBR): Like a grocery store predicting what you'll buy next based on your current cart.

3. The Two "Dirty Spots" (Scenarios)

ERASE tests the erasers in two specific, realistic situations:

• The "Sensitive Ingredient" Scenario: Imagine you are recovering from an addiction and want the system to forget you ever bought alcohol. The test checks: Did the system stop suggesting alcohol to you?
• The "Spam Chef" Scenario: Imagine a troll tries to trick the system by buying thousands of random items to push a specific product to the top. The test checks: Can the system remove the troll's influence and return to normal?

4. The Contestants (The Erasers)

The paper tested 7 different "eraser" algorithms (tools to remove data). Think of them as different cleaning crews:

• The General Cleaners: Tools designed to clean any type of machine learning model, not just recommenders.
• The Specialist Cleaners: Tools built specifically for recommendation systems (like the SCIF method).

5. The Results: Who Cleaned Best?

After running thousands of tests (generating over 600GB of data!), here is what they found:

• The Specialist Wins (Mostly): The SCIF algorithm (a specialist cleaner) was the most reliable. It cleaned the "sticky spots" effectively without ruining the rest of the recipe book. It worked well across different scenarios.
• The General Cleaners Struggle: The "General Cleaners" often got confused. When asked to clean repeatedly (like deleting a photo, then a song, then a book), they started to degrade the quality of the recommendations. They were like a janitor who, in trying to clean one spot, accidentally scrubbed the whole floor too hard and made it slippery.
• The "Speed vs. Quality" Trade-off:
  • Retraining: The "Gold Standard." This is like throwing away the whole recipe book and writing a new one from scratch without the bad ingredients. It's perfect, but it takes days to do.
  • Unlearning: This is the "quick fix." It takes minutes.
  • The Catch: Most current "quick fixes" are either too slow (not much faster than rewriting the book) or they leave the book messy (the system still remembers the bad stuff).

6. The Big Takeaway

The paper concludes that while we have some good tools, we don't have a perfect "magic eraser" yet.

• For small, specific deletions (like removing a sensitive item), the specialist tools work great.
• For repeated deletions (like cleaning up spam over time), the tools often break down or lose their effectiveness.
• The Future: We need tools that are as fast as a "quick wipe" but as thorough as "rewriting the book."

Why Does This Matter?

This isn't just about computers; it's about your rights.

• Privacy: You have the legal right (like GDPR in Europe) to be "forgotten." ERASE helps engineers build systems that actually respect that right without crashing or giving you bad recommendations.
• Safety: If a hacker tries to poison a system, ERASE helps us figure out how to clean the poison out quickly before it hurts anyone.

In short: ERASE is a giant, realistic "stress test" for the tools we use to delete data from AI. It shows us that while we are getting better at it, we still need to invent better "erasers" that are fast, accurate, and don't ruin the whole picture.

Technical Summary

Here is a detailed technical summary of the paper "ERASE – A Real-World Aligned Benchmark for Unlearning in Recommender Systems."

1. Problem Statement

Machine Unlearning (MU) is the process of efficiently removing the influence of specific training data from a trained model to comply with privacy regulations (e.g., GDPR "right-to-be-forgotten"), handle security threats (e.g., spam), or address liability (e.g., copyright infringement).

While MU is an active research area, existing benchmarks for Recommender Systems (RS) suffer from significant gaps when applied to real-world scenarios:

• Limited Task Coverage: Most benchmarks focus solely on Collaborative Filtering (CF), ignoring sequential tasks like Session-Based Recommendation (SBR) and Next-Basket Recommendation (NBR), which are critical in e-commerce and streaming.
• Unrealistic Deletion Scenarios: Existing benchmarks simulate unlearning by deleting large, static chunks of data (e.g., 5% of the dataset) at once. Real-world systems face continuous, small-scale deletion requests (e.g., a single user withdrawing consent or a few spam accounts) that must be handled with low latency between retraining cycles.
• Lack of Domain-Specific Patterns: Real unlearning often targets specific patterns (e.g., removing interactions with sensitive items like alcohol for recovering addicts, or removing malicious spam interactions), which current benchmarks do not simulate.
• Insufficient Efficiency Focus: Many proposed methods are only marginally faster than full retraining, failing to meet the strict latency requirements (seconds/minutes) needed for production deployment where retraining can take hours or days.

2. Methodology: The ERASE Benchmark

The authors introduce ERASE, a large-scale benchmark designed to align with real-world usage. It is built on the RecBole library and covers three core recommendation tasks, nine public datasets, and nine state-of-the-art models.

A. Scope and Design

• Tasks:
  1. Collaborative Filtering (CF): Matrix factorization and Graph Neural Networks (GNNs).
  2. Session-Based Recommendation (SBR): Recurrent (GRU4Rec, NARM) and Attention-based (SASRec) models.
  3. Next-Basket Recommendation (NBR): Graph-based (DNNTSP) and Attention-based (Sets2Sets) models.
• Unlearning Scenarios:
  1. Sensitive Item Removal: Sequentially removing interactions with specific categories (e.g., alcohol, meat) for specific users.
  2. Spam/Poisonous Data Removal: Removing interactions injected by malicious actors to manipulate recommendations.
• Algorithms Evaluated (7 total):
  • Recommender-Specific: SCIF (Selective and Collaborative Influence Function), GIF (Graph Influence Function), CEU (Certified Edge Unlearning), IDEA.
  • General-Purpose (from NeurIPS'23 Unlearning Competition): Fanchuan, Kookmin, Seif.
  • Baseline: Full Retraining.

B. Evaluation Metrics

ERASE evaluates algorithms across three dimensions:

1. Utility: Standard ranking metrics (nDCG, Recall, Hit Rate) on retained data.
2. Effectiveness:
   • Sensitive Items: RelItems@k measures the reduction in recommending sensitive items compared to a retrained model.
   • Spam: RelEff@k measures the recovery of model performance (nDCG) after removing poisoned data.
3. Efficiency: Latency of the unlearning process compared to full retraining (Speedup factor).

C. Experimental Protocol

• Sequential Unlearning: Instead of one large deletion, the benchmark executes a sequence of small batches (e.g., one user's data at a time for sensitive items; 256 spam users per batch for spam).
• Scale: The authors executed over 13,000 GPU hours, generating 600 GB of artifacts, including 1,069 model checkpoints (original, retrained, and unlearned) to allow researchers to test new algorithms without retraining from scratch.

3. Key Contributions

1. Real-World Alignment: ERASE is the first benchmark to systematically cover SBR and NBR tasks with sequential, small-batch unlearning scenarios that reflect actual industry constraints.
2. Comprehensive Artifact Release: The release of pre-trained and retrained checkpoints allows the community to evaluate new unlearning methods with minimal computational cost, focusing solely on the unlearning step.
3. Systematic Analysis: The benchmark provides a large-scale empirical foundation to compare general-purpose vs. domain-specific unlearning algorithms across diverse architectures.

4. Key Results and Findings

The authors analyzed the experimental logs to answer four guiding questions:

• Effectiveness Varies Widely: No single algorithm is universally best. Approximate unlearning can match retraining in utility, but robustness depends heavily on the dataset and architecture.
• Robustness of Recommender-Specific Methods:
  • SCIF emerged as the most consistent and robust algorithm across CF, SBR, and NBR tasks. It maintains high utility and effectiveness with low variance.
  • General-Purpose Methods (Kookmin, Seif): These often degrade significantly under repeated unlearning requests, particularly for attention-based and recurrent models. They tend to suffer from cumulative noise or parameter reset issues.
  • GNN-Specific Methods (GIF, CEU, IDEA): While effective in some cases, they frequently diverged during sequential unlearning due to numerical instability in Hessian approximations.
• Architecture Influence:
  • Attention-based and recurrent models are highly sensitive to general unlearning methods.
  • SCIF's ability to selectively update parameters (e.g., focusing on item embeddings) makes it superior for sequential tasks compared to methods that reset large parameter subsets.
• Efficiency and Deployability:
  • Most current methods fail the "deployability" test. While some are faster than retraining, they often lack the required 3-orders-of-magnitude speedup (reducing time from days to minutes) while maintaining effectiveness.
  • Only a few combinations (e.g., SCIF on specific CF tasks) fall into the "green zone" of high speedup and high effectiveness.

5. Significance

• Bridging the Gap: ERASE moves the field from theoretical "toy" benchmarks to practical evaluation, highlighting that many promising algorithms fail under the stress of real-world, sequential deletion requests.
• Guidance for Practitioners: The results suggest that for production systems, SCIF is currently the most reliable choice, while general-purpose methods require significant adaptation or hyperparameter tuning to be viable.
• Future Research Directions: The benchmark identifies specific challenges for future work:
  • Stabilizing unlearning for neural architectures (especially attention/GRU).
  • Reducing latency further to meet strict SLA requirements.
  • Developing ensemble methods to reduce variance in unlearning performance.

In conclusion, ERASE provides the necessary infrastructure and empirical evidence to drive the development of practical, privacy-compliant, and secure machine learning systems for recommendation.