Not Every Subject Should Stay: Machine Unlearning for Noisy Engagement Recognition

Problem Statement

The paper addresses a practical challenge in engagement recognition, a task involving the classification of user engagement levels from video data. Engagement datasets (e.g., DAiSEE, EngageNet) are typically subject-indexed and contain noisy, subjective supervision. Unlike standard noisy-label problems where individual clips may be mislabeled, the problematic unit in these datasets is often the entire subject. Multiple clips from the same individual may share annotation ambiguities, recording artifacts, or atypical behavioral patterns that introduce counterproductive supervision.

Once a baseline model is trained, the conceptually correct solution to remove a problematic subject is to retrain the model from scratch on the remaining data. However, this full retraining is computationally expensive, especially when multiple candidate subsets of subjects need to be audited. The paper poses the question: Can the influence of an entire problematic subject be removed from an already trained model without full retraining?

Methodology

The authors propose a subject-level machine unlearning framework as a post-hoc sanitization mechanism. The protocol consists of four main stages:

1. Baseline Training: A baseline model ($f_{\theta_0}$) is trained on the full set of training subjects ($S_{tr}$) using a fixed architecture, TCCT-Net (Tensor-Convolution and Convolution-Transformer Network).
2. Harmful Subject Identification: Instead of assuming ground-truth label corruption, the authors treat harmfulness as an operational ranking problem. They compute a score ($r_s$) for each subject based on the mean per-clip cross-entropy loss of that subject's data under the trained baseline. The top-$K$ ranked subjects form the forget-set ($F$).
3. Approximate Unlearning Update: A lightweight correction is applied to the baseline model to produce an unlearned model ($\tilde{f}_{\theta}$).
   • Architecture Restriction: To keep the update efficient, the lower-level feature extractor of TCCT-Net is frozen. Only the final fusion and classification layers (the "head") are updated.
   • Objective Function: The update minimizes a composite loss ($L_{unl}$):
     • $L_{ret}$: Cross-entropy on retained data ($D_R$) to preserve task utility.
     • $L_{cons}$: A consistency term (KL divergence) between the baseline and unlearned model predictions on retained data to prevent unnecessary drift.
     • $L_{forg}$: A negative cross-entropy term on forgotten data ($D_F$) to push the model away from fitting these subjects (anti-fitting).
     • Regularization: A term penalizing deviation from the original head parameters ($\phi_0$).
4. Oracle-Centered Evaluation: The unlearned model is compared against an Oracle model ($f^*_{\theta_F}$), which is retrained from scratch on the retained subjects ($D_R$) only. This oracle serves as the clean reference for the "retained-data solution."

Key Contributions

1. Problem Formulation: The paper defines subject-level post-hoc sanitization as a distinct problem setting for engagement recognition, moving beyond sample-level noisy-label correction to address subject-indexed data revision.
2. Evaluation Protocol: It introduces an oracle-centered evaluation that measures whether an unlearning update approximates the solution of a model retrained from scratch on retained data, rather than merely perturbing the model or degrading performance on forgotten data.
3. Empirical Study: A controlled study on DAiSEE and EngageNet demonstrates when subject-level forgetting is beneficial, how it depends on the construction of the forget-set, and the utility-efficiency trade-offs compared to full retraining.

Results

The study evaluates three model states (Baseline, Unlearned, Oracle) under small forget-set scenarios ($K \in \{1, 3, 5\}$).

• Performance Recovery:
  • On EngageNet ($K=3$), the unlearned model recovered 89.3% of the performance gain achieved by the oracle model (relative to the baseline).
  • On DAiSEE ($K=3$), the unlearned model recovered 92.5% of the oracle gain.
• Efficiency: The unlearning update required approximately one-quarter of the computational cost of full oracle retraining (relative wall-clock cost of ~0.22×–0.26× vs. ~0.97×–0.98× for retraining).
• Comparison to Naive Methods: A "naive removal + short finetuning" approach significantly underperformed the proposed unlearning method, recovering only 36.7% (EngageNet) and 23.3% (DAiSEE) of the oracle gain.
• Sensitivity to Forget-Set Size:
  • Performance generally improved from $K=1$ to $K=3$, suggesting that harmfulness is distributed across a small subset of subjects rather than concentrated in a single outlier.
  • At $K=5$, gains began to saturate or decline, indicating that removing too many subjects (beyond the "harmful core") may discard useful training variability.
  • The Oracle-Gap Recovery (OGR) was highest at the intermediate $K=3$ setting, suggesting that approximate unlearning is most effective when the removal regime targets a specific, small audited subset.

Significance and Claims

The paper maintains a modest and practical scope. It does not claim to provide exact or certified forgetting, nor does it propose a new engagement backbone. Instead, its significance lies in demonstrating that:

1. Post-hoc correction is viable: Approximate subject-level unlearning can serve as a low-cost correction mechanism for noisy engagement data, recovering a substantial portion of the benefits of full retraining.
2. Oracle reference is critical: Evaluating unlearning against an oracle retrained on retained data is necessary to distinguish between useful sanitization and mere model perturbation.
3. Regime dependence: The effectiveness of the method is highly dependent on the quality of subject selection and the size of the forget-set. It works best as a targeted correction for a small, identified subset of harmful subjects rather than a monotone pruning strategy.

The authors conclude that while the method is not a universal solution for all datasets or architectures, it provides a plausible framework for revising trained engagement models when supervision is noisy and organized by subjects.