🔢 mathematics

SMI: Statistical Membership Inference for Reliable Unlearned Model Auditing

This contribution introduces Statistical Membership Inference (SMI), a training-free auditing framework that overcomes the fundamental limitations and computational overhead of traditional Membership Inference Attacks by reformulating the verification of model forgetting as a problem of estimating non-member mixture proportions, thereby enabling more reliable and efficient performance evaluation with theoretical guarantees.

Original authors: Jialong Sun, Zeming Wei, Jiaxuan Zou, Jiacheng Gong, Jie Fu, Chengyang Dong, Heng Xu, Jialong Li, Bo Liu

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jialong Sun, Zeming Wei, Jiaxuan Zou, Jiacheng Gong, Jie Fu, Chengyang Dong, Heng Xu, Jialong Li, Bo Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The Trap of "Fake Amnesia"

Imagine you have a huge, super-smart library (a machine learning model) that has learned everything from a massive collection of books. One day, a user says: "I want my book removed from your memory. I have the 'right to be forgotten'."

The librarian tries to delete the influence of that book. But how can you know whether the librarian has actually forgotten the book or merely hidden it very well?

Currently, auditors use a method called Membership Inference Attacks (MIA). Imagine this as a detective trying to guess: "Did this specific book originally exist in the library collection?"

The old logic: If the detective cannot guess that the book was in the collection, the auditor assumes: "Great! The library has successfully forgotten it."
The paper's discovery: The authors say this logic is flawed. Just because the detective cannot find the book does not mean it is gone. It could simply mean the book is hidden in a strange, unfamiliar corner of the library that the detective is not looking at. The library may have "forgotten" the book in a clumsy way that still leaves a strange, invisible fingerprint. The old method is like trying to find a ghost in a room by looking for footprints; if there are no footprints, you assume the ghost is gone, but perhaps the ghost is simply floating silently.

The Solution: SMI (Statistical Membership Inference)

Instead of hiring a detective to hunt for a single ghost, the authors propose SMI, which acts more like a statistician counting the crowd.

The Core Idea: The "Smoothie" Analogy

Imagine the library's memory as a giant smoothie.

Member Data: The original fruits (strawberries, bananas) used to make the smoothie.
Non-Member Data: Water or ice that was never added to the smoothie.
Unlearned Data: The fruits the owner tried to remove.

The old method (MIA) tries to taste a single drop and guess: "Is this a strawberry?" If it cannot taste the strawberry flavor, it assumes the strawberry is gone.

SMI takes a different approach. It looks at the entire smoothie and asks: "How much of this smoothie still consists of the original fruits, and how much is just water?"

If the owner successfully "forgot" the strawberries, the smoothie should look almost exactly like a smoothie made only from water (non-members).
If the owner failed, the smoothie still has a distinct "strawberry flavor" (a statistical mixture of the original fruits).

SMI calculates a number (let's call it $\rho$ ) that tells you exactly what percentage of the "strawberry flavor" remains in the mixture. It does not need to find the specific strawberry; it simply measures the overall taste.

Why is SMI Better?

1. No "Shadow Libraries" Needed (The Cost Saver)
The old methods (MIA) are expensive. To check whether the library has forgotten the book, they had to create dozens of fake "shadow" libraries from scratch just to train a detective. This is like building 50 fake libraries just to test whether a real library has forgotten a book. It takes forever and costs a fortune.

The SMI trick: SMI is training-free. It builds no fake libraries. It simply examines the mathematics of the existing library's features. It is like checking the density of the smoothie with a simple scale instead of building an entire new kitchen.

2. It is Honest About Uncertainty
SMI does not give you just a single number; it gives you a confidence interval.

Analogy: Instead of saying "You are 90% forgotten," it says: "We are 95% confident that you are between 85% and 95% forgotten."
This is done using a technique called Bootstrapping (resampling the data), which is like taking 200 different scoops of the smoothie to ensure your taste test is consistent.

How It Works (Simplified "Math Magic")

The authors realized that when a model "forgets" something, the data does not simply disappear; it moves to another location in mathematical space.

Old View: "Is this data point a member or not?" (Yes/No question).
New View (SMI): "How much of this data point is a member and how much is a non-member?" (A mixture question).

They treat the "forgotten" data as a mixture of "original data" and "new data." They use statistical tools (such as comparing averages and variances of the data) to determine the exact ratio. If the ratio of "original data" drops close to zero, the model has truly forgotten.

The Results: What Did They Find?

The paper conducted many experiments (like testing the smoothie with different fruits and different blenders) and found:

SMI is more accurate: It correctly identified how much data was forgotten, far better than the old detective methods.
SMI is faster: It did not need to train those expensive "shadow" libraries. It was much cheaper and quicker to execute.
SMI is stable: Even with small data amounts, it delivered reliable results with clear confidence intervals.

Summary

The paper argues that the current way of auditing whether AI has "forgotten" data is broken because it relies on a flawed assumption: that if you cannot find the data, it is gone.

The authors propose SMI, a new method that stops "hunting" the data and instead measures the statistical mixture of the data. It is like switching from a detective looking for footprints to a chemist analyzing the ingredients of a soup. It is faster, cheaper, and tells you exactly how much of the "secret ingredient" remains in the pot.

Technical Summary: Statistical Membership Inference (SMI) for Reliable Auditing of Forgotten Models

1. Problem Statement

Machine Unlearning (MU) is a crucial mechanism for enforcing the "right to be forgotten," enabling the removal of specific training data from a model without retraining it from scratch. A central challenge in this field is auditing: the reliable verification of whether a model has truly forgotten specified data.

Current auditing standards rely predominantly on Membership Inference Attacks (MIAs). The prevailing assumption holds that if an MIA fails to identify a sample as a member, the model has successfully forgotten it. The authors argue that this assumption is fundamentally flawed for two main reasons:

Alignment Bias: Forgotten samples ( $F_u$ ) occupy a feature space that differs from both original member samples ( $F_m$ ) and true non-member samples ( $F_n$ ). While $F_m$ contributes positive gradients, $F_n$ contributes no gradients, and $F_u$ induces negative gradients. Conventional MIAs assume that $F_u$ aligns with $F_n$ ; however, a failed inference attempt often results from this misalignment rather than genuine forgetting, leading to "illusory forgetting."
Computational Costs: State-of-the-art MIA auditing methods (e.g., LiRA, RMIA) require training numerous "shadow models" to establish baselines, incurring prohibitive computational costs comparable to retraining the target model itself.

2. Methodology: Statistical Membership Inference (SMI)

To address alignment bias and computational inefficiency, the authors propose Statistical Membership Inference (SMI), a training-free auditing framework.

Theoretical Foundation

The paper reformulates the auditing problem using Optimal Transport Theory. Instead of treating forgetting as a binary classification task (member vs. non-member), SMI models the distribution of forgotten features $F_u$ as a convex mixture of the member distribution $F_m$ and the non-member distribution $F_n$ :
$F_u \approx (1 - \rho)F_m + \rho F_n$
Here, $\rho$ represents the mixing proportion (or forgetting rate). The auditing objective shifts from training a classifier to estimating $\rho$ . The authors prove using PAC-Bayes Theory (employing $\chi^2$ -divergence) that MIA-based auditing suffers from an inevitable alignment error controlled by the distributional shift between the attacker's training data and the actual forgotten data.

Algorithmic Variants

SMI estimates $\rho$ by matching statistical moments of the distributions without training auxiliary models:

SMI (Low-Order Moments):
- Assumes the mixture model holds for the first two moments (mean $\mu$ and covariance $\Sigma$ ).
- Derives a closed-form solution by minimizing the Frobenius norm deviation between the empirical covariance of the forgotten data and the covariance predicted by the mixture model.
- Solves a cubic equation in one variable to find the optimal $\rho$ .
SMI-M (Maximum Mean Discrepancy):
- Addresses the instability of covariance estimation in high-dimensional spaces and under non-Gaussian distributions.
- Embeds feature distributions into a Reproducing Kernel Hilbert Space (RKHS) using kernel mean embeddings.
- Formulates the estimation as a convex quadratic programming problem to minimize the MMD between the forgotten distribution and the convex combination of the member and non-member distributions.

Uncertainty Quantification

Unlike MIA methods that rely on shadow models for confidence intervals, SMI uses bootstrapping on the auditing feature set. By resampling the forgotten data and re-estimating $\rho$ , SMI provides reference ranges (e.g., 5th to 95th percentiles) to quantify the reliability of the auditing result.

3. Key Contributions

Theoretical Insight: The paper proves that a failed membership inference attempt does not imply genuine forgetting due to an inherent "alignment bias" between forgotten samples and non-members. It establishes that auditing should measure mixing proportions of distributions rather than the success of a binary classification.
Training-Free Framework: SMI eliminates the need for shadow models and training MIA classifiers, thereby significantly reducing computational overhead.
Reliability Metrics: The framework provides bootstrapping-based reference ranges, offering a grounded way to assess confidence in auditing decisions.
Empirical Superiority: Comprehensive experiments demonstrate that SMI (particularly SMI-M) consistently outperforms existing MIA-based baselines (including LiRA, RMIA, IAM, and RULI) in estimation accuracy across various datasets (CIFAR-10/100, CINIC-10) and model architectures (ResNet, ViT).

4. Experimental Results

The authors evaluated SMI against seven MIA-based baselines under white-box conditions:

Accuracy: SMI-M achieved the highest estimation accuracy ( $\rho^*$ ) across all datasets, often reaching accuracy above 90% in scenarios with complete forgetting (retraining targets), whereas MIA methods exhibited significant variance and lower accuracy.
Efficiency: SMI variants required only a single inference pass (plus bootstrapping) and no training of shadow models. In contrast, MIA baselines required training 8–64 shadow models, resulting in computational costs orders of magnitude higher than SMI.
Robustness: SMI-M remained stable under variations in kernel choices and bandwidths. The method demonstrated reliable performance with as few as 20 samples per class, although performance improved with larger audit sets.
Practical Scenarios: When applied to approximate forgetting methods (FineTune, Fisher, Scrub), SMI successfully distinguished between different levels of forgetting and identified Scrub as the most effective approximate method among those tested.

5. Significance and Claims

The paper positions SMI as a principled and efficient alternative to MIA-based auditing. Its significance lies in:

Correcting a Fundamental Flaw: It challenges the community's reliance on MIA for verifying forgetting and demonstrates that the heuristic "failed attack = forgotten" is structurally untenable.
Scalability: By eliminating dependence on shadow models, SMI makes reliable auditing feasible for large models where training shadow models is computationally prohibitive.
Theoretical Guarantees: The method is based on PAC-Bayes bounds and Optimal Transport Theory, providing a rigorous mathematical foundation for auditing forgetting performance.

The authors conclude that SMI establishes a new standard for auditing forgotten models, offering both theoretical guarantees and strong empirical performance without the computational burden of previous approaches.