🔬 materials science

Bounding Retraining Equivalence and the Deletion Floor in Materials Machine Unlearning

This paper addresses the ambiguity of post-deletion accuracy in materials machine learning by defining a "deletion floor" based on retraining equivalence, establishing theoretical bounds that link low deletion floors to retained data redundancy, and demonstrating empirically that request-level unlearning evaluations must jointly report reference loss, prediction change, and retained utility to distinguish between effective unlearning and mere data suppression.

Original authors: Can Polat, Mustafa Kurban, Erchin Serpedin, Hasan Kurban

Published 2026-09-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Can Polat, Mustafa Kurban, Erchin Serpedin, Hasan Kurban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of materials science, researchers use powerful computer models to predict the properties of new substances before they are ever built in a lab. These models learn from vast databases of known chemical structures, mapping the arrangement of atoms to outcomes like how much energy a material stores or how stable it is. Because the laws of physics are consistent, many different chemical formulas can produce nearly identical crystal structures. This creates a natural redundancy in the data: if one specific record of a crystal is removed from the training set, the model can often still predict its properties accurately because it has learned the same physical patterns from other, very similar records that remain. This poses a unique challenge for a field called machine unlearning, which aims to make artificial intelligence systems "forget" specific data points, often to comply with privacy laws. If a system can still predict a removed item perfectly because of its neighbors, has it truly forgotten, or is it just using a different path to the same answer?

A team of researchers set out to solve this ambiguity by defining a new standard for what it means to successfully unlearn a piece of data in materials science. They introduced a concept called the "deletion floor," which represents the lowest possible error a model can have on a removed item if it were simply retrained from scratch without that item. Their work reveals that in many cases, the best possible retraining still predicts the removed item with high accuracy, simply because the remaining data is so similar. They found that the success of an unlearning attempt cannot be judged by accuracy alone; instead, it must be measured by comparing the model's new predictions against this specific baseline of what retraining would naturally produce.

The researchers tested these ideas using a massive collection of over 44,000 crystal structures from a public database known as the Materials Project. They focused on predicting formation energy, a measure of how much energy is required to build a crystal from its elements. To understand how redundancy affects forgetting, they first created a controlled experiment with synthetic data. They built groups of nearly identical items, then removed one member from each group and retrained the model. When they removed an item that had no close relatives, the model's error on that item remained higher than when they removed an item that had similar relatives remaining. In fact, the median error fell by nearly eight times when just one similar record was left behind. This demonstrated that the model was not truly "forgetting" the physical behavior of the removed crystal; it was simply relying on the surviving relatives to fill in the gap.

To see if this held up in real-world scenarios, the team applied their methods to the thousands of real crystal structures. They compared two different ways of training the models: one with a simpler setup and another with a more complex setup that used many more features to describe the crystals. They discovered a surprising trade-off. The more complex model was generally more accurate overall, but it was also much more sensitive to the removal of a single record. When a record was deleted, the complex model's prediction for that specific item shifted much more than the simpler model's prediction did. Yet, paradoxically, the complex model still ended up with a lower error rate on the deleted item after retraining. This means the model was highly sensitive to the change in its training data, yet it still managed to predict the removed item's value almost perfectly because the surrounding data was so informative.

The study also examined various methods used to approximate the unlearning process without actually retraining the model from scratch, a task that is computationally expensive. They found that some methods successfully increased the error on the specific deleted item, but often at the cost of making the model worse at predicting other, unrelated items. Others kept the overall model performance high but failed to significantly change the prediction for the deleted item, essentially leaving the "memory" of that record intact. The researchers concluded that simply checking if a model is less accurate on a deleted item is not enough. A successful unlearning process must be evaluated by looking at three things together: how much the prediction for the deleted item changed, how that change compares to what a full retraining would have produced, and whether the model's ability to predict other materials was preserved.

By establishing this "deletion floor," the researchers provided a clear reference point for the scientific community. They showed that in fields like materials science, where physical laws create strong connections between different data points, a model can appear to forget a record while still predicting it accurately. This is not a failure of the unlearning process, but a reflection of the underlying physics. The paper argues that future evaluations should stop treating post-deletion accuracy as a simple pass or fail metric. Instead, they should report the specific baseline error left by retraining, the actual shift in the model's prediction, and the overall health of the model. This approach ensures that when a system is asked to forget, we know exactly what it has let go of and what it has kept, grounded in the reality of the data rather than a vague notion of privacy.

Technical Summary: Bounding Retraining Equivalence and the Deletion Floor in Materials Machine Unlearning

1. Problem Statement

In materials machine learning, datasets map chemical compositions and crystal structures to thermodynamic targets (e.g., formation energy). A fundamental challenge arises because closely related retained structures (due to ionic substitutions preserving crystal lattices) can sustain accurate property predictions even after a specific record is removed. Consequently, standard post-deletion prediction error metrics become ambiguous; a model may appear to "forget" a record simply because related records in the dataset allow the model to predict the target accurately via retraining.

Current evaluation frameworks often rely on comparing updated models against references trained without the deleted records. However, without establishing what exact retraining itself would predict for a removed target, continued post-deletion accuracy cannot serve as a definitive signal of unlearning performance. Furthermore, simple accuracy-gap measurements often conflict with distribution-based forgetting scores, and membership inference attacks may overestimate privacy guarantees. The core problem is distinguishing between the removal of a specific record's training contribution and the elimination of the broader physical behavior represented by that record.

2. Methodology

The Deletion Floor and Retraining Equivalence

The authors define the deletion floor ( $\Phi$ ) as the expected target loss under a specified retraining procedure at the deleted request. This serves as a baseline reference for what exact retraining inherently predicts for a removed target.

To evaluate updates constrained by retraining equivalence, the paper employs symmetric $(\epsilon, \delta)$ retraining equivalence, requiring the output distribution of an update algorithm ( $P$ ) to be statistically indistinguishable from the retrained distribution ( $Q$ ).

Sharp Bounded-Loss Transfer (Theorem 1): The authors derive a sharp interval bounding the expected target loss of an update relative to the deletion floor. This interval depends on the equivalence parameters $(\epsilon, \delta)$ and the loss bound $B$ . As $(\epsilon, \delta) \to (0,0)$ , the bounds converge to the deletion floor $\Phi$ .
Target-Suppression Thresholds: The paper establishes necessary conditions (Equation 1) for an unlearning objective to increase target error to a specific threshold $\tau$ while maintaining equivalence, quantifying the required "equivalence budget."

Mechanisms of Retained Predictability

The paper investigates why retraining often leaves deleted targets predictable:

Conditional Neighbor Bound: The authors prove that if a retained neighbor fits accurately, predictions vary smoothly, and labels align, the retained example constrains the prediction error on the removed target. This links the deletion floor directly to retained structural support, local label agreement, and prediction regularity.
FloorScore: An empirical diagnostic is constructed as a request-level surrogate for the theoretical bound, estimating the floor based on the nearest retained neighbor's fit and prediction variation.

Exact Ridge Decomposition

Using ridge regression, the authors derive an exact ridge identity that decouples residual model fit from deletion-induced prediction changes.

Retraining Residuals: An exact closed-form expression (Equation 4) calculates the retrained residual and deletion floor using the Sherman-Morrison identity. This reveals that retraining error is governed by the adjusted full-data residual and the deleted point's leverage.
Prediction Change: The total prediction change upon deletion is decomposed into a "penalty rescaling" component and a "removing the row" component (Equation 5). This demonstrates that a highly accurate model fit can still exhibit strong prediction sensitivity upon record removal.

Experimental Protocol

The study utilizes 44,344 crystal structures from the Materials Project (MP) and synthetic datasets with controlled redundancy.

Metrics: The protocol tracks squared target loss relative to the retraining floor, normalized error ratios, and prediction changes (signed residuals) between original, retrained, and updated models.
Comparisons: The study compares approximate updates (e.g., NegGrad+, SCRUB-style, Newton-style, gradient ascent) against exact retraining and an unedited "no-update" control.

3. Key Contributions

Derivation of a Request-Level Reference Interval: The paper formulates a sharp bounded-loss interval that precisely identifies target losses compatible with approximate retraining equivalence, moving beyond scalar agreement to distributional constraints.
Theoretical Mechanism for Retained Predictability: It proves a conditional neighbor bound linking the deletion floor to retained support and derives an exact ridge decomposition separating residual fit from deletion-induced sensitivity.
Robust Evaluation Protocol: The authors propose a protocol requiring the independent reporting of reference loss (deletion floor), prediction change, and retained utility. They demonstrate that these metrics measure distinct properties and that post-deletion accuracy must be interpreted against what retraining itself leaves behind.

4. Results

Impact of Redundancy: In controlled synthetic sweeps, transitioning from a dataset where a deleted record is the sole family member ( $k=1$ ) to one where a retained relative exists ( $k=2$ ) results in an approximately 8× drop in median normalized retraining loss. This confirms that retained relatives can sustain accurate predictions even after exact retraining.
Fitting Regime Dependence: In the Materials Project study, two distinct ridge regimes (2,048 features vs. 8,192 features) showed that the deletion floor varies systematically with model capacity and regularization. The higher-capacity regime achieved lower retraining floors (better accuracy) but exhibited larger prediction changes upon deletion.
Decoupling of Accuracy and Sensitivity: On a shared set of 200 requests, the higher-capacity model achieved a lower retraining floor on 132 requests while exhibiting a larger prediction change on 172 requests. This confirms that higher post-deletion accuracy does not imply lower deletion sensitivity.
Approximate Updates vs. Utility: Systematic comparisons showed that while some updates (e.g., NegGrad+, Gradient Ascent) successfully increased target loss (suppression), they often degraded overall model utility (increased test MSE). Conversely, methods like retain fine-tuning and Newton-style updates remained near the "no-update" control in terms of utility, highlighting that target suppression alone does not signify successful unlearning without considering reference agreement and utility.

5. Significance and Claims

The paper claims that the deletion floor establishes an explicit baseline for evaluating regression unlearning, measuring the residual target error left by exact retraining. The authors argue that:

Ambiguity Resolution: Continued post-deletion accuracy is an ambiguous signal unless interpreted relative to the deletion floor. If retraining itself predicts the target accurately due to retained structural support, an update matching that accuracy is not necessarily "failing" to forget; it is matching the retraining reference.
Evaluation Standardization: Standardizing unlearning evaluation requires reporting retraining predictions, record removal shifts, and retained model utility together. The paper emphasizes that proving formal retraining equivalence requires distributional certification beyond scalar point agreement.
Scope Limitation: The authors modestly state that their findings reflect specific experimental and theoretical scopes (e.g., specific fitting regimes, ridge leverage). They distinguish their work from broader objectives like "broader behavioral removal" or "distributional unlearning," noting that their protocol provides transparent accounting for request-level unlearning but leaves broader knowledge removal as a distinct objective requiring expanded frameworks.

The work concludes that request-level unlearning evaluations must interpret post-deletion accuracy against the "deletion floor" to decouple deliberate target suppression from preserved overall model utility.