🔢 mathematics

Optimal information deletion and Bayes' theorem

This paper revisits Arnold Zellner's seminal work on Bayes' theorem by demonstrating that the optimal rule for deleting information—updating a posterior to an antedata distribution without creating or destroying information—coincides with the leave-data-out posterior derived from Bayes' theorem.

Original authors: Hans Montcho, Håvard Rue

Published 2026-05-20

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Hans Montcho, Håvard Rue

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart detective who has solved a complex case using a massive pile of evidence. This detective has written a final report (the posterior distribution) that perfectly combines everything they know about the suspect (the prior) with all the clues they found (the data).

Now, imagine a situation where you need to "un-solve" part of the case. Perhaps a specific piece of evidence turns out to be a mistake, or a user demands that their personal data be completely erased from the detective's memory. You need the detective to produce a new report that looks exactly like the one they would have written if they had never seen that specific piece of evidence in the first place.

This is the problem of Bayesian Unlearning (or data deletion).

The Old Way vs. The New Insight

Usually, if you want to remove a piece of evidence, the easiest way is to throw away the whole report and start over from scratch, re-reading every single clue without the bad one. But if the detective has analyzed millions of clues, starting over is slow and expensive.

In 1988, a mathematician named Arnold Zellner proved that Bayes' Theorem is the most efficient way to add information. He showed that when you take a guess and add new clues, the standard mathematical formula for updating your guess is the only way to do it without accidentally inventing fake facts or losing real ones.

This new paper, written by Hans Montcho and Håvard Rue, asks the "backward" question: Is Bayes' Theorem also the most efficient way to remove information?

The "Perfect Eraser" Analogy

The authors treat information like a physical substance. They propose a rule for "Information Deletion."

The Goal: Take the detective's final report and surgically remove the influence of a specific clue ( $y_g$ ) to get a report based only on the remaining clues ( $y_{-g}$ ).
The Constraint: The rule must be "optimal." This means it cannot destroy real information (making the detective forget things they should remember) and it cannot create fake information (hallucinating new facts that weren't there).

The authors use a mathematical tool called "variational calculus" (think of it as a way to find the smoothest, most perfect path through a maze) to find the best rule for this deletion.

The Big Discovery

After doing the math, the authors prove a surprising and elegant result:

The "Optimal Information Deletion Rule" is exactly the same as the standard Bayes' Theorem.

In other words, the mathematical operation you use to add a clue is the exact same operation (just reversed) that you use to remove a clue.

Adding Info: You take your current belief and multiply it by the new clue.
Removing Info: You take your current belief and divide it by the clue you want to forget.

The paper proves that if you try to invent a shortcut to delete data, you will inevitably either lose too much information or accidentally create new, false information. The only way to delete data perfectly, without distortion, is to use the standard Bayes' formula in reverse.

Why This Matters (According to the Paper)

The authors highlight two main takeaways:

Symmetry: Just as the French chemist Lavoisier said "Matter is neither created nor destroyed," the authors suggest that in Bayesian learning, information is neither created nor lost; it is just transformed. Bayes' Theorem is the perfect tool for both learning (adding) and unlearning (removing).
Better Approximations: Since we know the "perfect" way to delete data is to divide by the likelihood of that data, we can now build better "approximate" methods. If we can't do the perfect math (because it's too hard), we can use this new understanding to build shortcuts that are guaranteed to be the best possible approximations, rather than just random guesses.

Summary

Think of the detective's report as a cake.

Learning is adding a new layer of frosting (data) to the cake.
Unlearning is trying to scrape off that specific layer of frosting to see the cake underneath.

This paper proves that the only way to scrape off the frosting without ruining the cake underneath is to use the exact same tool you used to put the frosting on, just in reverse. Any other tool will either leave crumbs behind (lost information) or scrape off part of the cake itself (destroyed information).

The paper concludes that Bayes' Theorem is the "Goldilocks" rule: it is perfectly balanced for both adding and removing information, ensuring that the truth remains intact throughout the process.

Technical Summary: Optimal Information Deletion and Bayes' Theorem

Problem Statement
This work addresses the problem of "information deletion" or "unlearning" within Bayesian statistics. Specifically, it investigates the theoretical rules required to update a posterior distribution, derived from a complete dataset, into an "antedata" distribution when a specific subset of data is removed. This problem is motivated by practical needs in cross-validation (where one seeks a leave-data-out posterior without refitting) and data deletion (where user data must be removed from a deployed statistical model without retraining from scratch). The central question is: What constitutes the optimal rule for removing information such that no information is destroyed or artificially created?

Methodology
The authors adopt a variational calculus framework inspired by Arnold Zellner's 1988 seminal work on optimal information processing. The methodology proceeds as follows:

Formalization of Information Deletion: The process is defined as an Information Deletion Rule (IDR). The inputs are the posterior distribution given the full data $\pi(\theta|y)$ , the marginal data distribution $\pi(y|I)$ , and the likelihood contribution of the data to be removed $\pi(y_g|\theta)$ . The output is the target distribution $q(\theta|y_{-g})$ representing the state after removing $y_g$ .
Information Loss Functional: Using Shannon's entropy, the authors define an information loss functional $\Delta[q(\theta|y_{-g})]$ , representing the difference between output and input information. Crucially, the term corresponding to the removed data $y_g$ is subtracted (indicated by a negative sign in the log-likelihood term) to reflect the removal of its information content.
Variational Optimization: The authors minimize this functional over the space of all possible probability distributions. By defining a class of neighboring functions and utilizing Lagrangian multipliers to enforce the constraint that the solution must be a proper density, they derive the distribution that minimizes information loss.
Equivalence Proof: The resulting optimal distribution is then compared against the standard "leave-data-out" posterior distribution $\pi(\theta|y_{-g})$ derived via Bayes' theorem.

Key Contributions and Results
The paper establishes two primary theoretical results:

The Optimal Information Deletion Rule: The authors prove that the distribution minimizing the information loss functional is:
$q^*(\theta|y_{-g}) = \frac{\pi(\theta|y) / \pi(y_g|\theta)}{\int_{\Theta} \pi(\theta|y) / \pi(y_g|\theta) d\theta}$
This rule is presented as the unique solution that does not destroy or create nonexistent information during the deletion process.
Equivalence to Bayes' Theorem: The paper proves that the optimal information deletion rule $q^*(\theta|y_{-g})$ is mathematically identical to the posterior distribution obtained by applying Bayes' theorem to the remaining data $y_{-g}$ (treating the full-data posterior as a prior and "dividing out" the likelihood of the removed data).
$\pi(\theta|y_{-g}) = q^*(\theta|y_{-g})$

Significance and Claims
The authors claim that this work provides a variational formulation for Bayesian unlearning, mirroring Zellner's earlier formulation for learning. The significance of these results is framed around the following points:

Duality of Learning and Unlearning: The results demonstrate that Bayes' theorem is not only the optimal rule for adding information (learning) but also the optimal rule for removing information (unlearning). The authors paraphrase Lavoisier to state that in Bayesian learning or unlearning, "information is not created, information is not lost, information is just transformed."
Justification for Approximate Methods: The variational formulation provides a formal justification for existing approximate methods in Bayesian unlearning, such as "variational Bayesian unlearning" and "posterior correction." By restricting the optimization space to a specific family of distributions $\mathcal{Q}$ , one can derive approximate solutions to the leave-data-out problem. The authors note that Equation (8) in their paper interprets this approximation as minimizing the information content of the removed data while regularizing the solution to prevent it from deviating too far from the original full-data posterior.
Optimality: The paper emphasizes that optimizing the information loss does not lead to an ad-hoc trick or a mere approximation; rather, the exact solution to the optimization problem is the standard Bayesian update.

The authors conclude by suggesting potential extensions, such as investigating these results under different information measures (beyond Shannon entropy) or within the framework of generalized variational inference where the likelihood and KL divergence are replaced by general loss functions and divergences. However, they do not propose specific experimental validations or new applications beyond the theoretical framework established.