🤖 AI

Proportional Analogies on Probability Distributions via Bayesian Updating

This paper introduces a novel framework for proportional analogies between probability distributions by defining their relationship through Bayesian updating, a concept validated for exponential family members and extendable to arbitrary distributions via Gaussian mixture approximations.

Original authors: Pierre-Alexandre Murena

Published 2026-08-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pierre-Alexandre Murena

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Science of "A is to B as C is to D"

Imagine you are trying to teach a robot how to think like a human. One of the most powerful tools in our mental toolbox is analogy. It's that "Aha!" moment when you realize that the way a key opens a lock is exactly like the way a password unlocks a computer. In science, we call this a "proportional analogy," written as $A : B :: C : D$ . It means the relationship between A and B is the same as the relationship between C and D.

For a long time, scientists have been great at teaching computers to spot these patterns in simple things, like words in a sentence or pixels in an image. But there's a whole world of data that is much trickier: probability distributions. Think of a distribution not as a single number, but as a cloud of possibilities—a map showing how likely different outcomes are. If you roll a die, the distribution is the shape of all the possible results. If you are predicting the weather, the distribution is the cloud of chances for rain, sun, or snow.

The big question is: How do you make an analogy between two clouds of possibilities? If you have a "sunny" weather map and a "rainy" weather map, how do you find a "cloudy" map that relates to a "stormy" map in the exact same way? This is the puzzle this paper tackles. It tries to build a bridge between the rigid logic of analogies and the wobbly, uncertain world of probability, using a famous mathematical tool called Bayesian updating. In simple terms, Bayesian updating is how we learn from new evidence: we start with a guess (a prior), see some data, and then shift our guess to a new, better version (a posterior). The author asks: Can they use this shifting process to define what it means for two probability clouds to be "analogous"?

The Paper's Discovery: Analogies as "Learning Journeys"

In this paper, the author proposes a fresh way to define proportional analogies for probability distributions. Instead of trying to subtract one cloud from another like numbers on a calculator, they suggest we look at the journey between them. They argue that two distributions are analogous if you can get from one to the other by "learning" from a specific set of observations.

Imagine you have a map of a city (Distribution A). If you take a bus ride and see a new neighborhood, your mental map updates to include it (Distribution B). The paper suggests that if you can find a "bus ride" (a set of observations) that turns Map A into Map B, and that same bus ride turns Map C into Map D, then A is to B as C is to D. The "bus ride" is the key! It's the hidden story that connects the pairs.

The author proves that for a huge family of common distributions (called the exponential family, which includes things like the bell curve and coin flips), this idea works beautifully. They show that if you translate these distributions into a special mathematical language (called "natural parameters"), the analogy becomes as simple as a math equation: the distance between A and B is the same as the distance between C and D. It's like saying, "If you walk 5 steps north from A to get to B, you must walk 5 steps north from C to get to D."

However, the paper is careful to point out that this isn't magic. They explicitly rule out the idea that you can just subtract distributions like normal numbers in all cases. In fact, they show that some previous attempts to do this failed because they didn't respect the unique rules of probability. They also argue against a stricter definition where the "bus ride" must be reversible (going back and forth perfectly), because in the real world, learning often changes things permanently—you can't always un-learn a lesson to get back to your old guess.

To test their idea, the author built a computer program that acts like a detective. Since we often don't know the exact "bus ride" (the observations) that created a distribution, the program has to guess what those observations were. It does this by simulating millions of possible journeys, checking if they turn the starting maps into the target maps. In their experiments, they created 3,121 fake analogy puzzles. Their program successfully solved about 61.5% of them.

The results were promising but not perfect. When the program did solve a puzzle, it was usually very accurate, especially for the average values (the "center" of the cloud). However, it sometimes struggled with the "spread" or variance of the data, often guessing the cloud was tighter than it really was. The author suggests this is because their method relies on sampling—taking random guesses to approximate the answer—and sometimes those guesses get stuck in a corner. They found that when the program used more "particles" (more guesses) to make its estimate, the errors dropped significantly.

So, what's the verdict? The paper doesn't claim to have solved every analogy problem in the universe. Instead, it provides a solid, mathematically proven foundation for how analogies should work for probability clouds, based on the logic of learning from data. It shows that for many standard types of data, the answer is a simple arithmetic rule. For more complex, messy data, it offers a sampling-based tool that works well enough to be useful, though it admits that finding the perfect answer is still a challenge. The author believes this approach could help machines learn better by transferring knowledge from one uncertain situation to another, much like how humans use analogies to understand the unknown.

Technical Summary: Proportional Analogies on Probability Distributions via Bayesian Updating

Problem Statement

Analogical reasoning, defined as the quaternary relation "A is to B as C is to D" ( $a:b::c:d$ ), has been extensively formalized in Boolean, symbolic, and real-valued domains. However, the extension of proportional analogies to probability distributions remains largely unexplored. Existing attempts to define analogies between distributions face significant limitations:

Murena, Cornuéjols, and Dessalles (2018) utilized parallel transport in information geometry. This approach is computationally demanding, restricted to distributions within the same parametric family, and fails to satisfy the fundamental axioms of proportional analogies on non-flat manifolds.
Prade and Richard (2025) extended classical proportional analogies to categorical distributions using arithmetico-geometric constraints. This framework is limited to categorical distributions, imposes overly restrictive conditions that reduce the set of valid analogies, and does not leverage the statistical nature of probability distributions.

The core challenge is to define a notion of proportional analogy for probability distributions that satisfies the classical axioms of analogical reasoning, applies to arbitrary distributions, and is grounded in the fundamental mechanisms of probabilistic inference.

Methodology

Bayesian Updating as the Foundation

The paper introduces a Bayes-based proportional analogy. The central hypothesis is that two distributions are related if one can be transformed into the other through Bayesian updating induced by a suitable set of observations.

Let $\mathcal{P}$ be a space of probability distributions over a parameter space $\Theta$ , and $\mathcal{L}$ be a statistical model with likelihood $p(x|\theta)$ . The author defines a symmetric relation $p \xleftrightarrow{x}_{\mathcal{L}} q$ if $q$ is the posterior of $p$ given observation $x$ , or vice versa.

A Bayes-based proportional analogy $R_{\mathcal{L}}(p_A, p_B, p_C, p_D)$ holds if there exist observations $x_1, x_2 \in \bar{\mathcal{X}}$ (where $\bar{\mathcal{X}}$ includes a null observation $\emptyset$ ) such that:

$p_A \xleftrightarrow{x_1}_{\mathcal{L}} p_B$ and $p_C \xleftrightarrow{x_1}_{\mathcal{L}} p_D$
$p_A \xleftrightarrow{x_2}_{\mathcal{L}} p_C$ and $p_B \xleftrightarrow{x_2}_{\mathcal{L}} p_D$

This definition relies on statistical reachability: the analogy exists if the "distance" (in terms of required updates) between $A$ and $B$ is equivalent to the distance between $C$ and $D$ , mediated by the same likelihood model.

Theoretical Characterization: Exponential Family

The author proves that for distributions belonging to the exponential family, this analogy admits a simple arithmetic characterization in the natural parameter space.

Proposition 4 & Theorem 1: For distributions $p(.|\xi)$ $p (.∣ ξ)$ in an exponential family with natural parameters $\eta(\xi)$ $η (ξ)$ , the analogy $p_A : p_B :: p_C : p_D$ $p_{A} : p_{B} :: p_{C} : p_{D}$ holds if and only if:
1. The natural parameters satisfy the arithmetic proportion: $\eta(\xi_B) - \eta(\xi_A) = \eta(\xi_D) - \eta(\xi_C)$ .
2. OR the "reverse solution" holds: $\eta(\xi_A) = \eta(\xi_D)$ and $\eta(\xi_B) = \eta(\xi_C)$ .
3. Crucial Condition: The difference vectors must belong to the image of the sufficient statistic $S(x)$ of the likelihood model, denoted $\text{Im}(S)$ .

This result implies that the "richness" of possible analogies is determined by the geometry of $\text{Im}(S)$ . If $\text{Im}(S)$ covers the entire parameter space, the analogy is maximal.

Approximate Solver for Arbitrary Distributions

To extend the framework beyond analytically tractable exponential families, the paper proposes a sampling-based algorithm inspired by Approximate Bayesian Computation (ABC).

Input: Finite samples from distributions $p_A, p_B, p_C$ .
Goal: Find a distribution $p_D$ (represented by samples) and latent observations $x, x'$ that satisfy the consistency conditions of the analogy.
Mechanism:
1. Posterior Sampling: Uses importance sampling to approximate posteriors $p(\theta|x)$ from the prior samples.
2. Inverse Updates: Handles the symmetric nature of the relation by allowing updates in reverse (recovering a prior from a posterior) using weighted resampling with weights proportional to $1/p(x|\theta)$ .
3. Optimization: Searches for latent datasets $(x, x')$ that minimize the discrepancy (e.g., Wasserstein distance) between the empirical distributions of the observed samples and the reconstructed samples.
4. Validation: Solutions are accepted only if the Effective Sample Size (ESS) is sufficient and the reconstruction error is below a threshold.

Key Contributions

Novel Definition: Introduces a definition of proportional analogy for probability distributions based on Bayesian updating, which satisfies the fundamental axioms of proportional analogies (reflexivity, symmetry, central permutation).
Arithmetic Characterization: Proves that for the exponential family, the analogy reduces to an arithmetic proportion between natural parameters, provided the parameter differences lie within the image of the likelihood's sufficient statistic.
Generalization: Develops a sampling-based algorithm that solves analogical equations for arbitrary probability distributions, bypassing the need for closed-form analytical solutions.
Theoretical Analysis of Expressiveness: Demonstrates that the expressiveness of the analogy is directly tied to the geometry of the sufficient statistic's image, defining "maximal" analogies for specific likelihood models (e.g., Gaussian linear regression).

Empirical Results

The author evaluated the sampling-based solver on a synthetic benchmark involving multivariate normal distributions where the analytical solution is known.

Setup: 3,121 analogical equations were generated. Two likelihood models were tested: a standard normal likelihood and a linear regression likelihood.
Performance on Normal Likelihood: The algorithm recovered 0% solutions for the normal likelihood. This was expected, as the support of the analogy for this model is of measure zero, confirming the algorithm does not generate false positives.
Performance on Regression Likelihood: Using Bayesian Optimization to search the latent observation space, the solver successfully recovered solutions for 1,918 out of 3,121 equations (61.45%).
Accuracy: For successful runs, the median Wasserstein distance between the analytical and recovered solutions was 0.132. The mean estimates showed a strong Pearson correlation (0.997) with the ground truth, while variance estimates tended to be under-estimated.
Limitations Observed:
- Search Space: Exhaustive grid search was computationally prohibitive for complex likelihoods; Bayesian Optimization was used for efficiency.
- Particle Degeneracy: Approximation errors were strongly correlated with low Effective Sample Size (ESS), indicating that particle degeneracy in the importance sampling step is a primary source of error.
- Non-Maximal Analogies: The algorithm correctly failed to find solutions for analogies based on non-maximal likelihoods (where the support is of measure zero), confirming it does not generate false positives.

Significance and Claims

The paper claims to provide a principled foundation for connecting analogical reasoning with machine learning by grounding analogies in the intrinsic mechanisms of probabilistic inference.

Theoretical Contribution: Unlike previous approaches that rely on geometric manifolds or ad-hoc arithmetic on probabilities, this framework naturally extends to arbitrary distributions and satisfies the axiomatic requirements of analogical reasoning.
Practical Utility: The framework opens new perspectives for applications such as transfer learning, data augmentation, distribution reconstruction, case-based reasoning, and meta-learning, where reasoning directly over probability distributions offers a natural mechanism for knowledge transfer.
Modesty: The author acknowledges that the current implementation is limited by the approximation quality of importance sampling. They explicitly state that the "reverse solution" (where $A=D$ and $B=C$ ) is a consequence of the symmetric definition, which some might find debatable, but is naturally covered by their theoretical framework. Future work is proposed to investigate more advanced inference techniques (e.g., posterior sampling) to improve robustness and scalability.