📊 statistics

MIRA: A Score for Conditional Distribution Accuracy and Model Comparison

This paper introduces MIRA, a sample-based score that assesses the accuracy of candidate conditional distributions and enables Bayesian model comparison by quantifying alignment with the true data-generating process without requiring evidence computation.

Original authors: Sammy Sharief, Justine Zeghal, Gabriel Missael Barco, Pablo Lemos, Yashar Hezaveh, Laurence Perreault-Levasseur

Published 2026-05-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Sammy Sharief, Justine Zeghal, Gabriel Missael Barco, Pablo Lemos, Yashar Hezaveh, Laurence Perreault-Levasseur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to teach a robot how to bake the perfect chocolate cake. You have a "Gold Standard" recipe (the true data) and a "Candidate" recipe (the model you are testing).

In the world of machine learning, we often ask: "Does the robot's cake taste like the real thing?" Usually, we try to answer this by baking thousands of cakes from both recipes and comparing them side-by-side. But what if you only have one real cake to compare against, while the robot bakes thousands of copies of its own version? And what if the "cake" isn't food, but complex data like images of galaxies or medical scans?

This is the problem the paper MIRA (Mass In Random Areas) solves. It introduces a new way to grade a robot's "conditional distribution" (its ability to predict an outcome based on a specific input) using just a single real-world example.

Here is how MIRA works, explained through simple analogies:

1. The Problem: The "One Real Cake" Dilemma

In many scientific fields (like astronomy or medicine), we can simulate thousands of fake scenarios, but we only have one real observation from nature.

The Old Way: Traditional methods try to count how many fake cakes fall into a specific "tasting zone." But if you only have one real cake, you can't really count how many "real" cakes are in that zone. It's like trying to judge a lottery by looking at only one winning ticket.
The Limitation: Existing tools often require massive amounts of real data to work, or they get confused when the data is high-dimensional (like a 3D MRI scan instead of a simple number).

2. The MIRA Solution: The "Random Dartboard" Game

MIRA changes the game. Instead of trying to measure the whole cake at once, it plays a game of chance using random dartboards.

Here is the step-by-step process:

The Setup: You have the Real Cake (the true data point, $y^*$ ) and a pile of Robot Cakes (the candidate model's samples, $y$ ).
The Random Dart: You throw a dart randomly onto the table to pick a center point ( $c$ ).
The Target: You look at the Robot's pile and pick one random robot cake ( $y_r$ ). You draw a circle around your dart center ( $c$ ) that just barely touches that robot cake.
The Count:
- How many other robot cakes fell inside this circle? Let's call this number $n$ .
- Did the Real Cake fall inside this same circle? Let's call this $k$ (1 if yes, 0 if no).
The Score: MIRA asks a simple question: "Given that $n$ robot cakes fell in this circle, what is the probability that the Real Cake would also be there?"

3. The Magic Math: Why It Works

The paper proves a beautiful mathematical trick: Because the circle is defined by a random robot cake, the size of the circle is essentially random.

If the Robot is Perfect: The Real Cake and the Robot Cakes are drawn from the same "flavor." The Real Cake has the exact same chance of falling in the circle as the Robot Cakes. The math shows that if the robot is perfect, the average score over many random dart throws will be exactly 2/3 (about 0.67).
If the Robot is Overconfident: The robot thinks the Real Cake is in a tiny, specific spot, but the Real Cake is actually more spread out. The robot's cakes cluster too tightly. MIRA detects this, and the score drops below 2/3.
If the Robot is Underconfident: The robot is too scared and spreads its cakes out too wide, missing the tight cluster where the Real Cake actually lives. MIRA detects this, and the score goes above 2/3.
If the Robot is Biased: The robot is consistently wrong (e.g., always baking a vanilla cake when asked for chocolate). The score drops significantly.

4. Why This is a Big Deal

No "Evidence" Needed: In Bayesian statistics (a fancy way of doing probability), comparing models usually requires calculating something called "evidence," which is like trying to count every single grain of sand on a beach to see which beach is bigger. It's computationally impossible for complex problems. MIRA bypasses this entirely. It just looks at the samples.
High Dimensions: It works even when the "cakes" are 100-dimensional (like a complex galaxy image). Traditional methods often break down in these high-dimensional spaces, but MIRA turns the problem into a simple 1D probability game.
Model Ranking: It doesn't just say "Pass/Fail." It gives you a number. If Model A gets 0.66 and Model B gets 0.55, you know Model A is much closer to the truth.

5. Real-World Tests in the Paper

The authors tested MIRA on several "toy" and real-world problems:

Detecting Confidence: They created fake scenarios where the model was too sure (overconfident) or not sure enough (underconfident). MIRA correctly identified the overconfident ones with low scores and the underconfident ones with high scores.
Image Generation: They tested two AI models trying to generate MNIST digits (handwritten numbers). The "Diffusion Model" got a score near the perfect 0.67, while the "VAE" model got a lower score (0.56), correctly indicating the Diffusion model was better.
Astrophysics: They tested models trying to reconstruct images of galaxies distorted by gravity (gravitational lensing). MIRA correctly identified which physical model of the universe was the most accurate, even when the data was noisy and complex.

Summary

MIRA is a "sample-based score" that acts like a referee. It doesn't need to know the secret recipe (the true mathematical formula) or have a million real examples. It just needs one real example and a bunch of robot guesses. By throwing random "darts" and seeing how often the real example lands in the same spots as the robot's guesses, it gives a single, easy-to-understand number that tells you how good the robot really is.

If the score is close to 0.67, the robot is trustworthy. If it's far away, the robot is either too narrow-minded, too scattered, or just plain wrong.

Technical Summary: MIRA – A Score for Conditional Distribution Accuracy and Model Comparison

Problem Statement

Generative modeling, particularly conditional generative models, has advanced significantly in high-dimensional spaces, finding applications in computer vision, medicine, drug discovery, and scientific inference (e.g., physics and geosciences). However, assessing the accuracy of these models remains a critical challenge. Existing methods for evaluating marginal distributions typically fall into two categories: likelihood-based methods, which require density evaluation, and sample-based methods, which compare generated samples to true data.

For conditional distributions $p(y|x)$ , where only joint samples $(x, y) \sim p(x, y)$ are available, current approaches face significant limitations:

Coverage Probability Tests: Methods like TARP provide sufficient conditions for accuracy but rely on coverage plots that are difficult to interpret quantitatively for model ranking.
Probability Integral Transform (PIT) & Simulation-Based Calibration (SBC): These often struggle with high-dimensional outputs, ignoring joint correlations or requiring extensive computational resources and large numbers of fiducial samples.
Two-Sample Tests (e.g., C2ST, MMD): These typically require multiple independent and identically distributed (i.i.d.) samples from both the candidate and true distributions. In many scientific settings, only a single ground-truth realization $y^*$ is available for each observation $x^*$ , rendering these methods incompatible.
Bayesian Model Comparison: Direct comparison via Bayes factors often involves intractable evidence computations, especially in high-dimensional parameter spaces.

The core problem addressed is the need for a sample-based score that can assess the accuracy of a candidate conditional distribution using only joint samples from the true data-generating process, without requiring multiple ground-truth samples per observation or intractable likelihood computations.

Methodology: The Mira Score

The authors introduce Mira (Mass In Random Areas), a framework based on the principle that two distributions coincide if and only if they assign equal probability mass to all measurable regions.

Core Concept

Mira constructs random regions $R$ in the output space $Y$ defined by a random center $c$ and a radius determined by the distance to a random candidate sample $y_r$ drawn from the candidate distribution $p(y|x, M)$ .
$R = \{y \in Y \mid d(y, c) \leq d(y_r, c)\}$
where $d$ is a distance metric and $c \sim p(c)$ is a distribution over the space.

For a given observation $x^*$ and a single ground-truth sample $y^* \sim p(y|x^*, M^*)$ , the method counts:

$n$ : The number of $N$ candidate samples $\{y_j\}_{j=1}^N \sim p(y|x^*, M)$ falling within $R$ .
$k$ : An indicator variable where $k=1$ if $y^* \in R$ , and $k=0$ otherwise.

Theoretical Derivation

Under the null hypothesis that the candidate distribution matches the true distribution ( $M = M^*$ ), the authors prove that the probability mass $\lambda_n$ assigned to the random region $R$ follows a Uniform distribution on $[0, 1]$ . This property allows for the derivation of the Mira statistic $p(k|n)$ , which represents the probability that the true sample falls in $R$ given $n$ candidate samples are in $R$ .

Using Laplace's rule of succession, the statistic is derived as:
$p(k|n) = \frac{n+1}{N+2} \mathbb{1}(k=1) + \frac{N-n+1}{N+2} \mathbb{1}(k=0)$

The Mira score $\mu_{Mira}(M)$ is defined as the expectation of this statistic over all sources of randomness (regions, observations, and samples).

Theoretical Properties

The paper establishes several key theoretical quantities for a correctly specified model:

Asymptotic Distribution: As $N \to \infty$ , the Mira statistic converges in distribution to a Beta(2, 1) random variable.
Expected Score: The theoretical mean of the score for a perfectly calibrated model is 2/3.
Uncertainty: The theoretical variance converges to 1/18.
Lower Bound: The score is bounded below by 1/2, approached when the candidate and true distributions have disjoint supports.

These properties allow Mira to provide not only a point estimate of accuracy but also theoretical uncertainty estimates, facilitating direct model comparison without evidence computation.

Key Contributions

Proposal of Mira: A novel framework for quantifying the accuracy of conditional distributions using only joint samples from the true model, bypassing the need for multiple ground-truth samples per observation.
Theoretical Foundation: Derivation of closed-form expressions for the Mira statistic and its moments, establishing a theoretical reference value (2/3) and uncertainty bounds for correctly specified models.
Practical Application: Demonstration of Mira's effectiveness across three distinct tasks:
- Quality assessment of conditional generative models.
- Posterior accuracy assessment in Bayesian inference.
- Bayesian model comparison in high-dimensional settings.

Experimental Results

The authors validate Mira through a series of experiments ranging from toy problems to complex physical simulations:

Detection of Calibration Errors: In toy inference tasks, Mira successfully distinguishes between correct, overconfident, underconfident, and biased distributions. Scores below 2/3 indicate overconfidence or bias, while scores above 2/3 indicate underconfidence.
Conditional Generative Models: When comparing a Conditional Variational Autoencoder (CVAE) and a Conditional Diffusion Model on MNIST, Mira correctly ranks the Diffusion Model as superior (score $\approx$ 2/3) compared to the CVAE (score $\approx$ 0.56), a finding corroborated by TARP coverage plots.
Bayesian Model Comparison (Gravitational Lensing):
- Likelihood Mismatch: Mira correctly identifies the true physical model (EPL lens + 3 Sérsic sources) among candidates with incorrect lens types or source counts, providing a clear ranking where TARP curves are ambiguous.
- Prior and Noise Shifts: In a high-dimensional source reconstruction task (64x64x3 images), Mira detects mismatches in prior distributions and noise levels, correctly ranking models even with a limited number of true samples ( $L=16$ ).
Comparison with Existing Metrics:
- Mira outperforms standard two-sample tests (MMD, Wasserstein, C2ST) in detecting distributional shifts that preserve low-order moments (e.g., heavy-tailed perturbations or rotated posteriors).
- Unlike SBC, which struggles with high-dimensional joint correlations, Mira operates natively in the full space.
- Mira provides a scalable alternative to Bayes factors, showing strong agreement in toy problems where evidence is tractable, while remaining applicable where evidence is intractable.

Significance and Claims

The paper claims that Mira offers a scalable, sample-based alternative for assessing conditional distribution accuracy and performing Bayesian model comparison. Its significance lies in:

Bypassing Intractable Computations: It enables model comparison in high-dimensional parameter spaces without computing marginal likelihoods (evidence), which are often intractable.
Handling Single Ground-Truth Constraints: It is uniquely suited for settings where only a single realization of the true data is available per observation, a common constraint in scientific simulations.
Interpretability: The score provides a scalar metric with a known theoretical baseline (2/3) and uncertainty, allowing for direct ranking of models and diagnosis of specific failure modes (over/under-confidence).
Robustness: Sensitivity analyses demonstrate that Mira remains robust across varying dimensionalities, numbers of samples, distance metrics, and center distributions.

The authors position Mira not as a replacement for all existing diagnostics but as a complementary tool that addresses specific gaps in current methodologies, particularly for high-dimensional, sample-limited, and likelihood-free inference scenarios.