📊 statistics

Reversible Jump MCMC With No Regrets: Bayesian Variable Selection Using Mixtures of Mutually Singular Distributions

This paper introduces Mixtures of Mutually Singular (MoMS) distributions as a transparent and efficient alternative to Reversible Jump MCMC for Bayesian variable selection, demonstrating that it achieves comparable or superior performance while allowing the use of standard fixed-dimensional sampling algorithms.

Original authors: Don van den Bergh, Merlise A. Clyde, Adrian E. Raftery, Maarten Marsman

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Don van den Bergh, Merlise A. Clyde, Adrian E. Raftery, Maarten Marsman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a list of 10 potential suspects (variables) who might have committed the crime. Your goal is to figure out which ones are actually guilty (included in the model) and which ones are innocent (excluded), and how strong their involvement is.

In the world of statistics, this is called Bayesian Variable Selection. The challenge is that you don't just need to guess the suspects; you need to calculate the probability of every possible combination of suspects (e.g., "Suspect A and B," "Suspect C only," "No one," etc.). With 10 suspects, there are over 1,000 possible combinations. With 20 suspects, there are over a million. Checking them all one by one is like trying to read every book in a library to find one specific sentence—it's too slow.

For decades, statisticians have used a clever, high-tech tool called Reversible Jump MCMC (RJMCMC) to solve this. Think of RJMCMC as a magical teleporter. It allows the detective to jump instantly between different "rooms" (models) where the number of suspects changes. In one room, there are 3 suspects; in the next, there are 5.

The Problem:
The paper's authors say that while this teleporter is powerful, it is notoriously difficult to build and use. The authors quote a piece of statistical folklore: "No one has ever designed and implemented RJMCMC and not regretted it." It's like trying to build a complex Rube Goldberg machine just to turn on a light switch. It works, but it's fragile, confusing, and easy to break.

The Solution: "Mixtures of Mutually Singular Distributions" (MoMS)
The authors propose a new, simpler way to do the same job, which they call MoMS. They argue that you don't need a teleporter at all.

Here is the analogy:
Imagine a large, fixed-size hotel with 10 rooms.

The Old Way (RJMCMC): You have a shapeshifting guest. Sometimes they occupy 3 rooms, sometimes 5. To move them, you have to physically tear down walls and build new ones, or use a teleporter to move them between rooms of different sizes. It's chaotic and hard to manage.
The New Way (MoMS): You keep the hotel exactly the same size (10 rooms) forever. However, you have a strict rule: If a suspect is "innocent" (excluded), they must sit in their room on a chair that is frozen solid (fixed at zero). If they are "guilty" (included), they are free to walk around the room (vary continuously).

In this MoMS hotel, every suspect always has a room. The "frozen" rooms and the "free" rooms are completely separate worlds that never touch (this is what "mutually singular" means). You don't need to change the size of the hotel or tear down walls. You just walk through the same hallway, checking who is frozen and who is moving.

Why is this better?

It's Easier to Build: You don't need the complex "teleporter" math. You can use standard, off-the-shelf tools (standard MCMC) that statisticians already know how to use.
It's Just as Accurate: The paper tested this on a real dataset about diabetes (predicting disease progression using age, BMI, blood pressure, etc.). They compared the new MoMS method against the "gold standard" (checking every single possibility) and the old RJMCMC method. The results were almost identical.
It's Faster: In their tests, the new method was just as good at finding the right answers, but it did it faster because it didn't waste time building complex bridges between different-sized rooms.

Real-World Tests in the Paper
The authors didn't just stop at the diabetes example. They also tested their new method on two other complex scenarios:

Sleep and Memory: They analyzed an experiment where people learned word lists and were tested later. They used the method to figure out which factors (like time of day or sleep duration) actually mattered for memory recall.
Psychological Testing: They looked at a test measuring attitudes toward mental health. They used the method to figure out which questions (items) actually measured which underlying traits (dimensions), effectively cleaning up the test to make it more accurate.

The Bottom Line
The paper claims that for the specific problem of choosing which variables to include in a statistical model, you don't need the complicated, regret-inducing "teleporter" (RJMCMC). Instead, you can use the "frozen room" approach (MoMS). It keeps the math simple, uses standard tools, and gives you the exact same answers without the headache.

As the title says: You can do Bayesian variable selection "without regrets."

1. Problem Statement

Bayesian variable selection involves inferring which subset of predictors (or effects) should be included in a statistical model. This creates a challenge because the parameter space dimension changes depending on the model configuration (i.e., the number of included predictors varies).

The Core Challenge: Standard Markov Chain Monte Carlo (MCMC) algorithms assume a fixed-dimensional parameter space. When the dimension changes, standard Metropolis-Hastings (MH) algorithms cannot be directly applied because the probability densities of the current and proposed states live on spaces of different dimensions.
Current Solutions & Limitations:
- Reversible Jump MCMC (RJMCMC): The standard solution (Green, 1995) augments the state space with auxiliary variables and uses deterministic transformations to match dimensions. While theoretically sound, it is notoriously difficult to design, implement, and debug correctly.
- Continuous Spike-and-Slab Priors: These keep the dimension fixed by allowing coefficients to be non-zero but very small (shrinkage). However, they often lack model-selection consistency (guaranteeing the true model is recovered as $N \to \infty$ ) unless variances are carefully tuned to scale with sample size. They also suffer from high computational costs in high-dimensional settings due to the need to sample the full parameter space.

2. Methodology: Mixtures of Mutually Singular (MoMS) Distributions

The authors propose Mixtures of Mutually Singular (MoMS) distributions as a transparent alternative that allows Bayesian variable selection to be performed within a standard fixed-dimensional MCMC framework.

Conceptual Framework

Mutually Singular Subspaces: Instead of shrinking coefficients toward zero (continuous spike-and-slab), MoMS treats variable selection as moving between distinct, non-overlapping subspaces of a fixed-dimensional space.
- Spike: A subspace where a specific coefficient is fixed exactly at zero ( $\beta_i = 0$ ).
- Slab: A subspace where the coefficient is free to vary continuously ( $\beta_i \in \mathbb{R} \setminus \{0\}$ ).
State Space Definition: The joint state space is defined as $\mathcal{X} = \mathbb{R}^p \times \{0, 1\}^p$ $X = R^{p} \times {0, 1}^{p}$ , containing $p$ $p$ continuous coefficients and $p$ $p$ binary inclusion indicators.
- Each model configuration $\gamma$ corresponds to a disjoint region $S_\gamma$ within this space.
- Because the regions are disjoint (mutually singular), the posterior distribution is supported on a union of these distinct slices.
Algorithm Implementation:
- The authors implement a Metropolis-within-Gibbs sampler.
- Between-Model Moves: The algorithm proposes flipping a single inclusion indicator $\gamma_i$ $γ_{i}$ .
  - If $\gamma_i$ changes from 0 to 1 (Add), $\beta_i$ is proposed from a continuous distribution (e.g., Random Walk Metropolis).
  - If $\gamma_i$ changes from 1 to 0 (Delete), $\beta_i$ is set exactly to 0.
- Acceptance Probability: Since the forward and reverse moves are defined on the same fixed-dimensional space, the acceptance probability follows the standard Metropolis-Hastings ratio without the need for Jacobian corrections or auxiliary variables (unlike RJMCMC).

Theoretical Equivalence

The paper demonstrates that under specific constructions (e.g., using identity transformations and specific proposal densities), the Metropolis-Hastings acceptance probability of MoMS is mathematically identical to that of RJMCMC. This proves that MoMS is not just an approximation but a reformulation that achieves the same theoretical target with simpler mechanics.

3. Key Contributions

Simplification of Transdimensional Sampling: The paper provides a "regret-free" alternative to RJMCMC. It eliminates the need for complex dimension-matching transformations and auxiliary variables, making the algorithm easier to design, reason about, and implement correctly.
Exact Spike-and-Slab Interpretation: Unlike continuous spike-and-slab priors which rely on shrinkage, MoMS enforces exact zeros, preserving the discrete nature of model selection while maintaining a fixed-dimensional parameter space.
Computational Efficiency: By avoiding the overhead of calculating Jacobians and managing auxiliary variables, MoMS iterations are computationally cheaper.
Generalizability: The framework is shown to be applicable not just to linear regression, but to complex hierarchical models like mixed-effects logistic regression and multidimensional item response theory (IRT) models.

4. Results

The authors evaluated the method on three datasets:

Benchmark: Diabetes Dataset (Linear Regression):
- Setup: 10 predictors ( $2^{10} = 1024$ models), allowing for full enumeration as a ground truth.
- Findings: Both RJMCMC and MoMS recovered posterior inclusion probabilities that matched the exact enumeration results to the second decimal place.
- Efficiency: While RJMCMC had slightly higher Effective Sample Size (ESS) per iteration due to sophisticated proposal mechanisms, MoMS achieved significantly higher ESS per second (wall-clock time). This is because MoMS iterations are computationally cheaper, allowing for more samples in the same time frame.
Case Study 1: Mixed-Effects Logistic Regression (Sleep & Memory):
- Applied to a study on false memory (DRM paradigm) with 488 participants.
- Successfully performed variable selection on fixed effects and interaction terms in a mixed-effects model.
- Results identified significant interactions between sleep delay and time of day, with convergence diagnostics ( $\hat{R} < 1.005$ ) indicating robust sampling.
Case Study 2: Multidimensional Generalized Partial Credit Model (MG-PCM):
- Applied to item-factor selection in a 15-item psychometric scale (healthcare worker stigma).
- Used MoMS to select factor loadings, effectively pruning the model structure.
- Demonstrated that model averaging with MoMS leads to shrinkage of irrelevant loadings, while the "Median Probability Model" provided a sparse, interpretable factor structure.

5. Significance and Conclusion

The paper fundamentally shifts the perspective on Bayesian variable selection. It argues that transdimensional problems do not necessarily require transdimensional algorithms.

Practical Impact: Researchers can now perform rigorous Bayesian variable selection using standard, fixed-dimensional MCMC software (like Stan, JAGS, or custom Gibbs samplers) without the "regret" associated with the complexity of RJMCMC.
Theoretical Insight: It clarifies that the difficulty of RJMCMC often lies in the implementation of the transformation rather than the theoretical necessity of it. By explicitly partitioning the parameter space into mutually singular subspaces, the need for dimension-matching disappears.
Limitations: The authors note that MoMS is most natural for nested variable selection problems (where models differ only by the inclusion/exclusion of parameters). For non-nested models where the parameterization itself changes fundamentally (e.g., changing the functional form of a model), RJMCMC may still be the more natural choice.

In summary, the paper establishes that Bayesian variable selection can be carried out within standard fixed-dimensional MCMC methodology—without regret.