📊 statistics

To select or not to select: predictively consistent priors instead of model selection

This paper argues that model selection is often unnecessary or even detrimental to predictive performance in finite data regimes, proposing instead the use of predictively consistent priors that maintain sensible predictive implications as model complexity increases, thereby allowing flexible models to match or outperform selected simpler models.

Original authors: Anna Elisabeth Riha, Leevi Lindgren, David Kohns, Paul-Christian Bürkner, Aki Vehtari

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anna Elisabeth Riha, Leevi Lindgren, David Kohns, Paul-Christian Bürkner, Aki Vehtari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect soup. You have a basic recipe (a simple model) that works okay, but you suspect adding more exotic spices and vegetables (complexity) could make it taste even better. However, you've heard a warning: "If you add too many ingredients, you might ruin the flavor and make it taste weird to anyone who hasn't tried it before" (overfitting).

For a long time, statisticians have followed a rule of thumb: "Keep it simple." They would build a complex soup, taste it, and if it seemed too complicated, they would throw away the extra ingredients and stick to the simpler version. This is called model selection.

This paper, written by Anna Riha and colleagues, asks a bold question: What if the problem isn't the complexity of the soup, but the way we seasoned it in the first place?

The Core Problem: The "Wild" Seasoning

The authors argue that when statisticians build complex models, they often use "default seasoning" (priors) that are too loose. Imagine sprinkling salt without measuring it. If you add just a pinch of salt to a small bowl of soup, it's fine. But if you use that same "pinch" logic for a giant pot with 50 different ingredients, you end up with a soup that is overwhelmingly salty and inedible.

In statistical terms, when you add more variables (ingredients) to a model but keep using independent, unmeasured "priors" (seasoning rules), the model's predictions can go wild. It starts predicting impossible things, like a soup that tastes like pure salt or pure water, just because the math allowed it. To fix this, people used to say, "Okay, let's just remove some ingredients (select a simpler model)."

The Solution: "Predictively Consistent" Priors

The authors propose a new way of thinking. Instead of throwing away ingredients, we should change how we season the pot from the start. They call these "predictively consistent priors."

Think of this as a smart measuring spoon.

Old Way: You have a spoon that holds a fixed amount of salt. If you add 10 ingredients, you add 10 spoonfuls. The soup gets too salty.
New Way (Predictively Consistent): You have a magical spoon that knows how many ingredients are in the pot. If you add 10 ingredients, the spoon automatically adjusts to ensure the total saltiness of the soup stays exactly where you want it.

By using these "smart priors," the model can handle 50 ingredients just as well as it handles 5. The predictions remain stable and sensible, no matter how complex the model gets.

The Experiments: Testing the Soup

The researchers ran several experiments to test this idea, comparing the "Old Way" (selecting simpler models) against the "New Way" (using smart priors on complex models).

Adding Ingredients (Linear & Logistic Regression): They added more and more variables to a model.
- Result: The "Old Way" (simple models) often performed well, but the "New Way" (complex model with smart priors) performed just as well or better. The complex model didn't get "salty" or weird.
Forward Selection (Picking the Best Ingredients): They tried to build a model by adding ingredients one by one, picking the best ones.
- Result: When using the "Old Way," the selection process often got confused and picked bad ingredients because the seasoning was off. When using the "New Way," the complex model with all ingredients was safe to use, and the selection process worked much better.
Non-Linear Shapes (Curvy Recipes): They tried to model complex, curvy patterns (like a rollercoaster) using polynomials.
- Result: The "Old Way" (raw polynomials) exploded into chaos as they added more curves. The "New Way" (using Gaussian Processes, a type of smart prior) kept the curves smooth and accurate, no matter how many they added.

The Big Takeaway

The paper concludes that we often don't need to select simpler models.

If you are worried about a model being too complex and overfitting, the solution isn't to cut it down. The solution is to fix the seasoning (the priors) so that the model knows how to behave as it grows.

If you select a simpler model and it works better: The paper suggests this is a red flag. It usually means your original "seasoning" was bad (too much mass on impossible predictions), and the simpler model just happened to hide that flaw.
The Recommendation: Stop obsessing over "sparseness" (having few ingredients). Instead, focus on "predictive consistency" (making sure your seasoning rules stay sensible as you add more ingredients).

A Final Metaphor

Imagine you are building a tower of blocks.

The Old View: "If the tower gets too tall, it will fall. Let's stop building at 10 blocks."
The Paper's View: "The tower falls because you are using wobbly glue. If you use strong, consistent glue (predictively consistent priors), you can build a 100-block tower that stands just as straight as the 10-block one. You don't need to stop building; you just need better glue."

In short: Don't fear complexity. Fear bad priors. If you get your priors right, you can use the biggest, most complex model available, and it will likely predict the future just as well as, or better than, any simplified version you could pick.

Technical Summary: To Select or not to Select: Predictively Consistent Priors instead of Model Selection

Problem Statement

In Bayesian modelling workflows, practitioners often face a trade-off between model complexity and generalisability. The statistical folklore suggests that selecting simpler models (parsimony) is necessary to avoid overfitting, particularly in finite data regimes. However, model selection itself is a source of instability and can introduce selection-induced bias, potentially favouring models that overfit or underfit depending on the criteria used.

The core problem addressed is determining when model selection is necessary versus when it is unnecessary or even harmful for out-of-sample predictive performance. The authors investigate whether the perceived need to select simpler models stems from the model structure itself or from poor prior specifications that fail to remain sensible as model complexity increases. Specifically, they examine how the choice of priors affects the stability of predictive performance when adding covariates or increasing non-linear complexity.

Methodology

Theoretical Framework

The paper formalises the concept of predictively consistent priors. The authors define a sequence of nested models $\{M_d\}$ with increasing complexity $d$ (e.g., number of predictors or basis functions). They introduce two key properties for priors in this sequence:

Second-Order Predictive Consistency: A prior sequence is predictively consistent if the variance of the induced signal distribution (the systematic component $\eta_d$ ) remains globally bounded and converges to a finite limit as $d \to \infty$ . Formally, $\sup_{d} \text{Var}_{q_d}(\eta_d) \leq C$ and $\lim_{d \to \infty} \text{Var}_{q_d}(\eta_d) = L$ . This ensures that adding model components does not cause the prior predictive distribution to explode or become nonsensical.
Predictive Parsimony: A weaker condition requiring that the Kullback-Leibler (KL) divergence between successive induced signal distributions is summable. The authors demonstrate that predictive parsimony alone is insufficient to prevent variance explosion (e.g., independent normal priors satisfy parsimony but fail consistency).

The authors contrast these with standard independent normal priors (e.g., $\beta_j \sim \mathcal{N}(0, \tau^2)$ ), which they show often violate predictive consistency because the total signal variance grows linearly with the number of predictors ( $d\tau^2$ ), leading to implausible prior predictive distributions.

Experimental Design

The authors evaluate these concepts through four experimental settings using simulated data:

Adding Covariates in Logistic Regression: A dense regime where the true data-generating process (DGP) includes all covariates. They compare independent normal priors against R2D2 priors (which place a prior on the Bayesian $R^2$ ).
Forward Stepwise Selection: A sparse regime where only a subset of covariates are relevant. They compare forward selection using independent normal priors versus R2D2 priors, alongside projection predictive inference (projpred).
Increasing Nonlinear Complexity: A single-predictor setting with increasing polynomial degree or basis functions (raw polynomials, orthogonal polynomials, thin plate splines, and Hilbert Space Approximate Gaussian Processes).
Randomised Controlled Trials (RCTs): A causal inference setting estimating treatment effects while conditioning on high-dimensional covariates. They compare joint priors (R2D2 over all coefficients) against split priors (R2D2 for covariates, independent normal for treatment).

Performance is measured using the Expected Log Predictive Density (elpd) on independent test data, estimated via PSIS-LOO-CV where test data is unavailable.

Key Contributions

Formalisation of Predictively Consistent Priors: The paper provides a rigorous mathematical definition for priors that maintain stable predictive implications as model complexity grows. It distinguishes between "parsimony" (penalising complexity) and "predictive consistency" (controlling the scale of the prior predictive distribution).
Demonstration of Harmful Selection: The authors show that when using predictively consistent priors, selecting a simpler model is often unnecessary. In many experiments, the full complex model with a consistent prior matches or outperforms selected simpler models. Conversely, when model selection appears to "help" (i.e., a simpler model performs better), it often indicates that the prior on the complex model was poorly specified (e.g., placing excessive mass on implausible values), rather than the model structure being inherently flawed.
R2D2 and Function-Space Priors as Solutions: The paper operationalises predictive consistency using:
- R2D2 Priors: By placing a prior on the proportion of variance explained ( $R^2$ ), the induced prior on coefficients automatically scales to prevent variance explosion as predictors are added.
- Function-Space Priors: Gaussian processes and penalised splines are shown to naturally satisfy predictive consistency by bounding the marginal variance of the function.
Causal Inference Implications: In the RCT setting, the paper reveals that a joint R2D2 prior can introduce bias in treatment effect estimation as the number of covariates increases, due to excessive shrinkage of the treatment coefficient. This highlights that while predictive consistency is crucial for prediction, causal inference may require splitting priors to avoid unintended shrinkage on specific parameters of interest.

Results

Linear and Logistic Regression: In experiments adding covariates, models with independent normal priors showed a sharp decline in out-of-sample predictive performance (elpd) as the number of predictors increased, due to the exploding prior variance. In contrast, models with R2D2 priors maintained stable performance up to the true model size.
Forward Selection: When using independent normal priors, forward selection often stopped early to avoid overfitting, but the resulting models still underperformed compared to the full model with an R2D2 prior. The R2D2 prior allowed the use of all covariates without performance degradation.
Nonlinear Modelling: Raw polynomial regression with independent priors exhibited exploding prior predictive variance and deteriorating test performance at higher degrees. Penalised thin plate splines and HSGPs (which are predictively consistent) remained stable.
Causal Inference: In the RCT experiment, the joint R2D2 prior led to severe downward bias in the estimated treatment effect as the number of covariates ( $p$ ) increased, resulting in poor coverage. The "split" prior (R2D2 for covariates, fixed normal for treatment) achieved the best balance, providing variance reduction from covariates without introducing bias.

Significance and Claims

The paper argues that the traditional reliance on model selection to manage complexity is often a workaround for poor prior specification. The authors claim that predictively consistent priors allow modellers to safely expand models to include all available information without the fear of overfitting, provided the priors are specified to keep the prior predictive distribution sensible.

Key takeaways for practitioners include:

Joint Prior Specification: Independence assumptions between model components are not a safe default; they can imply highly informative and unrealistic joint priors.
Prior Predictive Checks: Checking joint quantities (like prior $R^2$ ) is essential to detect when priors place too much mass on nonsensical values.
Re-evaluating Selection: If model selection favours a simpler model, it may be an indication of a poor prior on the complex model rather than a true need for parsimony.
Context Matters: While predictive consistency is vital for prediction, causal inference requires careful consideration of which parameters should be shrunk. A joint prior that is predictively consistent for the outcome may still be biased for specific causal parameters.

The authors conclude that the goal should be to specify priors that remain sensible in predictive space as models become more complex, rather than enforcing sparsity at the component level through selection. This approach supports safer, more iterative model expansion workflows.