Robust model selection using likelihood as data

1. Problem Statement

Standard model selection methods (e.g., AIC, BIC, and standard Bayesian posteriors) face significant limitations when the true data-generating process (DGP) is misspecified—meaning the true DGP is not contained within the set of candidate models.

• Lack of Uncertainty Quantification: Information criteria (AIC/BIC) provide point estimates of model quality but fail to quantify uncertainty regarding how well a model approximates the truth.
• Instability under Misspecification: Bayesian posteriors and information criteria tend to asymptotically concentrate on the model with the minimal Kullback–Leibler (KL) divergence. However, if multiple models have similar (minimal) KL divergence, these methods often become unstable, arbitrarily selecting one model or drifting toward increasingly complex models as sample size increases, without providing calibrated uncertainty.
• The Goal: The authors aim to develop a framework that robustly selects the simplest model that is "close enough" to the true DGP (within a tolerance $\delta$), while providing rigorous uncertainty quantification for the relative distances of all candidate models from the truth.

2. Methodology: Likelihood as Data (LaD)

The core innovation is treating the negative log-likelihood (NLL) values of individual observations as data points themselves, rather than just summary statistics for model comparison.

A. The LaD Framework

Given $n$ observations and $K$ candidate models, the authors construct an $n \times K$ matrix $Z$, where $Z_{ik} = -\log f_k(X_i)$.

• Interpretation: The expected value of the $k$-th column, $\mu_k = E[Z_{ik}]$, equals the KL divergence between the true DGP ($f_0$) and model $k$ ($f_k$), plus the entropy of the true DGP (an unknown constant).
  $$\mu_k = D(f_0 \| f_k) + H(f_0)$$
• Statistical Modeling: The vector of NLLs for observation $i$, $Z_i = (Z_{i1}, \dots, Z_{iK})^T$, is treated as a random vector. The authors model these vectors as i.i.d. draws from a multivariate normal distribution:
  $$Z_1, \dots, Z_n \mid \mu, \Sigma \sim \text{i.i.d. } N(\mu, \Sigma)$$
  where $\mu$ is the vector of expected NLLs (proxy for KL divergences) and $\Sigma$ is the covariance matrix.
• Bayesian Inference: A Normal-Inverse-Wishart (NIW) prior is placed on $(\mu, \Sigma)$. This allows for the computation of the posterior distribution of $\mu$, which quantifies the uncertainty in the relative KL divergences of the models.
• Handling Parameters: For parametric models with estimated parameters $\hat{\theta}_k$, the method uses the M-estimator (e.g., MLE). To correct for overfitting bias in small samples, a bias correction term $d_k/(2n)$ (based on the dimension of parameters) is added to the NLL values.

B. Robust Model Selection Criterion (SLC)

The goal is not just to find the model with the absolute minimum KL divergence, but the simplest model within a tolerance $\delta$ of the minimum.

1. $\delta$-Optimal Set: A model $k$ is $\delta$-optimal if its expected NLL is within $\delta$ of the minimum:
   $$M_\delta(\mu) = \{ k : \mu_k \le \min_j \mu_j + \delta \}$$
2. Complexity Penalty: Among $\delta$-optimal models, the method seeks the one with minimal complexity $c(k)$.
3. Smooth LaD Criterion (SLC): To avoid instability when multiple models are tied for the best fit, the authors propose a "soft" selection score rather than a hard minimum. The SLC score for model $k$ is:
   $$w_\delta(k \mid Z_{1:n}) = P(c^*_\delta(\mu) = c(k) \mid Z_{1:n}) \times E[r_{nk}(\mu) \mid Z_{1:n}]$$
   • Between-class factor: Probability that model $k$ belongs to the minimal complexity class among $\delta$-optimal models.
   • Within-class factor: A soft-selection score $r_{nk}(\mu) = \exp(-\alpha_n(\mu_k - \mu_{\min, c(k)}))$ that down-weights models with higher KL divergence within the same complexity class.
   • Temperature Parameter: $\alpha_n$ grows with $n$ but slower than $\sqrt{n}$ ($\alpha_n = o(\sqrt{n})$), ensuring the score concentrates on the true target set asymptotically while remaining stable at finite samples.

C. Interpretable Tolerance ($\tau$)

To make the choice of $\delta$ interpretable, the authors introduce a "noise model" (a baseline model capturing no signal). They define a rescaled tolerance:
$$\tau = \frac{\delta}{\mu_{\text{noise}} - \mu_{\min}}$$
This represents the proportion of "explainable information" (improvement over the noise model) the user is willing to sacrifice for model simplicity.

3. Key Contributions

1. Novel Paradigm: Introduces "Likelihood as Data," shifting the focus from comparing aggregate likelihoods to modeling the distribution of per-observation likelihoods to infer KL divergences directly.
2. Robustness to Misspecification: The method does not assume any candidate model is correct. It explicitly quantifies the uncertainty in the distance of each model from the true DGP.
3. Stability under Ties: Unlike standard Bayesian posteriors or hard-minimum rules, which can arbitrarily split probability mass among tied models (leading to instability), the SLC score provides a stable, reproducible inference that correctly identifies sets of equally good models.
4. Theoretical Guarantees: The paper proves that the SLC score is asymptotically consistent. As $n \to \infty$, the score concentrates on the true set of best minimal-complexity $\delta$-optimal models, even in the presence of ties.
5. Practical Utility: Provides a mechanism to trade off model complexity and fit via the interpretable parameter $\tau$, allowing users to select parsimonious models without overfitting.

4. Results and Empirical Evaluation

The authors validate the method through simulations and real-world applications:

• Gaussian Mixture Models (Shapley Galaxy Data): Standard methods (AIC, BIC, standard Bayes) selected increasingly complex models (large $k$) as sample size grew, failing to stabilize. LaD correctly identified that a simpler model (e.g., $k \approx 4$) was sufficient for a given tolerance, stabilizing as $n$ increased.
• Sparse Multivariate Normal Models: In a simulation with 7 candidate models (varying sparsity), LaD correctly identified the simplest model that was within $\delta$ of the truth. It outperformed "hard" selection methods and the Coarsened Posterior (c-posterior) in terms of Brier score, particularly when models were tied in performance.
• Thermal Performance Curves (Microbial Ecology): Applied to 9 different growth rate models. LaD successfully navigated the trade-off between complex mechanistic models and simpler phenomenological ones, showing that simpler models (e.g., 3-parameter Brière I) were sufficient for large tolerances, while complex models were only needed for extremely tight tolerances.
• Population Structure (Brook Trout Genetics): Compared against Evanno's method (a heuristic for STRUCTURE). Evanno's method suggested $k=2$, but LaD showed that $k=2$ captured only ~40% of the explainable improvement, whereas $k=1$ was nearly as good, and $k=3/4$ offered substantial gains. This highlighted the risk of over-interpreting small likelihood gains as structural necessity.

5. Significance

This paper addresses a critical gap in statistical practice: model selection under misspecification.

• Calibrated Inference: It moves beyond point estimates to provide a full posterior distribution over model quality, allowing researchers to make decisions based on risk tolerance (via $\delta$ or $\tau$).
• Reproducibility: By solving the instability issues associated with tied models, it ensures that model selection results are reproducible and not artifacts of random posterior fluctuations.
• Interpretability: The introduction of the rescaled tolerance $\tau$ bridges the gap between abstract statistical criteria and practical decision-making, allowing domain experts to specify how much predictive power they are willing to lose for the sake of simplicity.

In summary, the LaD framework offers a theoretically grounded, computationally feasible, and robust alternative to traditional model selection, particularly valuable in complex settings where the "true" model is unknown and candidate models are imperfect approximations.