Uncertainty-Aware Model Selection with a Calibrated Probability-Generating-Function-Based Bayesian Information Criterion

Technical Summary: Uncertainty-Aware Model Selection with a Calibrated PGF-BIC

Problem Statement
Selecting stochastic gene-expression models from single-cell RNA count data requires balancing goodness-of-fit against mechanistic complexity. While the Probability-Generating-Function-based Bayesian Information Criterion (PGF-BIC) offers a computationally efficient method for comparing models without reconstructing full count distributions, its conventional implementation relies on a zero-threshold rule. This rule selects the more complex model whenever its penalized score is lower than that of the simpler model. However, this approach fails to account for sampling uncertainty in the fitted score difference. In finite samples, fluctuations in the empirical probability generating function (PGF) and the estimated covariance weights can create apparent score advantages for complex models that are indistinguishable from noise, leading to the over-selection of unnecessarily complex models.

Methodology
The authors propose an uncertainty-aware PGF-BIC rule that replaces the fixed zero cutoff with a data-driven threshold. The methodology proceeds through the following technical steps:

1. Score Decomposition: The difference in penalized scores ($\Delta_n$) between a complex model ($M_2$) and a simpler model ($M_1$) is decomposed into a deterministic penalty term and a stochastic term representing the difference in minimized fitting distances. The stochastic component arises from the joint variability of the empirical PGF and the estimated covariance weights.
2. Threshold Formulation via Cantelli's Inequality: To determine a selection margin that controls the probability of incorrect selection, the authors employ Cantelli's one-sided inequality. This allows the derivation of a threshold expressed in terms of the standard deviation of the score difference, ensuring that the probability of selecting the complex model when it offers no true population-level advantage is bounded by a target level $\alpha$.
3. Influence Function Analysis: To estimate the required scale of sampling fluctuations without resampling (e.g., bootstrapping), the authors utilize influence functions. By treating the fitted score difference as a functional of the data distribution, they derive a first-order Taylor expansion. This yields a representation of the sampling fluctuation as a sum of cell-specific contributions ($\psi(X_i)$).
4. Variance Estimation: The variance of the influence function sum is estimated to quantify the standard deviation of the score difference. This estimate accounts for variability in both the empirical PGF and the estimated covariance weights.
5. Selection Rule: The complex model is selected only if its score advantage exceeds the calculated threshold ($\Delta_n > c_{n,\alpha}$), where $c_{n,\alpha}$ is derived from the estimated standard deviation and the target error rate.

Key Contributions

• Calibrated Decision Rule: The paper introduces a modified PGF-BIC rule that incorporates sampling uncertainty into the model selection decision, moving beyond the binary "lower score wins" approach.
• Analytical Variance Estimation: A novel derivation using influence functions provides a first-order description of sampling fluctuations in the PGF-BIC score difference. This enables the calculation of a data-driven threshold without the computational cost of resampling or additional optimization.
• Preservation of Efficiency: The calibration retains the computational efficiency of the original PGF-BIC framework, as it utilizes existing model fits and does not require reconstructing full count distributions or performing repeated likelihood evaluations.

Results
The authors validate the proposed method using a benchmark comparing a Poisson model (simpler) against a Bursty model (more complex), where the Poisson distribution is a boundary limit of the Bursty distribution.

• Simulation Findings: In simulations where data were generated by the Poisson model, the conventional PGF-BIC rule frequently selected the more complex Bursty model (incorrect selection rates of ~70% at $n=100$ and ~35% at $n=1000$).
• Calibration Performance: The uncertainty-aware rule significantly reduced the rate of incorrect selection of the complex model across various sample sizes ($n=100, 500, 1000$). The calibrated threshold successfully distinguished between genuine model advantages and fluctuations caused by sampling noise.

Significance and Claims
The paper claims that the proposed calibration addresses a critical limitation of standard model selection criteria in the context of single-cell data: the inability to distinguish reproducible fitting advantages from noise-induced artifacts in finite samples. By quantifying the uncertainty of the score difference, the method prevents the over-fitting of stochastic gene-expression models. The authors emphasize that this approach integrates uncertainty quantification into model selection while maintaining the computational tractability required for analyzing large-scale single-cell datasets, requiring neither resampling nor additional optimization steps.