A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning

Technical Summary: A Tale of Two Variances

"A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning" investigates a critical flaw in how machine learning models are evaluated: the over-reliance on single-seed endpoint metrics. The authors argue that in limited-data settings, a single reported value (such as the Continuous Ranked Probability Score, CRPS) is a random variable that may fail to represent the true expected performance of a method.

1. The Problem: Benchmark Unreliability

In probabilistic regression, benchmarks typically report a single mean value of an evaluation metric (e.g., CRPS) obtained from one training run. However, because training involves stochastic processes (data sampling, random initialization), this endpoint is itself a random variable.

The authors identify a specific phenomenon: non-monotone variance trajectories. While one might expect evaluation uncertainty to decrease smoothly as more data is added (following a power law), certain methods exhibit "variance spikes" at intermediate training sizes. In these regions, a single-seed report becomes a highly unreliable estimate of the method's true mean performance, potentially leading to incorrect conclusions about which model is superior.

2. Methodology

The researchers conducted an extensive empirical study across six regression datasets (synthetic and real-world) using 50 independent repetitions per training size to capture the true distribution of the metrics.

• Metric of Interest: The Continuous Ranked Probability Score (CRPS), chosen for its ability to evaluate both calibration and sharpness.
• Method Categorization:
  • Direct Parameterization Methods: (e.g., MAP, Deep Ensembles) These learn predictive variance through a heteroscedastic output head (learning $\sigma^2$ directly).
  • Implicit Uncertainty Methods: (e.g., MC Dropout, Bayes by Backprop) These obtain uncertainty through stochastic averaging or posterior sampling without a dedicated variance head.
• Analytical Framework:
  • Trajectory Analysis: They model the variance of the CRPS as a function of training size $n$ using a power-law form ($Var \approx Cn^{-\alpha}$). They use the coefficient of determination ($R^2$) and monotonicity to classify trajectories as "regular" or "irregular."
  • Reliability Quantification: They measure the "self-estimation error" of a single run using relative RMSE and the probability that a single run falls within $\pm 10\%$ of the repeated-run mean.
  • Intervention: To test the cause of instability, they replaced the standard heteroscedastic Negative Log-Likelihood (NLL) with a $\beta$-NLL objective.

3. Key Contributions

1. Identification of Variance Spikes: They prove that direct parameterization methods (MAP, Deep Ensembles) exhibit pronounced, reproducible variance peaks at intermediate training sizes on real-world datasets.
2. Diagnostic Framework: They propose a two-level view of reliability:
   • Method-level: Using $R^2$ and monotonicity to identify methods with globally irregular behavior.
   • Training-size level: Using local variance as a direct signal of the risk of mismeasurement.
3. Causal Insight: They provide evidence that the standard heteroscedastic training objective is a primary driver of this instability.
4. Validation of Local Variance as a Predictor: They demonstrate a near-perfect Spearman correlation ($>0.96$) between local CRPS variance and the actual error of single-seed estimates.

4. Key Results

• The Failure of Single Seeds: On the Seoul Bike dataset, the relative RMSE of a single-seed MAP estimate reaches 93.6% at the variance peak, and the probability of the result being within 10% of the true mean drops to a mere 5.9%.
• Direct vs. Implicit Split: Implicit methods (MCD, BBB) generally show smooth, monotone variance contraction. Direct methods (MAP, Ensembles) show irregular, non-monotone trajectories with low $R^2$ on real datasets.
• The $\beta$-NLL Effect: Replacing the standard objective with $\beta$-NLL (where $\beta=0.5$) significantly regularizes the trajectories, restoring smooth power-law decay and high $R^2$ values (e.g., increasing Kin8nm MAP $R^2$ from 0.158 to 0.950).
• Metric Sensitivity: CRPS and Interval Score are highly reliable for diagnosing this instability, whereas NLL and Interval Coverage Probability (PICP) are much less consistent.

5. Significance and Practical Implications

This paper shifts the perspective of benchmarking from "comparing final numbers" to "evaluating the stability of the measurement process."

For Practitioners:

• Don't trust single seeds in small-data regimes: If your training size is in the "intermediate" range, a single run is statistically dangerous.
• Report Trajectory Summaries: Instead of just reporting the final mean, researchers should report the regularity ($R^2$ and monotonicity) of their evaluation metrics.
• Targeted Repetitions: Rather than running 50 seeds for every single data point, practitioners should concentrate computational resources on high-variance regions identified by local variance analysis.
• Objective Choice: For methods learning variance directly, using $\beta$-NLL is a low-cost way to ensure more stable and reliable benchmarks.