Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning

Technical Summary: "The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning"

Problem Statement
Despite the theoretical promise of Bayesian Deep Learning (BDL) to provide principled uncertainty quantification, the practical adoption of Sampling-Based Inference (SAI) remains limited. The field is currently dominated by optimization-based approximate methods (e.g., Variational Inference, Deep Ensembles, Laplace Approximation) due to persistent misconceptions regarding the feasibility, scalability, and efficiency of sampling. The authors argue that the community has reached a turning point where SAI has achieved computational parity with optimization-based methods but is hindered by outdated beliefs and a lack of robust end-to-end workflows for managing posterior samples.

Methodology and Position
This position paper does not propose a single new algorithm but rather advocates for a paradigm shift in research focus and workflow design. The authors synthesize recent advancements in core sampling methodology (e.g., Hamiltonian Monte Carlo, No-U-Turn Sampler, Stochastic Gradient MCMC) and software frameworks (e.g., blackjax, posteriors) to argue that SAI is now viable for large-scale problems.

The paper outlines three primary pillars for this new course:

1. Dispelling Misconceptions: The authors systematically address common barriers, including:
   • Time Complexity: Recent evidence shows that once a good initialization is found, the time cost of sampling is comparable to the initial optimization step, often yielding superior performance even under equal computational budgets.
   • Scalability: Contrary to the belief that sampling cannot scale, recent studies demonstrate effectiveness in models with millions of parameters (e.g., ResNets, Vision Transformers, LLM adapters).
   • Prior Choices: Simple isotropic Gaussian priors are shown to be effective regularizers in overparameterized regimes, contrary to the view that they are too simplistic.
   • Discretization Error: In high-dimensional BNNs, the bias introduced by discretization (often corrected via Metropolis-Hastings adjustments) is often subdominant to initialization and Monte Carlo errors. Strict adjustment can lead to low acceptance rates and inefficiency; thus, unadjusted or lightly adjusted sampling is often preferable in practice.
2. Scalable Exploration Strategies: The paper advocates for moving away from restrictive posterior approximations (like fully factorized variational families) toward flexible exploration of the posterior landscape. Key strategies include:
   • Parallelized Exploration: Leveraging parallel chains to reduce autocorrelation and sensitivity to local approximation errors, often using optimization techniques to "warm-start" chains near the typical set.
   • Hybrid Approaches: Integrating optimization-based methods (like Deep Ensembles) to provide robust initializations for sampling routines.
3. Efficient Inference and Distillation: Recognizing that generating samples is only half the battle, the authors emphasize the critical need for efficient storage and inference.
   • Storage: They argue against "naive thinning" (discarding samples) and advocate for storing all generated samples on disk, as the cost of storage is low compared to the information loss.
   • Distillation: The paper proposes "posterior sample distillation" as a post-hoc optimization stage. This involves compressing a large set of posterior samples into a smaller, more informative set (via techniques like learned generative surrogates or importance sampling) to drastically reduce the number of forward passes required at test time.
   • Software Tooling: The authors call for the development of mature, end-to-end software ecosystems that handle sample storage, distillation, and serving, similar to established tools in classical Bayesian modeling (e.g., PyMC) and deep learning.

Key Results and Evidence
The paper synthesizes empirical results from recent literature (e.g., Sommer et al., 2025; Paulin et al., 2025; Deng et al., 2023) to support its claims:

• Performance Parity and Superiority: In tasks ranging from CIFAR-10 regression to ImageNet classification and LLM fine-tuning, SAI methods (such as Bayesian Deep Ensembles and SG-MCMC) consistently outperform or match strong optimization-based baselines (Deep Ensembles, SWA, Laplace) in terms of Log Pointwise Predictive Density (LPPD) and calibration, often with comparable or lower runtime.
• Scalability: Table 1 in the paper highlights successful applications of SAI on models with up to 218 million parameters (Llama 3 adapters) and 73 million parameters (Wide CNNs), demonstrating that SAI is not limited to small-scale toy problems.
• Inference Costs: Figure 3 illustrates that while inference costs for sampled BNNs can be high (up to 1000x vanilla networks), parallelization and distillation strategies can mitigate these costs, making them practical for deployment.

Significance and Claims
The paper claims that the field is at a critical juncture where the theoretical potential of SAI can finally be realized as a practical reality. The authors assert that:

• SAI is the Future: Sampling-based inference should supersede optimization-based approximations as the standard for effective and efficient inference in BNNs, offering superior uncertainty quantification and predictive performance.
• Research Focus Must Shift: The community must move beyond narrow algorithmic improvements to address "end-to-end" challenges, specifically the management, distillation, and deployment of posterior samples.
• Misconceptions are the Primary Barrier: The main obstacle to adoption is not computational intractability but rather persistent misconceptions and a lack of tooling. By addressing these, the community can unlock the full potential of BDL.
• Pragmatic Application in Foundation Models: While full-model sampling of massive foundation models (e.g., GPT-5) may remain infeasible, the paper argues for hybrid approaches (e.g., sampling low-rank adapters) that allow for robust uncertainty quantification in discriminative tasks and structured generation without prohibitive costs.

In conclusion, the paper positions SAI not merely as an alternative to optimization but as a necessary evolution for Bayesian Deep Learning, provided the community commits to building the necessary infrastructure and workflows to support it.