Training Neural Networks with Optimal Double-Bayesian Learning

Technical Summary: Training Neural Networks with Optimal Double-Bayesian Learning

Problem Statement
Training neural networks remains largely an empirical process, with hyperparameters such as the learning rate and momentum typically selected based on rules of thumb, experience, or systematic grid searches rather than theoretical derivation. While the importance of these parameters in avoiding overfitting and ensuring unbiased generalization is widely acknowledged, there is no conclusive theory for determining their optimal values. Current literature often relies on adaptive learning rates (e.g., Adam) or second-order methods, yet the "final verdict" on the optimal training method and specific hyperparameter values remains unresolved. The sensitivity of training results to the learning rate—where small values risk local optima and large values cause oscillation—creates a trade-off that is difficult to negotiate without a principled framework.

Methodology: The Double-Bayesian Framework
The paper proposes a theoretical framework grounded in a "double-Bayesian" decision mechanism involving two antagonistic Bayesian processes. This approach addresses the inherent uncertainty in measurements by modeling the training process as a resolution of uncertainty between a measurement object and its measurement value.

1. Theoretical Foundation:

   • Dual Processes: The framework posits two parallel processes: one estimating the certainty of a measurement value ($P(A)$) and the other estimating the certainty of the measurement object ($P(B)$). These processes are linked by Bayes' theorem but operate with distinct interpretations of conditional probabilities ($P(A|B)$ and $P(B|A)$).
   • Inner and Outer Equations: The framework derives two equations. The "outer Bayes equation" relates the prior probabilities to the ratio of posterior probabilities via a logarithmic base $\lambda$. The "inner Bayes equation" is derived by inverting the input and base of the logarithm, moving the log expression to the priors.
   • Fixed Points and Uncertainty: Using information theory, the paper treats learning as the search for a fixed point where the input estimate maps to the target value. The model introduces an "intrinsic uncertainty" principle: knowing the angle $\phi$ (representing measurement value uncertainty) perfectly implies maximum uncertainty about the base $\lambda$ (representing object uncertainty), and vice versa.

2. Derivation of Optimal Hyperparameters:

   • The Golden Ratio: The solution to the inner Bayes equation occurs when the probability terms align with the golden ratio ($\phi \approx 0.62$).
   • Momentum ($\alpha$): By satisfying the inner equation at $\phi = \pi/4$ and accounting for the gradient of the outer equation, the paper derives a theoretical momentum weight: $\alpha = \sqrt{2} \cdot \phi \approx 0.874$.
   • Learning Rate ($\eta$): Through a dual process derivation, the learning rate is calculated as $\eta = (1 - \alpha)^2 \approx 0.016$.
   • Training Mechanism: The framework suggests that training involves alternating between satisfying the inner Bayes equation (minimizing uncertainty in one variable) and not satisfying the outer equation (maximizing uncertainty in the counterpart), effectively using $\alpha$ as the momentum term for the previous iteration and $\eta$ as the learning rate for the current iteration.

Experimental Validation
To validate the theoretically derived values ($\eta \approx 0.016$, $\alpha \approx 0.874$), the authors conducted extensive grid search experiments across four distinct tasks:

• Handwritten Digit Classification: MNIST dataset using a custom 11-layer CNN.
• Tuberculosis Classification: Chest X-rays (TBX11K dataset) using DenseNet121.
• Lung Segmentation: Chest X-rays (COVID-19 dataset) using YOLOv8m-seg.
• Malaria Cell Detection: Blood smear images using YOLOv8m.

The experiments systematically varied learning rates (0.0001 to 0.2) and momentum values (0.0 to 0.925) and compared Stochastic Gradient Descent (SGD) against the Adam optimizer. Evaluations included varying training set sizes (100%, 75%, 50%, 25%) and the introduction of noise to test sets to measure generalization and robustness.

Key Results

• Optimal Hyperparameters: The grid search results consistently showed that the theoretically derived learning rate ($\eta \approx 0.016$) and the close value of 0.01 yielded the best or near-best performance across classification, segmentation, and detection tasks. The derived momentum value ($\alpha \approx 0.874$) also fell within the top-performing range for SGD.
• SGD vs. Adam:
  • Performance: SGD, when tuned with the derived hyperparameters, consistently outperformed Adam in terms of final test accuracy, IoU, and mAP50 across most tasks and training set sizes.
  • Convergence: Adam converged significantly faster (2–3 times fewer epochs) to its minimum validation loss, explaining its popularity. However, this speed often came at the cost of generalization.
  • Overfitting: In several cases, Adam achieved lower validation loss but failed to achieve the best test set performance, indicating a higher tendency to overfit compared to SGD.
• Noise Robustness: Under noisy test conditions (randomly inverted pixel intensities), models trained with SGD demonstrated superior robustness. The error gap between SGD and Adam widened as noise levels increased, with SGD consistently maintaining lower error rates.

Significance and Claims
The paper claims to provide a theoretical justification for hyperparameter selection in neural network training, moving beyond empirical heuristics. Its primary contributions are:

1. Theoretical Derivation: It offers a specific, theoretically derived learning rate ($\approx 0.016$) and momentum weight ($\approx 0.874$) based on a double-Bayesian framework, linking information theory and Bayesian decision processes to network training.
2. Reinterpretation of Momentum: The framework provides a new explanation for the momentum term, suggesting it is not merely a smoothing factor but a necessary component to resolve intrinsic uncertainties between two parallel Bayesian processes.
3. Generalization Superiority: The results suggest that while adaptive methods like Adam offer faster convergence, optimally tuned SGD yields less biased models with better generalization, particularly in noisy environments.
4. Decision-Making Model: The double-Bayesian approach is presented as a novel decision-making model where knowledge is distributed across two processes, and the "optimal" state is a balance of uncertainties, analogous to the Heisenberg Uncertainty Principle.

The authors conclude that their framework offers new insights into the concept of optimality in network training and supports the use of SGD with specific, theoretically grounded hyperparameters to achieve unbiased and robust models.