Technical Summary: Leveraging AutoML for Sustainable Deep Learning on Deep Shift Neural Networks

Problem Statement

Deep Learning (DL) models, while effective at extracting complex patterns, impose significant computational and environmental burdens, particularly in resource-constrained environments such as edge devices and IoT systems. Deep Shift Neural Networks (DSNNs) have emerged as a promising alternative to traditional Deep Neural Networks (DNNs) by replacing expensive floating-point multiplication operations with computationally cheaper bit-wise shift operations. This architectural shift aims to reduce inference latency and power consumption. However, DSNNs remain less understood and less optimized than standard DNNs. Key challenges include determining the optimal level of precision for shift operations to minimize quantization errors without negating efficiency gains, and the lack of systematic knowledge regarding how specific design choices (e.g., shift depth, quantization strategy) impact the trade-off between predictive accuracy and energy consumption.

Methodology

The authors propose a Green AutoML framework that integrates Multi-Fidelity (MF) and Multi-Objective (MO) Hyperparameter Optimization (HPO) to automatically design and optimize DSNNs.

1. Configuration Space

The study defines a configuration space tailored specifically for DSNNs, encompassing:

• Standard Training Hyperparameters: Batch size, optimizer, learning rate, momentum, and weight decay.
• DSNN-Specific Hyperparameters:
  • Shift Depth: The number of layers converted to shift operations.
  • Shift Type: Choice between DeepShift-Q (quantization-based) and DeepShift-PS (direct training of shifts).
  • Precision: Bit precision for weights and activation integers/fractions.
  • Rounding Type: Deterministic or stochastic rounding.

2. Optimization Framework

The authors extend the SMAC3 HPO package to implement a Multi-Fidelity Multi-Objective (MFMO) approach:

• Multi-Fidelity (MF): To reduce computational costs, the framework employs the HyperBand algorithm. Configurations are evaluated at varying levels of approximation (e.g., fewer training epochs) before promising candidates are trained longer.
• Multi-Objective (MO): The optimization targets two competing objectives simultaneously:
  1. Minimizing Loss: To maximize classification accuracy.
  2. Minimizing Emissions: To reduce environmental impact.
• Algorithm: The ParEGO algorithm is used to compute Pareto optimal configurations by transforming the multi-objective problem into a series of single-objective problems with varying weights.
• Emission Tracking: The CodeCarbon library is integrated to track energy consumption and carbon emissions (in kgCO2eq) during both training and inference phases.

3. Experimental Setup

• Datasets: CIFAR10 and Caltech101.
• Architectures: ResNet20, MobileNetV2, GoogLeNet, and EfficientNetV2.
• Hardware: NVIDIA A100 GPUs.
• Evaluation: The study compares the Pareto fronts generated by the AutoML approach against the default DSNN configurations proposed by Elhoushi et al. (2021).

Key Contributions

1. Tailored Configuration Space: The specification of the first configuration space specifically designed for DSNNs, optimized via a combination of multi-objective and multi-fidelity AutoML.
2. Empirical Trade-off Analysis: An empirical exploration of how specific DSNN design choices influence the trade-off between accuracy and energy efficiency, providing actionable insights for developing energy-efficient applications.
3. Superior Configurations: The identification of specific DSNN configurations that dominate default baselines in both performance (loss) and efficiency (emissions) dimensions.

Key Results

• Performance and Efficiency Gains: The MFMO approach discovered configurations that significantly outperformed default DSNN settings. Specifically, the authors observed:
  • An approximate 20% increase in performance (reduction in loss) compared to defaults.
  • Emission reductions ranging from 10% to over 60% in various scenarios.
  • For EfficientNetV2, emissions were reduced by more than 20% while maintaining or improving performance.
• Counter-Intuitive Findings on Quantization:
  • Shift Depth: Contrary to the expectation that deeper shift networks yield better efficiency, the optimal configurations often utilized very low shift depths (e.g., 1 to 3 layers), rather than the maximum depth (20).
  • Activation Precision: High precision in activation fraction bits (often near the upper bound of 32 bits) was frequently observed in optimal configurations, suggesting that retaining fractional precision is crucial for maintaining accuracy when using shift operations.
  • Shift Type: The choice between DeepShift-Q and DeepShift-PS showed low importance in the optimization landscape, with neither type consistently dominating the other across the Pareto front.
• Hyperparameter Importance: Analysis using fANOVA revealed that activation integer and fraction bits are the most critical factors for minimizing emissions, while shift depth is the most critical for minimizing loss. Surprisingly, the learning rate was identified as a highly influential factor for emissions, where suboptimal choices led to significant variability in energy consumption.
• Transferability: Configurations optimized on CIFAR-10 demonstrated robustness when transferred to ImageNet100, often outperforming the default configuration on the new dataset without re-optimization.

Significance and Claims

The paper claims that systematic, automated optimization is essential for unlocking the full potential of DSNNs. The authors argue that:

• Default configurations are suboptimal: The manually designed defaults by Elhoushi et al. (2021) are Pareto-dominated by configurations found via AutoML.
• Nuanced Design is Required: The relationship between hyperparameters (shift depth, precision) and efficiency is non-linear and counter-intuitive. Simple heuristics (e.g., "maximize shift depth") do not yield the best results.
• Green AutoML Viability: The approach successfully balances the "Green" objective (minimizing emissions) with the "ML" objective (maximizing accuracy). The computational cost of the AutoML search process is amortized by the long-term inference savings, particularly in real-time applications like automated driving.
• Generalizability: The insights regarding low shift depths and high activation precision hold across diverse architectures (ResNet, MobileNet, GoogLeNet), suggesting a broader applicability to quantized and resource-efficient neural networks.

The authors conclude that their work provides a foundation for designing sustainable AI systems, emphasizing that automated, multi-objective search is necessary to navigate the complex design space of efficient neural networks.