Technical Summary: Towards Leveraging AutoML for Sustainable Deep Learning

Problem Statement

Deep Learning (DL) models, while effective at extracting complex patterns, impose significant computational and environmental costs, particularly regarding energy consumption and carbon emissions. This challenge is acute in resource-constrained environments like IoT and edge devices. Deep Shift Neural Networks (DSNNs) offer a potential solution by replacing costly floating-point multiplication operations with bit-wise shift operations and sign flipping, thereby reducing computational complexity and latency. However, the configuration of DSNNs—specifically the precision of shift operations and the depth of shift layers—significantly impacts both performance and efficiency. Manually tuning these hyperparameters is inefficient, and there is a lack of systematic approaches to optimize DSNNs for the dual goals of high accuracy and minimal energy consumption.

Methodology

The authors propose a Green AutoML framework that integrates Multi-Fidelity (MF) and Multi-Objective (MO) optimization to automate the configuration of DSNNs. The approach utilizes the SMAC3 framework for hyperparameter optimization (HPO).

Core Components:

1. Multi-Fidelity Optimization (MF):

   • To address the high computational cost of training full DSNN configurations, the authors employ a multi-fidelity strategy.
   • Fidelity Definition: The fidelity level is defined by the number of shift layers in the model. Training begins with models having fewer shift layers (lower fidelity, lower computational cost, higher intrinsic uncertainty) and progressively increases the number of layers.
   • Algorithm: The HyperBand algorithm is used to implement MF. It runs "Successive Halving" across multiple brackets, allocating a small initial budget ($b_I$) to many configurations ($n_c$) and iteratively training the best-performing ones on larger budgets. This allows the search to discard poor configurations early while focusing resources on promising ones.

2. Multi-Objective Optimization (MO):

   • The optimization targets two competing objectives simultaneously:
     • Accuracy: Minimizing the loss function ($f_{loss}$) to maximize predictive performance.
     • Sustainability: Minimizing energy consumption and carbon emissions ($f_{emission}$) during training and inference.
   • Algorithm: The ParEGO algorithm is used to handle the multi-objective nature. It transforms the MO problem into a series of single-objective problems by introducing varying weight parameters for the objectives in each iteration. This allows the framework to approximate the Pareto front, identifying configurations that offer the best trade-offs between accuracy and emissions.

3. Measurement:

   • CodeCarbon is integrated into the workflow to track real-time energy consumption and calculate carbon emissions (in gCO2eq) based on regional energy mixes.

Experimental Setup

• Dataset: CIFAR-10.
• Base Architecture: Pre-trained ResNet20 adapted as a DSNN.
• Hardware: NVIDIA A100 GPUs.
• Search Space: Includes hyperparameters such as batch size, optimizer, learning rate, momentum, epochs, weight bits, activation integer/fraction bits, shift depth, shift type (Quantization vs. Powers of two/Sign), rounding method, and weight decay.
• Baselines: Comparison against a DSNN with default configurations (as defined by Elhoushi et al., 2021) and unoptimized models.

Key Results

The study evaluated 33 configurations using the MFMO approach. Key findings include:

• Performance Improvement: The MF algorithm successfully identified an optimal configuration that outperformed the default DSNN configuration by at least 3% in Top-1 test accuracy.
• Pareto-Optimal Solutions: Two distinct Pareto-optimal solutions were identified:
  • Solution 1: 83.50% accuracy with 0.1661 gCO2eq emissions. This configuration utilized fewer shift layers (5) and fewer representation bits.
  • Solution 2: 84.67% accuracy with 0.1673 gCO2eq emissions. This configuration utilized significantly more shift layers (16) but fewer representation bits for weights/activations.
• Trade-off Insights: The results suggest a trade-off between the number of shift layers and the number of representation bits. Solution 1 achieved similar emissions to Solution 2 despite having fewer layers, likely because the reduced bit-width compensated for the computational demand. Conversely, Solution 2 achieved higher accuracy by increasing shift depth but required more representation bits.
• Validation of MF: The experiment confirmed that evaluating models under low model-intrinsic uncertainty (fewer shift layers) effectively guides the search toward high-performing networks when instantiated with full shift layers.

Contributions and Significance

The paper claims the following contributions to the field of sustainable AI:

1. DSNN-Specific Configuration Space: The definition of a tailored hyperparameter search space specifically for Deep Shift Neural Networks.
2. Green AutoML Framework: A novel approach combining Multi-Fidelity and Multi-Objective optimization within SMAC3 to build efficiency-driven models.
3. Design Insights: Empirical evidence regarding the trade-offs between shift depth, representation bits, accuracy, and energy consumption, offering guidance for future DSNN design.
4. Methodological Integration: A demonstration of how ParEGO and HyperBand can be combined to optimize for both performance and environmental impact.

The authors conclude that their method accelerates efficient model development and enables sustainable AI applications. They acknowledge that future work is needed to extend this approach to broader benchmarks and architectures, and to refine the integration of ParEGO and HyperBand to further reduce the computational load of generating Pareto fronts.