Can Adjusting Hyperparameters Lead to Green Deep Learning: An Empirical Study on Correlations between Hyperparameters and Energy Consumption of Deep Learning Models

This empirical study demonstrates that strategically adjusting hyperparameters in deep learning models can significantly reduce energy consumption and promote "green" AI, particularly when multiple models are trained in parallel, without compromising performance.

The Gist

Imagine you are a chef running a massive, high-tech kitchen. Your goal is to cook the perfect dish (a Deep Learning model) that tastes amazing (high accuracy). But there's a catch: your kitchen is incredibly energy-hungry. The ovens, mixers, and lights are burning through electricity so fast that your carbon footprint is huge, and your electric bill is skyrocketing.

For years, chefs (data scientists) have been focused only on making the dish taste better, often ignoring how much energy it takes to cook it. They just crank up the heat and throw in more ingredients, assuming that's the only way to get a good result.

This paper is like a new kind of energy audit for that kitchen. The researchers asked a simple question: "If we tweak the recipe just a little bit—without changing the ingredients or the main cooking method—can we cook the same delicious dish using less electricity?"

Here is the breakdown of their study using everyday analogies:

1. The "Recipe Tweaks" (Hyperparameters)

In the world of AI, the "recipe" has settings called hyperparameters. Think of these as the knobs on your stove or the timer on your oven.

• Epochs: How many times you stir the pot. (Do you need to stir it 100 times, or is 60 enough?)
• Learning Rate: How fast you add ingredients. (Do you pour the sauce in a slow drizzle or a fast flood?)
• Other settings: Things like how much salt (weight decay) or heat (gamma) you use.

The researchers didn't just guess; they used a technique called Mutation Testing. Imagine a robot chef that randomly turns these knobs up or down by a small amount, just like a real chef might experiment. They created hundreds of "mutated" versions of the same recipe to see what happened.

2. The Experiment: Single vs. Double Cooking

They ran two types of experiments:

• The Solo Chef: Cooking one dish at a time.
• The Busy Kitchen: Cooking two dishes at the same time on the same stove (Parallel Environment). This is how most real-world servers work today.

They measured two things for every single "mutated" recipe:

1. The Taste: Did the model still work well? (Accuracy)
2. The Electric Bill: How much energy did the GPU (the super-chef's brain) and the CPU (the kitchen staff) use?

3. The Big Discoveries

Discovery A: The "Sweet Spot" Exists
They found that many of these recipe knobs are directly linked to the electric bill.

• The "Stirring" Knob (Epochs): If you stir the pot too many times, you waste energy. The study showed that you can often stop stirring a bit earlier than you think and still get a perfect dish. You save energy without ruining the taste.
• The "Pouring" Knob (Learning Rate): Changing how fast you add ingredients has a weird effect. Sometimes, slowing it down saves a ton of electricity on the GPU (the most expensive part of the kitchen) without hurting the final taste.

Discovery B: The "Busy Kitchen" Effect
This was the most surprising part. When they cooked two dishes at once (Parallel Environment), the energy usage became much more sensitive to these tiny tweaks.

• Analogy: Imagine two people trying to talk in a quiet room vs. a noisy party. In the quiet room (single model), a small change in volume doesn't matter much. In the noisy party (parallel models), a small change in volume causes a huge reaction.
• Result: In a busy server, tweaking the settings can save even more energy than when cooking alone, but it's also riskier. You have to be more careful.

4. The "Green" Conclusion

The main takeaway is that you don't need to invent a new engine to save fuel; you just need to drive better.

The researchers found that by simply adjusting these existing settings (the knobs), developers can create "Green AI." They can train models that:

• Use significantly less electricity.
• Produce less carbon dioxide.
• Save money on hardware costs.
• And most importantly: They still taste just as good (maintain the same accuracy).

Why This Matters

Think of Deep Learning models as cars. For a long time, we've been building bigger, faster cars with V12 engines, ignoring the gas mileage. This paper suggests that we don't necessarily need a new car; we just need to learn how to drive the one we have more efficiently. By paying attention to the "knobs" (hyperparameters) and understanding how they affect the "fuel gauge" (energy), we can make Artificial Intelligence much more sustainable for our planet.

In short: Small tweaks to the settings can lead to a massive drop in energy bills, making AI friendlier to the environment without sacrificing its smarts.

Technical Summary

Here is a detailed technical summary of the paper "Can Adjusting Hyperparameters Lead to Green Deep Learning: An Empirical Study on Correlations between Hyperparameters and Energy Consumption of Deep Learning Models."

1. Problem Statement

As Deep Learning (DL) models grow in complexity and dataset sizes increase, their computational resource requirements and energy consumption have surged dramatically (reportedly increasing 3,000,000× between 2012 and 2018). This high energy consumption leads to significant environmental impact (carbon footprint) and financial costs. While software engineering (SE) researchers have begun addressing "Green DL," existing studies primarily focus on framework comparisons or model architecture. There is a lack of empirical research investigating how hyperparameter tuning (e.g., learning rates, epochs) specifically influences energy consumption. The core problem is determining whether adjusting these parameters can reduce energy usage without compromising model performance.

2. Methodology

The authors propose an empirical approach based on mutation testing to simulate how practitioners adjust hyperparameters. The methodology consists of three main phases:

• Model Selection & Mutation Construction:

  • Five real-world DL models (ranging from simple CNNs to ResNet and HRNet) were selected using datasets like MNIST, CIFAR-10, and Market-1501.
  • Mutation Operators: Instead of introducing faults (traditional mutation testing), the authors used mutation operators to generate variations in hyperparameters within reasonable ranges.
  • Hyperparameters Studied: Epochs, Learning Rate, Gamma, Weight Decay, and Threshold.
  • Experimental Scale: For each of the 5 models, 3 specific hyperparameters were mutated 5 times, with each configuration trained 5 times to mitigate randomness. This resulted in 375 mutated models.

• Data Collection (Metrics):

  • Experiments were conducted on a server with Intel Xeon CPUs and NVIDIA RTX 3080 GPUs.
  • Energy Metrics: Measured using perf (for CPU Package and RAM energy) and nvidia-smi (for GPU power draw).
  • Performance Metrics: Training time and model accuracy.
  • Scenarios: Experiments were run in two modes:
    1. Single: Models trained individually.
    2. Parallel: Two models trained simultaneously on the same hardware to simulate shared server environments.

• Analysis Techniques:

  • Spearman Correlation Analysis: To determine the relationship between hyperparameters and energy/performance metrics.
  • Trade-off Analysis: Using Wilcoxon signed-rank tests and Cliff's delta to classify mutations as "Win" (better energy/performance), "Tie" (no significant change), or "Loss" (worse).
  • OLS Regression: To quantify the influence of hyperparameters on metrics.

3. Key Contributions

1. Novel Approach: Introduced a mutation-based framework to analyze the correlation between hyperparameter adjustments and energy consumption in DL models.
2. Scenario Expansion: Investigated both single and parallel training environments, revealing that parallelism significantly alters the sensitivity of energy consumption to hyperparameter changes.
3. Empirical Evidence: Provided extensive data from 375 model variations across 5 real-world models, offering concrete evidence that hyperparameter tuning is a viable strategy for "Green DL."

4. Key Results

RQ1: Correlation between Hyperparameters and Energy

• Epochs: Showed a strong positive correlation with all energy metrics (Package, RAM, GPU) and time cost. Reducing epochs significantly lowers energy consumption with minimal impact on accuracy.
• Learning Rate: Exhibited a negative correlation with energy consumption (specifically GPU and RAM). Lowering the learning rate often reduced energy usage, though it required careful tuning to avoid performance degradation.
• Other Parameters: Gamma and Weight Decay showed weaker but significant correlations, particularly with GPU energy consumption.

RQ2: Can DL Models be Made Greener?

• Feasibility: Yes. The study identified "Win" scenarios where mutations resulted in lower energy consumption with comparable or better performance.
• Specific Findings:
  • Epochs: Reducing epochs within a suitable range is the most effective way to save energy without hurting accuracy.
  • Learning Rate: Purposeful adjustment (rather than random mutation) can yield greener models, especially for GPU-bound tasks.
  • Trade-offs: While random mutations often hurt performance, targeted adjustments can optimize the energy-performance trade-off.

RQ3: Impact of Parallel Environments

• Increased Sensitivity: In parallel environments, energy consumption became more sensitive to hyperparameter changes than in single-model training.
• Stability: Model performance (accuracy) remained more stable in parallel environments, while energy metrics fluctuated more significantly.
• Discrepancies: The correlation patterns for weak correlations (e.g., Gamma vs. RAM) shifted slightly in parallel settings compared to single settings.

5. Significance and Implications

• For Practitioners: DL developers should treat hyperparameter tuning not just as a performance optimization task but as a critical lever for energy efficiency. Small adjustments to epochs and learning rates can yield significant energy savings.
• For Green Software: This study bridges the gap between Software Engineering and Green Computing, demonstrating that "Green DL" can be achieved through configuration management rather than solely through hardware upgrades or architectural changes.
• Future Research: The findings highlight the need to consider the training environment (single vs. parallel) when optimizing for energy, as parallelism changes the energy landscape. The authors provide a replication package to encourage further study in this domain.

Conclusion: The paper empirically proves that adjusting hyperparameters is a practical and effective method to create greener Deep Learning models, reducing energy consumption without sacrificing model quality, particularly when accounting for parallel training environments.