🤖 AI

Automated Computational Energy Minimization of ML Algorithms using Constrained Bayesian Optimization

This paper demonstrates that Constrained Bayesian Optimization can effectively minimize the energy consumption of machine learning model training while ensuring that predictive performance remains above a specified threshold.

Original authors: Pallavi Mitra, Felix Biessmann

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pallavi Mitra, Felix Biessmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of computing, a quiet but urgent problem has emerged alongside the dazzling rise of artificial intelligence. As machines learn to recognize faces, translate languages, and predict trends, they require vast amounts of electricity to train. This energy cost is no longer just a line item on a utility bill; it is a growing barrier to sustainability. For decades, researchers focused almost exclusively on making these models more accurate, often ignoring how much power they consumed to get there. However, as models grow larger and more complex, the energy required to train them is doubling roughly every three and a half months. This trend threatens to make the development of new intelligence tools unsustainable. The challenge, then, is not just to build smarter machines, but to build them in a way that respects the limits of our power grid. To solve this, scientists are turning to a method called Bayesian optimization, a smart search strategy that helps computers find the best settings for a task without needing to try every single possibility, much like a hiker using a map to find the lowest point in a valley without climbing every hill.

A team of researchers at the Berlin University of Applied Sciences has applied this search strategy to a new goal: minimizing the energy used to train machine learning models. Instead of simply looking for the settings that produce the most accurate predictions, they asked a different question. They wanted to find the settings that use the least amount of time and energy, provided the model still performs well enough to be useful. In their approach, they treated the model's accuracy not as the thing to be maximized, but as a strict rule that must be followed. If a model's performance dropped below a certain acceptable level, the search would discard those settings immediately. This method, known as constrained Bayesian optimization, allowed the researchers to navigate the vast landscape of possible choices—such as how to clean the data, how complex the model should be, and how long to train it—while keeping one foot firmly planted on the ground of required performance.

The researchers tested this approach on a variety of standard tasks, including predicting house prices and sorting news articles into categories. They compared their new method against the traditional way of doing things, where the computer tries to minimize energy but is only told to "try not to fail" on accuracy, rather than being given a hard limit. In the traditional method, the computer often finds a setting that uses very little energy but produces a model that is too inaccurate to be useful. To fix this, the traditional method usually adds a heavy penalty to the score whenever the accuracy is too low, hoping this discourages the computer from choosing bad settings. However, the researchers found that this penalty approach is clumsy and often fails to find the true balance. In contrast, their constrained method successfully identified settings that used significantly less time and energy while consistently meeting the required accuracy standards.

The results showed that by treating performance as a hard constraint rather than a secondary goal, the system could find a "sweet spot" that the other methods missed. In the experiments, the constrained approach consistently reached the target accuracy faster and with less total runtime than the traditional penalty-based method. This is crucial because, for these algorithms, the time spent training is directly linked to the energy consumed. The study demonstrates that it is possible to automate the selection of machine learning settings to save energy without sacrificing the quality of the results. The researchers noted, however, that their method relies on having a clear, pre-defined standard for what counts as "good enough" performance. In cases where the best possible performance is unknown or where the relationship between settings and energy is extremely complex, the method might need further refinement. Nevertheless, the findings offer a practical path forward for a more sustainable future in artificial intelligence, proving that we can train powerful models without burning through our energy resources.

Technical Summary: Automated Computational Energy Minimization of ML Algorithms using Constrained Bayesian Optimization

Problem Statement
The paper addresses the escalating energy consumption associated with training machine learning (ML) models, driven by increasing model sizes and the computational intensity of hyperparameter optimization (HPO). While traditional HPO research focuses primarily on minimizing validation loss to maximize predictive performance, this work argues that energy efficiency has become a critical constraint. The authors note that existing ML suites often lack appropriate tools for energy measurement and that the process of selecting optimal hyperparameters is itself a major energy consumer. The core problem is to automate the selection of hyperparameters to minimize computational energy consumption (proxied by wall-clock runtime) while strictly adhering to a predefined threshold for predictive performance (accuracy for classification, mean squared error for regression).

Methodology
The authors propose a framework using Constrained Bayesian Optimization (CBO) to solve this multi-objective problem. The methodology is structured as follows:

Objective and Constraint Formulation:
- Objective: Minimize $\tau(x)$ , the time consumed to train an ML model with hyperparameters $x$ .
- Constraints: Ensure the model meets a baseline performance $c_0$ . For classification, accuracy $c_c(x) \geq c_0$ ; for regression, mean squared error $c_r(x) \leq c_0$ .
Surrogate Modeling:
- The authors employ Gaussian Processes (GPs) with Matérn 5/2 kernels to model both the objective and constraint functions independently.
- To ensure the GP priors are valid (as raw time and error metrics are strictly positive), the authors apply logarithmic transformations relative to baseline values. Specifically, the objective is modeled as $f(x) = \log \tau(x) - \log \tau_b$ , and constraints are modeled as log-ratios of performance metrics against their baseline thresholds.
Acquisition Strategy:
- A joint acquisition function is defined as the product of two components:
  1. Expected Improvement (EI): Optimizes the objective (minimizing runtime).
  2. Probability of Feasibility (PoF): Estimates the likelihood that a candidate point satisfies the performance constraint.
- This joint approach allows the algorithm to learn feasible regions for the constraint simultaneously with optimal regions for the objective, ensuring that feasibility is considered during the decision-making process for the next sampling point.
Baseline Comparison:
- The proposed CBO is compared against an Unconstrained BO approach that incorporates a quadratic penalty function. In this baseline, the constraint violation is added to the objective function as a penalty term: $f'(x) = f(x) + \frac{1}{2}\rho \max(0, c(x))^2$ .

Experimental Setup and Results
The framework was evaluated on standard regression models (Lasso, Elastic Net, KNN, Decision Tree, AdaBoost) and classification models (Ridge, Logistic Regression, KNN, Random Forest). Experiments utilized two large datasets: California Housing (regression) and 20-Newsgroups (classification).

Performance: The results demonstrate that while Unconstrained BO with penalties often finds the minimum objective value, it frequently violates the performance constraints, leading to the addition of large penalty terms that degrade the effective solution.
Efficiency: CBO successfully identified hyperparameter configurations that minimized cumulative runtime (energy proxy) while consistently satisfying the predefined performance thresholds.
Visual Evidence: Figures 1 and 2 illustrate that CBO achieves lower cumulative runtimes compared to the penalized Unconstrained BO while maintaining the MSE or accuracy above/below the required thresholds.

Key Contributions and Significance
The paper claims the following contributions and significance:

Shift in HPO Focus: It proposes a paradigm shift in HPO where energy consumption (runtime) is the primary objective to be minimized, with predictive performance treated as a hard constraint rather than the sole optimization target.
Effective CBO Framework: The study demonstrates that a joint acquisition function (EI $\times$ PoF) is more effective than penalized unconstrained BO for this specific task. It avoids the pitfalls of penalty tuning and ensures that the final selected hyperparameters are both energy-efficient and performant.
Scalability and Sustainability: The work highlights the potential of CBO for modern ML applications involving large datasets and high-capacity models, offering a tool to secure a more sustainable and scalable future for ML engineering by reducing the computational cost of model training without compromising utility.

Limitations Acknowledged by the Authors
The authors modestly note two primary limitations:

Predefined Thresholds: The approach assumes a reasonable, pre-defined performance threshold exists (e.g., from service level agreements). In scenarios where default hyperparameters are already suboptimal or no clear baseline exists, the constraint definition may be challenging.
Independent Surrogate Modeling: The current framework models the energy and performance surrogate models independently. The authors acknowledge that hyperparameters often impact both metrics simultaneously (e.g., regularization strength or data preprocessing), and future work aims to explore acquisition functions that can capture these dependencies.

Technical Summary: Automated Computational Energy Minimization of ML Algorithms using Constrained Bayesian Optimization

More like this