💻 computer science

Adaptive constrained reinforcement learning for energy-aware scheduling under changing load

This paper introduces CLASP, an adaptive constrained reinforcement learning scheduler that enforces explicit latency constraints via a PID-controlled Lagrangian price and price-conditioned policy, thereby maintaining stable energy efficiency and violation targets across varying workloads where traditional fixed-weight reward approaches fail.

Original authors: Saher Elsayed, Khairi Azhar Aziz

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saher Elsayed, Khairi Azhar Aziz

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Size-Fits-All" Trap

Imagine you are the manager of a busy coffee shop. You have two main goals:

Save Money: Don't run the espresso machines at full blast if you don't need to (save energy).
Keep Customers Happy: Make sure no one waits too long for their drink (meet the "latency target").

For years, computer scientists have tried to solve this by giving their AI a single "scorecard." They would say, "Every time you save a dollar of energy, you get 1 point. Every time a customer waits too long, you lose 10 points." The AI then tries to maximize its total score.

The paper argues this approach is broken.

Think of it like driving a car with a cruise control set to a fixed speed.

On a flat road (Light Load): The car is fine. But if you are just driving through a quiet neighborhood, that fixed speed is too fast. You are wasting gas (energy) when you could be driving slower.
On a steep hill (Heavy Load): That same fixed speed isn't enough. The car slows down, you miss your arrival time, and the passengers get angry (violating the service target).

The problem is that the "perfect speed" changes depending on how many cars are on the road (the workload). A single setting that works for a quiet morning fails miserably during the lunch rush. The paper calls this "operating point drift." The AI's "sweet spot" drifts away from what you actually want as soon as the situation changes.

The Solution: CLASP (The Smart Thermostat)

The authors introduce a new system called CLASP (Constrained Lagrangian Adaptive Scheduling Policy). Instead of giving the AI a fixed scorecard, they give it a rule and a thermostat.

1. The Rule (The Constraint):
Instead of saying "Maximize this score," the manager says: "I don't care how you do it, but no more than 5% of customers can wait too long." This is a hard rule, like a speed limit sign.

2. The Thermostat (The PID Controller):
This is the magic part. The system has a "thermostat" that constantly checks the temperature (the violation rate).

If too many customers are waiting (the room is too hot), the thermostat turns up the "heat" (the price of waiting).
If almost no one is waiting (the room is too cold), the thermostat turns down the heat.

This "price" is a number the AI sees. It tells the AI: "Hey, things are getting crowded! You need to work faster, or you'll get penalized heavily."

3. The "Price-Sensitive" AI:
The AI is trained to listen to this price. It's like a worker who knows:

"If the price is low, I can take my time and save energy."
"If the price is high, I need to sprint to keep the customers happy."

Because the AI learns to react to the price rather than a fixed setting, it can handle both the quiet morning and the lunch rush with the same brain.

What They Found (The Results)

The researchers tested this in a computer simulation of a server (like a digital coffee shop).

The Old Way (Fixed Weights): They tried every possible "scorecard" setting. None of them worked for all situations. Some saved energy but broke the rules when it got busy. Others followed the rules but wasted huge amounts of energy when it was quiet.
The New Way (CLASP):
- Accuracy: CLASP kept the "late customer" rate right at the 5% target, even as the workload changed from light to heavy.
- Efficiency: It used about half the energy of the "always run at max speed" method (which was the only other method that never broke the rules).
- Adaptability: When the workload suddenly jumped (like a sudden rush of customers), the thermostat quickly adjusted the price, and the AI sped up to handle it. When the rush died down, it slowed back down to save power.

The "Ablation" Test (Taking the Engine Apart)

To prove their idea worked, they broke the system apart:

Without the Thermostat: If they stopped adjusting the price and just kept it fixed, the system failed. It was too slow when busy and too fast when quiet.
Without the "Price-Sensitive" Brain: If they let the thermostat adjust the price but didn't tell the AI what the price was, the AI didn't know how to react. It kept doing the same thing regardless of the pressure.

Conclusion: You need both the thermostat to adjust the pressure and a brain that knows how to react to that pressure.

The Bottom Line

The paper shows that trying to find a single "perfect setting" for energy and speed is a fool's errand because the world changes. Instead, you should set a clear goal (e.g., "5% max errors") and build a system that constantly adjusts its internal "price" to meet that goal.

CLASP turns a rigid, brittle system into a flexible, self-regulating one. It's the difference between a robot that follows a script and a human manager who watches the line, sees the rush coming, and tells the staff to "pick up the pace" or "take a breather" as needed.

Note: The authors emphasize that these results are from computer simulations. While the math looks promising, they haven't tested this on real physical hardware yet.

Technical Summary: Adaptive Constrained Reinforcement Learning for Energy-Aware Scheduling

Problem Statement
Energy-aware schedulers must balance two conflicting objectives: minimizing energy consumption and maintaining responsiveness (meeting latency targets or Service-Level Objectives, SLOs). The dominant approach in recent literature uses Reinforcement Learning (RL) to learn this trade-off by scalarizing the objectives into a single weighted reward (e.g., a weighted sum of energy and latency penalty).

The authors identify a critical structural failure in this scalarized approach: operating-point drift. A fixed reward weight corresponds to a specific trade-off, but the resulting operating point (the actual balance of energy vs. latency) is not stable; it shifts as the workload changes.

Under light load: A fixed weight often leaves the processor running faster than necessary, wasting energy because the latency budget is underutilized.
Under heavy load: The same weight fails to provide sufficient speed, causing the system to silently drift past the latency target, accumulating violations.
Consequently, no single fixed weight can maintain a specific violation target across a range of changing loads.

Methodology: CLASP
The paper introduces CLASP (Constrained Lagrangian Adaptive Scheduling Policy), a scheduler that treats the latency-violation rate as an explicit constraint rather than a component of the reward. The method addresses two main challenges in applying constrained RL to scheduling: the sensitivity of dual variable updates and the non-stationarity of workloads.

Constrained Formulation: The problem is modeled as a Constrained Markov Decision Process (CMDP). The goal is to minimize expected power subject to a constraint that the long-run violation rate remains below a target $\epsilon$ (e.g., 5%).
Adaptive Dual Control (PID): Instead of using standard gradient ascent to update the Lagrange multiplier (the "price" on violations), CLASP employs a Proportional-Integral-Derivative (PID) controller. This controller adjusts the price $\lambda$ based on the error between the observed violation rate and the target. This stabilizes the dual variable, preventing the oscillations common in naive updates and allowing the price to settle at the value required by the current load.
Price Conditioning: To handle non-stationary workloads, the learned policy is conditioned on the current constraint pressure (the price $\lambda$ ) and a load estimate. The state space is augmented with a discretized bin of the current price. This allows a single policy to adapt its behavior (e.g., becoming more conservative or aggressive) based on the price signal, effectively spanning the entire range of operating points required by different loads.

Key Contributions

Identification of Drift: The authors quantify the operating-point drift of scalarized schedulers, demonstrating that no fixed reward weight in a tested sweep can maintain a 5% violation target across loads ranging from light to near-saturation.
Stabilized Dual Updates: The integration of a PID controller for the Lagrange multiplier successfully dampens oscillations and allows the price to converge to the necessary level for the current load.
Policy Conditioning: The paper demonstrates that conditioning the policy on the dual variable is essential for a single policy to function correctly across varying loads, solving the non-stationarity problem without retraining.

Results
Experiments were conducted in a simulated single-server environment with adjustable speed (modeling DVFS) and Poisson-distributed job arrivals.

Target Tracking: CLASP successfully tracked a 5% violation target across arrival loads of 1.5 to 3.0 jobs/step. Even near saturation (load 3.5), where the target becomes difficult to satisfy, CLASP degraded gracefully to ~6.9%, whereas fixed-weight policies drifted significantly (e.g., a balanced weight drifted from 3% to 31% violations).
Energy Efficiency: Compared to a "run-fastest" baseline (which is always compliant but energy-inefficient), CLASP used approximately 50% less modeled power on average. At light loads, savings were up to 70%; at heavy loads, savings were around 20%.
Non-Stationary Adaptation: In experiments with abrupt workload shifts, the PID controller adjusted the price online, bringing the violation rate back to the target after brief transients.
Ablation Studies: Removing the PID controller (fixing the price) resulted in over-conservative behavior at low loads and constraint violations at high loads. Removing price conditioning caused the policy to fail to adapt to different price levels, leading to significant drift. Both components were confirmed as necessary.

Significance and Claims
The paper argues that the central difficulty in learning energy-aware schedulers is not finding a good trade-off, but keeping the chosen trade-off stable as conditions change.

Shift in Framing: The work proposes a shift from "tuning a weight" (which encodes a fragile trade-off) to "tracking an outcome" (a bounded violation rate).
Controllability: By making the service objective an explicit constraint and controlling its price via feedback, the operating point becomes a controlled quantity rather than an emergent, drifting one.
Generalizability: The authors suggest this pattern—replacing scalarized rewards with constraint-based formulations using feedback-controlled prices—is applicable wherever a learned policy must honor a service objective under changing conditions, such as network admission control or edge device power management.

Limitations
The authors note that the results are based on a single-server, discrete-time simulation with a tabular Q-learning agent. The power values are modeled, not measured on hardware. The study isolates the constraint mechanism, and future work is needed to validate these findings on hardware testbeds and to extend the approach to deep RL, heterogeneous jobs, and multi-server environments.