🤖 machine learning

When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

This paper introduces RLScale-Bench, a reproducible benchmark demonstrating that a properly calibrated rule-based autoscaler consistently outperforms six mainstream deep reinforcement learning algorithms in cost efficiency across diverse workloads, challenging the assumption that DRL is inherently superior for adaptive resource control.

Original authors: Guilin Zhang, Chuanyi Sun, Kai Zhao, Shahryar Sarkani, John Fossaceca

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guilin Zhang, Chuanyi Sun, Kai Zhao, Shahryar Sarkani, John Fossaceca

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy restaurant kitchen. Your job is to decide how many chefs (computers) to keep on duty based on how many customers (requests) are walking through the door. You have two main goals: keep the customers happy (no long wait times) and keep the bill low (don't hire too many chefs).

For years, researchers have been trying to teach computers to make these decisions using Deep Reinforcement Learning (DRL). Think of DRL as a super-smart, ambitious apprentice chef who learns by trial and error. The hope was that this apprentice would eventually outperform the Rule-Based System, which is like a veteran head chef who follows a simple, strict rulebook: "If the kitchen is more than 70% full, hire one more chef. If it's empty, fire one."

This paper, titled "When Does Deep RL Beat Calibrated Baselines?", is a massive, rigorous taste-test to see if the apprentice can actually beat the veteran. The researchers built a simulator called RLSCALE-BENCH to run 240 different "dinner rushes" to see who performs better.

Here is what they found, explained simply:

1. The Veteran Chef (Rule-Based) Usually Wins on Cost

The most surprising finding is that the simple, rule-based chef (the "calibrated baseline") was the cheapest option in almost every scenario.

The Analogy: The rule-based chef is like a cautious driver who always stays in the right lane and follows the speed limit perfectly. They never get a ticket (service violations) and they never waste gas (money).
The Result: In six different types of traffic patterns (from steady flow to sudden rushes), the rule-based system spent the least amount of money while keeping the restaurant running smoothly. The "smart" AI apprentices often hired too many chefs, driving up the cost without needing to.

2. The "Apprentice" Only Shines in Chaos

The only time the AI apprentice (specifically one called PPO) beat the veteran chef was during unpredictable, chaotic rushes (like a sudden flash sale or a viral event).

The Analogy: Imagine a sudden flood of customers. The rule-based chef reacts after the kitchen gets crowded. The AI apprentice, having "seen" similar chaos in training, hires extra chefs before the line gets too long.
The Trade-off: The AI did reduce the number of angry customers (violations) by 54% during these chaotic moments, but it cost 24% more to run. The paper suggests this is only worth it if angry customers cost you more money than hiring extra chefs does.

3. The "Wrong Tool" Problem (Continuous vs. Discrete)

The study found a huge difference between two types of AI algorithms: those that think in discrete steps (like "add 1 chef" or "remove 1 chef") and those that think in continuous numbers (like "add 1.43 chefs").

The Analogy: You can't hire 1.43 chefs. You have to hire whole people.
The Result: The algorithms that tried to think in decimals (Continuous) were a disaster. They made mistakes 10 to 100 times worse than the ones that thought in whole numbers. It's like trying to drive a car with a steering wheel that only turns in tiny, invisible fractions—the car ends up crashing because the driver can't make a clear turn.

4. The "One Size Fits All" Myth

There is no single "best" AI algorithm.

The Analogy: It's like asking, "What is the best vehicle?" The answer depends on the road. A boat is great on water, a truck is great on dirt, and a sedan is great on a highway.
The Result: An algorithm that was the best at handling steady traffic might be the worst at handling a sudden spike. If you train an AI on one type of traffic and then send it to a different type of traffic, its performance ranking can drop by four spots. The "best" AI changes depending on the situation.

The Big Takeaway

The paper concludes that the reason AI hasn't taken over resource management yet isn't because the AI algorithms are bad. It's because:

The Rule-Based Baseline was too weak in past studies: Researchers often compared AI against a poorly tuned rule-based system, making the AI look good by default. When they tuned the rule-based system properly (the "calibrated" part), the AI struggled to beat it.
The Wrong Tools were used: Using "continuous" algorithms for "discrete" problems (like hiring whole people) leads to failure.
The Test was too easy: Many past studies only tested on one type of traffic. When you test on many different types, the results change completely.

In short: If you want to save money on a predictable schedule, stick to the simple, well-tuned rulebook. If you are dealing with wild, unpredictable chaos and can afford to pay a little extra for safety, a specific type of AI might help. But don't expect the AI to be a magic wand that beats a simple rulebook in every situation.

Technical Summary: When Does Deep RL Beat Calibrated Baselines?

Problem Statement

Adaptive resource control—allocating compute resources to dynamic workloads while adhering to cost and service-level objectives (SLOs)—is a critical decision-making problem in cloud computing. While Deep Reinforcement Learning (DRL) has been increasingly applied to this domain, existing literature presents conflicting conclusions regarding its efficacy compared to traditional rule-based controllers. Some studies claim significant cost reductions, while others find rule-based methods remain competitive.

The authors attribute this inconsistency to three systemic gaps in current research practices:

Uncalibrated Baselines: Comparisons often use poorly tuned rule-based controllers (e.g., simple threshold autoscalers), where apparent RL gains reflect baseline weakness rather than algorithmic superiority.
Single-Seed Reporting: DRL training exhibits high variance across random seeds. Evaluations based on single runs often fall within the noise margin, rendering benchmark rankings unreliable.
Narrow Workload Coverage: Most studies evaluate on one or two traffic patterns (e.g., diurnal or bursty), yielding conclusions that do not generalize to diverse production environments or distribution shifts.

Methodology: RLSCALE-BENCH

To address these gaps, the authors introduce RLSCALE-BENCH, a reproducible benchmark and evaluation protocol instantiated on Kubernetes Horizontal Pod Autoscaling (HPA). The benchmark evaluates six mainstream DRL algorithms against a rigorously calibrated rule-based baseline across six distinct workload patterns and five random seeds (totaling 240 evaluation runs).

Environment and Setup

Task: The agent manages pod replica counts for a service handling dynamic request streams.
State Space: A 6-dimensional vector including CPU utilization, memory usage, request rate, p95 latency, error rate, and current replica count.
Action Space: Discrete changes to replica counts $\{-2, -1, 0, +1, +2\}$ . Continuous-action algorithms (SAC, TD3, DDPG) are wrapped to map continuous outputs to these discrete bins.
Reward Function: A composite of infrastructure cost (USD per replica) and a penalty for SLO violations (latency threshold or error rate > 5%).
Workloads: Six patterns spanning production diversity: Constant, Periodic, Variable (random walk), Bursty (Poisson spikes), Ramp (linear increase), and Flash (sudden 3x spike).

Algorithms and Baselines

DRL Agents: PPO, DQN, A2C (discrete action); SAC, TD3, DDPG (continuous action). All share identical MLP architectures ( $[256, 256]$ ) and a matched training budget of 50,000 timesteps.
Calibrated Baseline: A production-faithful Kubernetes HPA tuned to a 70% CPU utilization target with realistic clamping bounds and decision intervals.
Random Baseline: A uniform random policy establishing a performance lower bound.

Evaluation Protocol

Models are evaluated on all six workload types over 240 decision steps. Metrics include total infrastructure cost, total SLO violations, and mean replica count, reported with mean $\pm$ standard deviation across five seeds. A distribution-shift study trains agents on the "Variable" workload and deploys them on the other five without retraining.

Key Results

1. Calibrated Baselines Outperform RL on Cost

Contrary to the assumption that DRL inherently reduces costs, the calibrated HPA baseline achieved the lowest infrastructure cost across all six workloads. It maintained fewer replicas on average (2.0–3.0) compared to RL agents (2.7–3.8) while achieving reasonable SLO compliance.

2. Discrete vs. Continuous Action Mismatch

A dramatic performance gap exists between algorithm families due to the action-space mismatch:

Discrete algorithms (PPO, DQN, A2C) significantly outperformed continuous algorithms (SAC, TD3, DDPG) in terms of SLO violations, often by one to two orders of magnitude.
Continuous algorithms suffered from "reward aliasing" caused by the discretization wrapper: flat reward plateaus within bins led to near-zero gradients, while bin edges caused high-variance jumps.
DDPG specifically converged to a degenerate single-replica policy, minimizing cost but incurring catastrophic SLO violations (300–989 per workload).

3. No Single Algorithm Dominates

Algorithm rankings are highly workload-dependent. No single DRL algorithm maintained a consistent rank across all patterns. For instance, SAC excelled on variable workloads but performed poorly on ramp traffic, while PPO showed the most stable ranking (2nd–3rd).

4. Distribution Shift and the "Transfer Tax"

When trained on the "Variable" workload and deployed on shifted distributions:

PPO generalized well to steady-state patterns (Constant, Periodic, Ramp) with zero violations but struggled with heavy-tail traffic (Bursty, Flash), showing a 54% reduction in violations compared to HPA on bursty traffic but at a 24% higher cost.
SAC demonstrated robustness on dynamic workloads but at the cost of systematically higher resource usage.
The calibrated HPA incurred no transfer tax because its policy is a fixed function of current utilization, avoiding overfitting to a specific training distribution.

Significance and Contributions

The paper challenges the prevailing narrative that deep RL straightforwardly outperforms rule-based control in resource management. The authors argue that the bottleneck in RL-based resource control is not algorithm selection, but rather:

Baseline Calibration: Properly tuned rule-based controllers are surprisingly strong competitors and should not be treated as strawmen.
Reward Engineering: The choice of action space (discrete vs. continuous) and reward design is critical; mismatched action spaces can lead to catastrophic failures.
Evaluation Protocols: Benchmarks must include distribution-shift evaluation and multi-seed reporting to reflect real-world deployment challenges.

Key Contributions:

C1: A matched-budget, multi-algorithm benchmark protocol eliminating implementation confounds.
C2: A calibrated HPA baseline serving as a strong, realistic comparator.
C3: A statistically rigorous evaluation (240 runs, 5 seeds, 6 workloads) with error bars.
C4: A distribution-shift generalization study revealing algorithm instability under deployment shifts.
C5: Three counter-intuitive findings demonstrating that calibrated baselines often win on cost, discrete actions are superior for indivisible resources, and no single RL algorithm dominates all workloads.

The authors conclude that progress in this field requires moving beyond algorithm novelty toward better reward engineering, environment calibration, and evaluation protocols that reflect real-world constraints. They release the full benchmark suite, trained models, and evaluation data to support these goals.