💻 computer science

Research on Cloud Task Scheduling Problem Based on Pareto Optimization Strategy for Deep Reinforcement Learning

This paper proposes ITS-D3QN, a deep reinforcement learning scheduler utilizing a two-stage architecture and Pareto optimization strategy to effectively balance task response time, energy consumption, and resource utilization in dynamic cloud environments, achieving significant performance improvements over existing methods.

Original authors: Zhou Wenqin, Dai Lu, Wu Zhoulong, Hua Na

Published 2026-11-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhou Wenqin, Dai Lu, Wu Zhoulong, Hua Na

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, invisible infrastructure that powers the modern internet, data centers act as the beating hearts of our digital lives. These facilities house thousands of computers, or servers, that work around the clock to process everything from streaming movies to complex scientific calculations. However, these centers face a constant, chaotic challenge: millions of digital tasks arrive at random times, each demanding different amounts of computing power and memory. The system must decide instantly which task goes to which machine. If the decision is poor, some machines sit idle while others are overwhelmed, leading to slow response times, wasted electricity, and frustrated users. For decades, engineers have tried to solve this by creating rules that balance these competing needs, but the sheer speed and unpredictability of modern traffic have made simple rules insufficient.

Researchers at Tianshui Normal University in China have recently proposed a new way to manage this chaos, moving beyond static rules to a system that learns and adapts like a living organism. Their work focuses on a method called deep reinforcement learning, a type of artificial intelligence where a computer program learns to make decisions by trial and error, much like a child learning to ride a bicycle. Instead of following a fixed manual, the program interacts with the data center environment, trying different scheduling choices and receiving feedback on how well those choices performed. The goal is to find a "sweet spot" where the system finishes tasks quickly, uses as little energy as possible, and keeps all its machines busy at the same time. These goals often conflict; for instance, running a machine at full speed might finish a task faster but burn more electricity. The researchers' challenge was to teach the computer to navigate these trade-offs without getting stuck in a local optimum, a state where it settles for a "good enough" solution rather than the best possible one.

To tackle this, the team developed a new scheduling system they call ITS-D3QN. Imagine the data center as a busy airport with thousands of flights and limited gates. A traditional scheduler might assign planes to gates based on a simple list, but a smart scheduler would look at the weather, the fuel levels, and the arrival times of connecting flights to make a dynamic decision. The researchers' system works on two levels. The lower level acts as the immediate gate agent, deciding which specific task goes to which virtual machine right now. The upper level acts as the air traffic controller, watching the big picture of the entire cluster. This upper layer does not just look at individual tasks; it monitors the overall health of the system, measuring how well the current set of decisions covers the entire range of possible good outcomes. It uses a mathematical concept known as the hypervolume, which essentially measures the size of the "good solution space" the system has discovered so far. If the system is only finding mediocre solutions, the upper layer adjusts the priorities, telling the lower layer to try a different balance between speed and energy.

This two-layer approach allows the system to escape the traps that have plagued previous methods. Older artificial intelligence schedulers often relied on fixed weights, meaning they were programmed to care a certain percentage about speed and a certain percentage about energy, regardless of the situation. If the load changed, these fixed rules could not adapt. The new system, however, constantly re-evaluates these priorities. It uses a technique called incremental sampling to ensure its measurements of the system's performance are accurate and not just random noise. By running many small simulations in the background, it builds a reliable picture of how well its current strategy is working before making a change. This prevents the system from reacting wildly to temporary fluctuations in traffic, ensuring that its learning is stable and grounded in reality.

The researchers tested their system using real-world data from the Alibaba Cluster Trace, a massive dataset that records the actual activity of thousands of servers over several days. They simulated the arrival of tens of thousands of tasks to see how their system performed compared to older methods, including standard round-robin scheduling and other types of artificial intelligence. The results were clear: the new system consistently outperformed the others. Relative to the compared algorithms, it reduced average task response time by approximately 4.75% and cut the average energy used per task by about 4.32%. However, when examining specific load conditions in detail, the improvements varied; for instance, the reduction in response time ranged from 6.3% to 43.8% depending on the task volume. Perhaps most impressively, it increased the average utilization of the available resources by roughly 5.62%, meaning fewer machines were left sitting idle while others struggled. These improvements held true even as the number of tasks increased, showing that the system remained robust under heavy pressure.

The study suggests that by combining a high-level view of the system's overall performance with a low-level ability to make quick, specific decisions, cloud data centers can become significantly more efficient. The researchers found that the key to this success was not just in the speed of the decisions, but in the ability to see the trade-offs clearly. By using the hypervolume metric to guide the learning process, the system learned to find a balanced path through the complex landscape of competing objectives. While the results were achieved through computer simulations rather than a live deployment in a commercial data center, the consistency of the improvements across different scenarios indicates a strong potential for real-world application. The work points toward a future where cloud infrastructure can automatically adjust to the ebb and flow of digital demand, saving energy and speeding up services without human intervention. As the volume of data continues to grow, such adaptive intelligence may become essential for keeping the digital world running smoothly and sustainably.

Technical Summary: Research on Cloud Task Scheduling Problem Based on Pareto Optimization Strategy for Deep Reinforcement Learning

Problem Statement
Cloud data centers face increasing pressure from highly dynamic cluster loads and sharply heterogeneous task demands. Existing scheduling mechanisms often rely on local state features, leading to poor adaptability to load fluctuations and an inability to effectively balance competing objectives such as task completion time, energy consumption, and resource utilization. Traditional multi-objective approaches frequently decompose objectives into scalar sub-problems using fixed weighting or linear combinations, which can result in resource hoarding, node overloads, or idle capacity. Furthermore, existing Deep Reinforcement Learning (DRL) schedulers often suffer from limited state representations that fail to capture cluster-level information, causing agents to perceive the environment incompletely and perform poorly under complex, non-stationary load patterns.

Methodology: ITS-D3QN
The paper proposes ITS-D3QN (Interactive Two-Stage D3QN), a deep reinforcement learning scheduler built upon the Dueling Double Deep Q-Network (D3QN) architecture and driven by a Pareto optimization strategy. The method employs a two-layer interactive architecture:

Two-Layer Scheduling Model:
- Lower Layer (Execution Layer): Operates on a real-time task scale. It handles single-step task placement, mapping tasks to virtual machines (VMs) based on immediate state features (CPU, memory, queue status) and weight instructions from the upper layer. It optimizes for task completion time, system energy consumption, and resource utilization.
- Upper Layer (Management Layer): Operates on a macro time scale. It captures cluster-wide load and resource conditions using the Pareto Hypervolume (HV) as a core reward signal. Instead of fixed weights, it outputs a multi-objective preference weight vector that re-parameterizes the lower layer's reward function, guiding the system toward the global Pareto front.
Network Architecture (D3QN):
- Dueling Network: Decomposes the Q-function into a state value function $V(s)$ and an advantage function $A(s, a)$ to enhance sample utilization and convergence.
- Double Q-Network: Mitigates Q-value overestimation by decoupling action selection (using the online network) from action evaluation (using the target network).
- Action Masking: An action mask is applied to the feasible set of VMs to reduce ineffective exploration.
Pareto Optimization and Reward Mechanism:
- The scheduling problem is cast as a Markov Decision Process (MDP).
- The upper layer utilizes the Hypervolume (HV) indicator to quantify the quality of the non-dominated solution set. The HV increment ( $\Delta HV$ ) serves as a macro-reward signal, driving the policy to expand the dominated region in the objective space.
- Dynamic Preference Switching: To address dynamic loads, the system employs a mechanism that updates multi-objective weights every 200 task steps based on HV feedback and window-mean rewards. This allows the scheduler to shift operating points along the Pareto front (e.g., prioritizing utilization vs. energy conservation) rather than being stuck in a fixed trade-off.
Incremental Sampling Strategy:
- To address the noise in Monte Carlo HV estimation and the instability of preference switching, an incremental sampling strategy is introduced. The number of independent repeated samples ( $n^*$ ) is increased sequentially until the standard error of the mean falls below a specific accuracy threshold. This balances estimation stability with computational cost.

Key Contributions

Interactive Two-Stage Architecture: The paper introduces a decoupled framework where a macro-management layer dynamically adjusts objective weights for a micro-execution layer, enabling the system to adapt to non-stationary loads without manual weight tuning.
HV-Driven Pareto Optimization: Unlike methods that rely on scalarized rewards, this approach uses the Hypervolume increment as a direct reward signal to guide the evolution of the Pareto front, overcoming the tendency of conventional methods to settle in local optima.
Robustness to Dynamic Loads: The combination of dynamic preference switching and incremental sampling allows the scheduler to maintain stability and adaptability in highly dynamic environments where traditional DRL agents often oscillate or fail.

Experimental Results
The proposed method was evaluated using the Alibaba Cluster Trace v2018 dataset across varying task volumes (5,000 to 20,000 tasks). It was compared against non-DRL baselines (Round Robin, REINFORCE) and DRL baselines (DQN, D3QN).

Performance Metrics:
- Task Response Time: The abstract reports that relative to the compared algorithms, ITS-D3QN reduces average task response time by approximately 4.75%. Detailed controlled experiments in Section 3.2.2 further show that ITS-D3QN reduces response time by 6.3% to 43.8% relative to DRL baselines and by 29.1% to 59.7% relative to non-DRL baselines, with the advantage growing as load increases.
- Energy Consumption: The abstract reports that average energy consumption per task is reduced by approximately 4.32% relative to the compared algorithms. Detailed results indicate reductions of 9.0% to 15.1% relative to non-DRL baselines and 3.1% to 9.0% relative to DRL baselines.
- Resource Utilization: The abstract reports that average resource utilization increases by approximately 5.62% relative to the compared algorithms. Detailed results show increases of 6.9% to 36.4% over non-DRL baselines and 3.2% to 15.4% over DRL baselines.
Convergence and Stability: While standard D3QN converged quickly but plateaued due to fixed weights, ITS-D3QN demonstrated superior global search capabilities, escaping local optima and achieving higher cumulative rewards.
Ablation Studies: Experiments confirmed that both the two-layer dynamic weighting and the incremental sampling strategy contributed significantly to performance gains, with the dynamic weighting providing the most substantial improvement.

Significance and Claims
The paper claims that ITS-D3QN achieves balanced scheduling in complex cloud environments by effectively addressing the "Pareto bottleneck" and "policy oscillation" common in multi-objective DRL. By utilizing the Hypervolume indicator to drive weight adaptation, the method successfully balances throughput against energy costs in high-concurrency scenarios. The authors assert that the incremental sampling strategy successfully suppresses noise under highly dynamic loads, stabilizing the policy under complex constraints.

The work is presented as a step toward more adaptive cloud scheduling, with the authors noting that future extensions could apply this Pareto strategy to cloud-native environments involving delay jitter and link failures, as well as incorporating task priorities and Service Level Agreement (SLA) guarantees.

Technical Summary: Research on Cloud Task Scheduling Problem Based on Pareto Optimization Strategy for Deep Reinforcement Learning

More like this