💻 computer science

Joint Scheduling and Resource Allocation in Heterogeneous Queuing Systems with Bursty Traffic: A Constrained Soft Actor-Critic Approach

This paper proposes a Constrained Soft Actor-Critic (CSAC) approach that decouples stringent delay constraints from the reward function and employs a two-stage mapping mechanism to effectively maximize throughput utility while minimizing delay violations in heterogeneous queuing systems with bursty traffic, outperforming both unconstrained learning and heuristic baselines.

Original authors: Ao Fang, Jianyu Cao, Weihua Qian

Published 2026-09-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ao Fang, Jianyu Cao, Weihua Qian

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the invisible highways of modern communication, data travels in packets, waiting in lines to cross a shared bridge. This bridge, a network link, has a limit on how much it can carry at any single moment. When too many packets arrive at once, they must queue up, and if the line grows too long or waits too long, the data becomes useless. This is the fundamental problem of network scheduling: how to decide which packet gets to cross the bridge first when multiple lines are competing for space. The challenge intensifies when the traffic is unpredictable, surging in sudden bursts, and when different types of data have different needs. Some data, like a live video call or an emergency alarm, cannot tolerate even a moment of delay, while other data, like a file download, can wait a little longer. The goal for network engineers is to keep the bridge busy enough to be efficient, yet organized enough to ensure the most urgent messages never get stuck in a long line.

Researchers at the Guilin University of Electronic Technology have tackled this problem by designing a new way to manage these competing lines of data. They focused on a system where several queues, each with its own priority level, share a single output link. In their setup, one queue carries a special kind of traffic that arrives in unpredictable bursts and has a strict rule: its packets must not wait longer than a specific amount of time. The other queues carry less urgent traffic that can wait longer, but the system still needs to keep them moving. The difficulty lies in the fact that the link's capacity changes constantly, and the bursty traffic can overwhelm the system in an instant. Traditional methods, which rely on fixed rules, often fail to react quickly enough to these sudden changes. Meanwhile, newer methods that use artificial intelligence to learn how to manage the traffic often make a dangerous trade-off: they might let the urgent packets wait too long just to keep the overall system moving faster.

To solve this, the team developed a new approach based on a type of artificial intelligence called a constrained soft actor-critic. Instead of simply telling the AI to maximize the total amount of data sent, they gave it a strict, separate budget for how often the urgent queue could violate its time limit. Think of this as giving a driver a strict rule about how many times they can run a red light, separate from their goal of getting to the destination quickly. The AI learns to balance the need for speed with the hard limit on delays. A key part of their design is a two-step process that translates the AI's continuous, fluid decisions into concrete, whole numbers of packets to send. This ensures the system can actually execute the plan without getting stuck in theoretical calculations. The researchers tested this method in a simulated environment that mimicked real-world conditions, including a high-priority queue with bursty traffic and two lower-priority queues with steady traffic.

The results showed a clear advantage for the new method. In their simulations, the traditional fixed-rule methods failed to protect the urgent traffic, allowing the delay limit to be broken nearly 18 percent of the time with one method and over 34 percent with another. Even a standard AI approach without the strict constraint broke the rule almost 8 percent of the time. In contrast, the new constrained approach kept the violation rate for the urgent queue down to a tiny fraction, averaging just 0.05 percent, with very little variation between different test runs. Crucially, this strict protection did not come at the cost of the other traffic. The system maintained high efficiency, sending almost the same amount of data as the other methods, and it actually reduced the number of packets that were dropped because the lines were too full. The lower-priority queues also experienced shorter wait times compared to the standard AI method.

The study demonstrates that by separating the strict safety rules from the general goal of efficiency, an AI system can learn to manage complex, unpredictable traffic much more effectively than previous methods. The researchers found that their approach successfully prevented the most critical data from being delayed while keeping the entire system running smoothly. This suggests that for networks handling mixed traffic, where some data is life-critical and some is not, using a constrained learning model is a viable path forward. The work confirms that it is possible to have both high speed and strict reliability, provided the system is designed to treat the safety limits as a separate, non-negotiable budget rather than just another factor to balance.

Technical Summary: Joint Scheduling and Resource Allocation in Heterogeneous Queuing Systems with Bursty Traffic

Problem Statement
The paper addresses the challenge of joint scheduling and resource allocation in heterogeneous queuing systems where multiple queues with distinct priorities and delay requirements compete for a single, time-varying output link. The system is characterized by stochastic packet arrivals (including bursty traffic modeled by Markov-modulated Poisson processes) and fluctuating link capacity. The core objective is to maximize system throughput utility while strictly adhering to per-queue delay constraints.

A fundamental tension exists in this problem: prioritizing high-priority traffic reduces its delay but risks starving lower-priority queues, whereas throughput-oriented scheduling may allow high-priority packets to exceed their maximum allowable delay. Conventional heuristic methods (e.g., fixed-weight or backlog-driven rules) lack real-time reactivity to abrupt traffic mode transitions or link capacity fluctuations. Conversely, unconstrained learning-based methods that encode delay requirements into a scalar reward often trade deadline compliance for higher cumulative rewards, failing to guarantee strict delay bounds during burst periods.

Methodology: Constrained Soft Actor-Critic (CSAC)
The authors propose a Constrained Soft Actor-Critic (CSAC) approach, specifically tailored for heterogeneous queuing systems (CSAC-HQS), to address these limitations. The methodology is built on two primary design innovations:

Decoupled Constraint Formulation:
Unlike standard approaches that fold delay violations into the reward function with hand-tuned weights, the proposed method treats the delay constraint of the highest-priority queue as a standalone constraint.
- Reward Function: The reward focuses on maximizing throughput utility and penalizing violations for lower-priority (best-effort) queues and packet drops.
- Constraint Signal: The delay violation of the high-priority queue is treated as a cost signal ( $c_t$ ) separate from the reward. The policy is optimized to satisfy a long-term violation budget ( $J_C^{ep} \leq d_c$ ) using a Lagrangian multiplier ( $\lambda$ ).
- Architecture: The system employs a twin reward critic network to estimate cumulative rewards and a separate constraint critic network to estimate the expected cumulative delay violation risk. The actor network is updated using a Lagrangian loss that balances reward maximization against the constraint penalty.
Two-Stage Continuous-to-Discrete Action Mapping:
Since the underlying Soft Actor-Critic (SAC) algorithm operates in continuous action spaces, but the queuing system requires integer transmission quotas, a specific mapping mechanism is introduced:
- Stage 1 (Logits to Weights): The actor outputs continuous logits, which are sharpened by a factor $\beta$ and passed through a masked-softmax function to generate normalized allocation weights. The masking ensures empty queues receive zero weight.
- Stage 2 (Weights to Quotas): These weights are converted into integer transmission quotas. The process involves calculating ideal allocations, flooring them, and distributing residual capacity based on the largest remainder rule, prioritized by queue index. A recursive clipping mechanism ensures quotas never exceed the current backlog of any queue.

Experimental Setup
The approach was evaluated in a system with three queues:

Queue 0: Highest priority, carrying bursty traffic (MMPP) with a stringent delay violation requirement (maximum 1% violation rate).
Queues 1 & 2: Lower priority, carrying independent Poisson traffic with best-effort delay requirements.
Baselines: The proposed method was compared against:
- Static: A fixed-weight round-robin heuristic.
- Dynamic: A heuristic allocating resources proportional to instantaneous queue lengths.
- Unconstrained SAC: A standard SAC baseline without explicit constraint handling (delay penalties included in the scalar reward).

Key Results
The experimental results demonstrate significant improvements in meeting strict delay constraints without sacrificing overall system performance:

Delay Violation Rate (Queue 0): The proposed CSAC-HQS (with sharpening factor $\beta=5$ ) reduced the delay violation rate to 0.05% ± 0.09%, well within the 1% target. In contrast, the unconstrained SAC achieved 7.87% ± 5.98%, while the Static and Dynamic heuristics resulted in 18.38% and 34.28% respectively. While CSAC-HQS ( $\beta=1$ ) achieved a mean violation rate of 0.91%, its standard deviation of 1.30% results in an upper bound exceeding the 1% target; only the $\beta=5$ variant consistently satisfied the constraint across all random seeds (mean + std < 1.0%).
Stability: The CSAC approach showed significantly lower variance across random seeds compared to the unconstrained SAC, indicating more stable policy optimization.
Throughput and Lower-Priority Performance: The proposed method maintained throughput levels comparable to the unconstrained SAC and heuristics. It also achieved lower average queuing delays for the best-effort queues (1 and 2) compared to the unconstrained SAC.
Packet Loss: The CSAC approach reduced overflow packet counts for lower-priority queues compared to the unconstrained SAC and the Static baseline.

Significance and Claims
The paper claims that the proposed CSAC approach effectively resolves the trade-off between maximizing throughput and satisfying stringent delay constraints in heterogeneous, bursty environments. The significance lies in:

Explicit Constraint Handling: By decoupling the high-priority delay constraint from the reward, the method ensures that deadline compliance is not sacrificed for aggregate reward maximization, a common failure mode in unconstrained reinforcement learning.
Action Discretization: The two-stage mapping mechanism allows the use of continuous policy networks (SAC) to directly execute discrete, integer-based scheduling decisions without post-processing artifacts, ensuring physical realizability.
Robustness: The approach demonstrates robustness against stochastic traffic bursts and link capacity fluctuations, outperforming both fixed heuristics and standard learning-based baselines in maintaining Quality of Service (QoS) guarantees.

The authors conclude that this framework provides a viable solution for differentiated QoS in complex network environments such as 5G network slicing and industrial IoT, where bursty traffic and strict latency requirements coexist. Future work is suggested to extend this to scenarios with multiple bursty queues and to integrate admission control.

Technical Summary: Joint Scheduling and Resource Allocation in Heterogeneous Queuing Systems with Bursty Traffic

More like this