💻 computer science

ER-Curriculum-DDQN for Critical Task Offloading in UAV-MEC via Transparent LEO Relays

This paper proposes ER-Curriculum-DDQN, a deep reinforcement learning framework that integrates feasibility masking, critical task pressure features, and reservation-guided curriculum learning to optimize online task offloading in UAV-MEC systems via transparent LEO relays, thereby maximizing the timely completion ratio of critical tasks under strict service window constraints.

Original authors: Ziang Zhang, Zhenjiang Zhang, Jiaxing Du, Zhaoyuan Liu, Yujie Wang

Published 2026-09-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziang Zhang, Zhenjiang Zhang, Jiaxing Du, Zhaoyuan Liu, Yujie Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, silent stretches of the sky where the ground's infrastructure fades away, a new kind of network is taking shape. Imagine a fleet of drones, or unmanned aerial vehicles, hovering over a disaster zone or a remote mountain range, carrying the weight of urgent data from sensors and cameras. These drones act as flying computers, but they have limits. To solve complex problems, they need to send their heavy data to powerful servers on the ground or in the clouds. When the ground is too far or the roads are blocked, these drones look up to the stars for help. They connect to low-orbit satellites that act as transparent mirrors, simply bouncing signals between the drone and a distant gateway without stopping to process the information themselves. This creates a fleeting bridge, a narrow window of time where the drone, the satellite, and the ground station are all aligned. The challenge is that this window is short and unforgiving. If a drone tries to send a massive, non-urgent file through this bridge, it might lock up the entire connection for the duration of the transaction. If a critical emergency message arrives while that bridge is occupied, it cannot be sent until the first task is finished, potentially missing a life-saving deadline. The core question for scientists is how to manage this scarce, time-limited connection so that the most important jobs always get through, even when the system is crowded.

Researchers at Beijing Jiaotong University and the Army Arms University of the PLA have tackled this problem by designing a smart decision-making system for these flying networks. They focused on a scenario where twenty drones are working together, sharing two ground-based computing centers and two satellite paths. The system must decide, the moment a new task arrives, whether to process it locally on the drone, send it to a nearby ground server, or forward it through the satellite. The catch is that the satellite path operates on a strict "first-come, first-served" rule that cannot be interrupted once started. If a routine task is admitted to the satellite path, it reserves the entire connection for its journey, blocking any other task, no matter how urgent, until the round trip is complete. The researchers needed a way to predict when to say "no" to a routine task to save the satellite path for a future emergency, all without knowing what tasks would arrive next.

To solve this, the team developed a new method called ER-Curriculum-DDQN. This is a type of artificial intelligence that learns by trial and error, but with a specific set of rules to keep it grounded in reality. The system uses a "feasibility mask," which acts like a filter that instantly removes any impossible choices. For example, if the satellite window is closed or the local drone memory is full, the system simply cannot consider those options. This prevents the AI from wasting time thinking about actions that would fail due to physical constraints. Beyond just knowing what is possible, the system tracks a "pressure" feature. This is a measure of how much demand there has been for the satellite path from critical tasks in the recent past. If the system senses that critical tasks are arriving frequently, it becomes more cautious about letting routine tasks use the satellite, effectively saving the bridge for the emergencies that might come next.

The learning process itself was guided by a "curriculum," a teaching strategy that starts simple and gets harder. In the early stages of training, the AI was explicitly told to feel a penalty whenever it let a routine task use the satellite path if that action increased the pressure on the system. This taught the AI the value of holding back resources. As the training progressed, this explicit penalty was gradually faded away, forcing the AI to internalize the lesson and make the right decisions based on the patterns it had learned, rather than following a simple rule. The goal was not just to get tasks done, but to ensure that the most critical ones were finished on time.

The researchers tested their system through extensive computer simulations, pitting it against other known methods and simple rule-based approaches. They varied the conditions, changing how many tasks arrived per second, how long the satellite windows lasted, and how many of the tasks were critical emergencies. In every scenario, especially when the system was under heavy load or the satellite windows were very short, the new method outperformed the others. It achieved the highest rate of on-time completion for critical tasks. For instance, when the number of incoming tasks was high and the satellite windows were tight, the new system managed to complete critical tasks about 69% of the time, while other methods struggled significantly lower. The results showed that by combining a strict filter for impossible moves with a learned sense of future demand, the system could protect the most important data without needing to know the future.

The study confirms that in these high-stakes, time-sensitive environments, the best strategy is not just to react to what is happening right now, but to understand the scarcity of the connection itself. The researchers found that simply prioritizing critical tasks by giving them higher weights was not enough; the system had to understand the cost of occupying the satellite path with a routine task. By using a learning approach that respects the physical limits of the network and learns from the history of demand, the drones can make smarter choices. This work does not claim to have solved every problem in space communications, nor does it promise to work in every possible future scenario, but it demonstrates a clear and effective way to manage the delicate balance between routine work and urgent needs in a network where time is the most valuable resource.

Technical Summary: ER-Curriculum-DDQN for Critical Task Offloading in UAV-MEC via Transparent LEO Relays

Problem Statement
This research addresses the challenge of online task offloading in Unmanned Aerial Vehicle (UAV)-enabled Mobile Edge Computing (UAV-MEC) systems assisted by transparent Low Earth Orbit (LEO) relays. In scenarios where terrestrial edge coverage is limited, transparent LEO relays connect UAVs to remote cloud resources. However, these connections are constrained by short, time-varying joint service windows between the relay and the gateway.

The core difficulty lies in the atomicity and non-preemptive nature of these service windows. A task admitted to a satellite (SAT) path must complete a full request-response transaction (transmission, forwarding, remote processing, result return, and guard intervals) within a single contiguous window. Once a non-critical task is admitted, it locks the path and reserves the necessary contiguous capacity under First-In-First-Out (FIFO) semantics. This creates a critical tradeoff: admitting a non-critical task may consume the limited service capacity required by a future, time-sensitive Critical task, which cannot preempt the existing reservation. The system operates under uncertainty, with no prior knowledge of future task arrivals or window availability, necessitating an online decision policy that prioritizes the timely completion of Critical tasks without relying on future predictions.

Methodology: ER-Curriculum-DDQN
The authors propose ER-Curriculum-DDQN, a reinforcement learning framework designed to solve this masked Markov Decision Process (MDP). The methodology integrates four key components:

Path-Locked FIFO Reservation Model: The system models SAT services with hard feasibility conditions. A task is only admitted to a SAT path if the complete transaction can be guaranteed to finish before the current joint window closes, considering the "tail" of existing reservations.
State Representation and Feasibility Masking: The state vector includes task attributes, UAV local states, shared MEC queue states, SAT window/reservation states, and a novel Critical SAT-demand pressure feature. An atomic feasibility mask is applied to the action space (LOCAL, two MEC options, two SAT paths, DROP). This mask structurally eliminates infeasible actions (e.g., selecting a SAT path if the window is too short for the transaction) based solely on current causal state, ensuring the agent never selects physically impossible actions.
Critical Demand Pressure Feature: To anticipate resource scarcity without predicting the future, the agent utilizes a pressure feature ( $q_s$ ). This metric estimates the ratio of expected Critical-task demand (based on historical arrival intensity and transaction duration) to the currently available contiguous capacity on a SAT path. This allows the agent to learn the "opportunity cost" of admitting non-critical tasks.
Reservation-Guided Curriculum: The training process employs a curriculum learning strategy. Early in training, a penalty term is added to the reward function for non-critical tasks that increase the pressure on SAT paths. This coefficient is annealed (gradually reduced) over training episodes. This guides the Double DQN to initially learn the tradeoff between admitting non-critical tasks and protecting capacity for future Critical tasks, eventually relying on the learned Q-values and causal state features.

The algorithm utilizes a Double DQN backbone to mitigate value overestimation, with the online network selecting actions and the target network evaluating them, all constrained by the feasibility mask.

Key Contributions
The paper outlines three primary contributions:

System Modeling: Development of a path-locked FIFO reservation model for transparent LEO relay offloading, treating coverage, queue capacity, and complete transaction fulfillment as hard feasibility constraints.
Algorithm Design: Proposal of the ER-Curriculum-DDQN algorithm, which uniquely combines an atomic feasibility mask, a causal Critical-demand pressure feature, and an annealed reservation curriculum to protect future Critical services.
Performance Validation: Demonstration through extensive simulations that the proposed method achieves superior or tied-best performance in Critical-task Timely Completion Ratio (Critical-TCR) compared to baselines, particularly in high-load, short-window, and high-criticality scenarios.

Experimental Results
The study evaluates the method against several baselines, including ER-Curriculum-PPO, Masked DDQN-Base, and rule-based approaches (WRAO, EFT-Greedy) across varying task arrival rates (8–24 tasks/s), joint service window durations (2.5–6.0 s), and Critical-task ratios (0.05–0.35).

Critical-TCR Performance: ER-Curriculum-DDQN consistently achieved the highest or tied-best Critical-TCR across all tested conditions. For instance, under a high arrival rate of 24 tasks/s, it maintained a Critical-TCR of approximately 0.69, outperforming other methods.
Robustness: The method demonstrated stronger protection for Critical tasks in high-load and short-window scenarios compared to PPO-based and rule-based methods.
Ablation Insights: Results indicated that the Double DQN backbone combined with the feasibility mask provided the primary performance gains, while the pressure-aware design and curriculum added robustness.
Trade-offs: While the method excelled in Critical-task satisfaction, the paper notes a tradeoff where the Overall-TCR (completion of all task classes) might be lower than some rule-based methods that prioritize total throughput over priority protection.

Significance and Scope
The paper claims significance in addressing the specific constraints of transparent relay-assisted UAV-MEC, where the atomicity of request-response transactions within joint windows creates a unique resource contention problem distinct from standard edge computing or onboard computing scenarios.

The authors maintain a modest scope, clarifying that:

The improvement is specific to Critical-task deadline satisfaction, not necessarily overall throughput, latency, or energy efficiency for all task classes.
The solution relies on causal, online information and does not predict future task arrivals or window availability.
The current evaluation is limited to controlled joint windows and a fixed discrete action space.
Future work is identified as necessary to address realistic visibility traces, multi-relay/multi-cloud systems, continuous resource allocation, and task partitioning.

The study concludes that integrating structural feasibility masking with a pressure-aware curriculum in a Double DQN framework effectively learns the long-term tradeoff between admitting non-critical tasks and reserving capacity for critical services in dynamic aerial-terrestrial networks.

Technical Summary: ER-Curriculum-DDQN for Critical Task Offloading in UAV-MEC via Transparent LEO Relays

More like this