🤖 machine learning

Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

This paper proposes a curriculum-guided adaptation framework for reinforcement learning that aligns temporal-difference error distributions across increasing adversarial intensities to ensure robust UAV deconfliction and bounded performance degradation under unseen GNSS spoofing attacks.

Original authors: Deepak Kumar Panda, Adolfo Perrusquia, Weisi Guo

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Deepak Kumar Panda, Adolfo Perrusquia, Weisi Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Flying Drones in a Foggy, Deceptive World

Imagine you are teaching a drone to fly through a busy city, dodging buildings and other drones to get to a specific destination. You use a "brain" (an AI) that learns by trial and error, much like a student learning to ride a bike. This is called Reinforcement Learning (RL).

However, there is a problem: GPS Spoofing.

Think of GPS spoofing as a "magic trick" played by a hacker. The hacker doesn't break the drone; instead, they send fake signals that make the drone think it is in a different location than it actually is.

The Reality: The drone is safe, 100 feet away from a building.
The Fake Signal: The drone's "brain" thinks it is crashing into the building right now.

When this happens, the drone panics, makes bad decisions, and might crash. This is called a distribution shift—the world the drone sees (the fake data) is completely different from the world it trained in (the real data).

The Problem with Current Solutions

Current methods try to make the drone "tough" by training it against specific types of tricks.

The Analogy: Imagine training a boxer only against a left-handed puncher. If the boxer learns to block that one specific punch perfectly, they will be great. But if a right-handed puncher shows up, or someone uses a kick, the boxer might get knocked out.
The Paper's Critique: Existing AI safety methods are like that boxer. They are too focused on specific attacks they've seen before. If a new, unseen type of GPS trick happens, the AI forgets everything it learned and fails catastrophically.

The Solution: A "Graduated" Training Camp

The authors propose a new way to train the drone called Curriculum-Adapted Robust Reinforcement Learning.

Think of this as a martial arts dojo with a very specific training method:

The "Expert" Sensei: First, they train a very strong, basic drone pilot (the "Expert") that is already good at dodging.
The Graduated Curriculum: Instead of throwing the drone into a chaotic battle immediately, they introduce "fake attacks" (adversarial perturbations) slowly.
- Level 1: The fake signals are very weak. The drone barely notices.
- Level 2: The fake signals get slightly stronger. The drone has to adjust.
- Level 3: The signals get even stronger.
- The Goal: The drone moves up these levels one by one, getting tougher at each stage.

The Secret Sauce: Listening to the "Inner Voice" (TD-Error)

Here is the clever part. Usually, when we train AI, we look at the input (what the drone sees). But this paper says: "Don't just look at the eyes; look at the brain's confidence."

In AI terms, this is called the TD-Error (Temporal-Difference error).

The Analogy: Imagine you are walking in the dark. Your "inner voice" (the AI's value estimation) says, "I think I'm safe, but I'm a little unsure."
The Trick: When the hacker sends a fake signal, the drone's "inner voice" starts screaming, "I'm confused! I don't know where I am!" This confusion is the TD-error.

The authors' method forces the drone to keep its "inner voice" calm and consistent as it moves through the training levels.

Even if the fake signals get stronger, the drone is trained to ensure its internal calculation of "how good is this move?" doesn't change wildly.
By keeping this internal confidence stable, the drone learns a general rule: "No matter what weird signals I get, trust my core logic."

The Result: Beating the Unseen

The paper tested this in a simulation where drones had to fly through 3D obstacles. They tested it against two types of GPS tricks they never saw during training:

Fixed Spoofing: A constant, steady lie about where the drone is.
Dynamic Spoofing: A sneaky lie that changes based on where obstacles are, trying to lure the drone into a trap.

The Results:

Standard AI: Failed miserably. It crashed or got lost (Success rate: 20–56%).
The New Method: Almost never failed (Success rate: Near 100%).
Efficiency: Even when the drone had to dodge, it did so much faster and with fewer steps than the others.

Why This Matters (According to the Paper)

The paper claims that by training the drone to keep its internal confidence (TD-error) consistent while facing gradually stronger lies, the drone learns a "superpower." It doesn't just learn to fight one specific liar; it learns how to ignore any liar, even ones it has never met before.

In short: Instead of memorizing the answers to every possible test question, the drone learned how to stay calm and think clearly, no matter how confusing the test questions became. This allows it to fly safely even when hackers try to trick its GPS.

Technical Summary: Curriculum-Adapted Robust Reinforcement Learning for UAV Deconfliction in Adversarial Environments

Problem Statement
Autonomous Unmanned Aerial Vehicles (UAVs) increasingly rely on Reinforcement Learning (RL) for navigation and deconfliction in complex airspace. However, these systems face significant security vulnerabilities, particularly Global Navigation Satellite System (GNSS) spoofing attacks. Such attacks induce out-of-distribution observation shifts that corrupt value estimation, leading to catastrophic forgetting and unsafe behaviors. Existing robust RL approaches typically improve resilience against specific, predefined attack models but fail to generalize to unseen attack mechanisms. Furthermore, current defense strategies often rely on detection-centric methods or specific sensor redundancies, neglecting the need for decision-making policies that can adapt to adversarially induced distribution shifts without compromising mission efficiency.

Methodology
The authors propose a curriculum-guided adaptation framework that operates in the Temporal-Difference (TD) error distribution space rather than the raw observation space. The core hypothesis is that while attacks may differ in how they corrupt sensor inputs, their ultimate impact on policy behavior is reflected in the value distribution space.

The framework proceeds through four main steps:

Expert Critic Formulation: An initial robust policy is trained using a two-player zero-sum game (DDPG) where an agent and an adversary compete in the action space. This yields an "expert critic" ( $Q_{AR}$ ) that is resilient to action-space perturbations.
Adversarial Curriculum Generation: The system employs a curriculum strategy where the policy is progressively exposed to gradient-based adversarial observation perturbations of increasing intensity ( $\epsilon$ ). These perturbations are generated via multi-step projected gradient ascent to maximize policy loss within a bounded norm.
TD-Space Adaptation: During adaptation, the framework aligns the TD-error distributions between the current stage and the previous stage. The critic from the previous stage acts as a reference (expert), and the adaptation minimizes the Wasserstein-1 distance between the TD-error distributions of the nominal and adversarial observations. This alignment is designed to prevent catastrophic forgetting and value-function drift.
Generalization Certificate: The authors derive a theoretical "TD-space generalization certificate." This theorem establishes that if the TD-error distribution induced by a test-time attack remains sufficiently close to that of the final curriculum stage, the resulting performance degradation is bounded. This provides a formal guarantee that robustness learned under synthetic perturbations can transfer to unseen cyber-physical attacks.

Key Contributions

TD-Space Curriculum Adaptation: The paper introduces a novel adaptation strategy that enforces consistency in TD-error distributions across curriculum stages. Unlike conventional robust RL that assumes fixed uncertainty sets, this approach adapts to increasing perturbation strengths while bounding the drift in value estimates.
Theoretical Generalization Guarantee: The authors derive a formal certificate showing that robustness transfers across fundamentally different attack domains (e.g., from synthetic gradient perturbations to real GNSS spoofing) provided the induced TD-error distribution shift remains bounded. This addresses the gap where existing methods lack theoretical guarantees for unseen attack mechanisms.
Empirical Validation in UAV Deconfliction: The framework is validated in a 3D UAV deconfliction environment against both fixed and dynamic GNSS spoofing attacks. The study demonstrates that the curriculum-adapted policy maintains high mission success rates and navigation efficiency under attack conditions not encountered during training.

Results
Experiments were conducted in a simulated environment with dynamic 3D obstacles and varying aerial traffic densities.

Fixed Spoofing Attacks: Under fixed GNSS spoofing (constant position bias), the curriculum-adapted policy achieved near-perfect mission success rates. In contrast, standard DDPG and other robust RL baselines (including adversarial meta-learning and NR-MDP) achieved success rates between 20% and 56%, often failing to reach the destination within the flight budget.
Dynamic Spoofing Attacks: Against dynamic, obstacle-luring spoofing attacks (where the bias is adjusted to distort obstacle perception), the proposed method achieved the highest episodic rewards. It reduced mission completion steps by up to 45% compared to baselines across increasing traffic densities.
Perception-Reality Divergence: The curriculum-adapted policy exhibited the smallest divergence between perceived and true obstacle clearance, indicating a superior ability to maintain accurate internal risk representations despite corrupted observations.
Ablation and Sensitivity: The study identified $\epsilon = 4.0$ as the optimal curriculum stopping point, balancing robustness and performance. Furthermore, smaller curriculum step sizes ( $\Delta\epsilon = 0.25$ ) were found to produce smoother TD-alignment and higher stability compared to larger steps, which induced mode-collapse-like behaviors.

Significance and Claims
The paper claims that its primary significance lies in demonstrating that adversarial curriculum adaptation can transfer resilience from synthetic perturbations to realistic GNSS spoofing attacks without attack-specific retraining. By focusing on preserving TD-error distribution consistency rather than fitting to a specific attack model, the framework provides a principled mechanism for improving the resilience of autonomous decision-making systems under unknown cyber-physical threats. The authors assert that this approach moves beyond conventional robust RL by offering both empirical improvements in mission performance and provable theoretical bounds on generalization to unseen attack domains. The work suggests that maintaining value-function consistency is a critical factor in preventing catastrophic forgetting in adversarial environments.