Deep deterministic policy gradient with symmetric data augmentation for lateral attitude tracking control of a fixed-wing aircraft

This paper proposes a sample-efficient offline reinforcement learning approach for fixed-wing aircraft lateral attitude tracking that leverages system symmetry to augment training data and employs a dual-critic Deep Deterministic Policy Gradient (DDPG) structure to accelerate policy convergence.

The Gist

Imagine you are trying to teach a robot pilot how to fly a plane. The plane is tricky; if you turn the stick to the left, it banks left. If you turn it to the right, it banks right. The physics are perfectly symmetric: the world looks the same whether you are flying "up" or "down" relative to the horizon, just mirrored.

Usually, to teach a robot this, you let it crash and try again thousands of times (a process called Reinforcement Learning). But flying a real plane is expensive, and crashing is dangerous. So, we use a simulator. Even in a simulator, the robot has to "explore" the sky to learn. It might spend hours flying only in the northern sky, never realizing that flying south is just the mirror image of what it already knows. This is inefficient.

This paper proposes a clever trick to make the robot learn twice as fast without flying twice as much. Here is how it works, broken down into simple concepts:

1. The "Mirror World" Trick (Symmetric Data Augmentation)

Imagine you are learning to juggle. You practice juggling three balls with your right hand. You get pretty good at it.
Now, imagine you have a magical mirror. If you look in the mirror, you see yourself juggling with your left hand. You didn't actually do the left-hand juggling, but your brain knows exactly how it would feel because the physics are symmetrical.

In this paper, the researchers do the same thing with flight data.

• The Problem: The robot flies a simulation, collects data (State A + Action A = Result B), and stores it.
• The Trick: The computer takes that single piece of data and creates a "mirror image" of it instantly. If the robot flew a bank angle of +10 degrees with a rudder input of +5, the computer creates a fake data point: "Bank angle -10 degrees with rudder input -5."
• The Result: The robot thinks it has flown both scenarios. It learns the "left" side of the sky by studying the "right" side. This doubles the amount of training data without the robot ever having to actually fly the extra miles.

2. The "Two-Coach" System (Dual-Critic Structure)

In Reinforcement Learning, there are usually two main characters:

• The Actor: The pilot (the brain that decides what to do).
• The Critic: The coach (the brain that says, "Good job!" or "That was a bad move").

The authors realized that if you mix the "real" flight data with the "mirror" data in one big bucket, the coach gets confused. It's like a coach trying to grade a student's homework while simultaneously grading a photocopy of that homework. The signal gets muddy.

Their Solution: They built two separate coaches.

• Coach 1 only looks at the real flight data the robot actually experienced.
• Coach 2 only looks at the mirror (augmented) data.

The robot (the Actor) listens to both coaches. Coach 1 says, "You did well here in the real world." Coach 2 says, "And based on the mirror world, you would have done well there too." By separating the lessons, the robot learns more efficiently because it isn't getting mixed signals.

3. The "Two-Step" Dance (Two-Step Approximate Value Iteration)

Instead of having the robot take one step forward, look at the data, and stop, they made it take a two-step dance:

1. Step 1: The robot practices on the real data. It updates its pilot skills based on what actually happened.
2. Step 2: Immediately after, the robot practices on the mirror data. It updates its pilot skills again, this time using the "what if" scenarios.

This allows the robot to refine its strategy twice as often in the same amount of time, leading to much faster learning.

4. The Smoothness Rule (CAPS)

There's one more problem: Robots can be jerky. They might jerk the controls violently because they are over-analyzing.
The authors added a rule called Conditioning for Action Policy Smoothness (CAPS). Think of this as a "smooth driving" penalty. If the robot makes a sudden, jerky move, it gets a "frown" (a penalty). If it makes a smooth, gradual turn, it gets a "thumbs up." This ensures the final flight control isn't just accurate, but also comfortable and safe, just like a human pilot.

The Big Picture: Why Does This Matter?

The researchers tested this on a fixed-wing aircraft model.

• Without the trick: The robot had to fly a lot to learn how to handle turns in both directions. It struggled when asked to fly in a direction it hadn't explored much.
• With the trick: The robot learned much faster. Because it "imagined" the mirror world, it could handle turns in the negative direction (which it never actually flew) just as well as the positive direction.

In summary: This paper teaches us that we don't always need to experience everything to learn it. By understanding the symmetry of the world (that left is just the mirror of right), we can create a "virtual training ground" that doubles our learning speed, saves computational power, and makes AI pilots smarter and safer.

Technical Summary

1. Problem Statement

Reinforcement Learning (RL) has shown promise in flight control, particularly for reducing reliance on accurate aerodynamic models. However, a major bottleneck in applying RL to complex, high-dimensional systems like fixed-wing aircraft is sample efficiency.

• Exploration vs. Exploitation: As the control policy converges, the agent explores less, leading to insufficient coverage of the state-action space. This limits the generalization of the learned policy to unvisited regions.
• Cost of Exploration: Generating data through physical interaction or high-fidelity simulation is computationally expensive and time-consuming.
• Generalization Gap: Policies trained on limited, asymmetric datasets often fail to generalize to symmetric states (e.g., negative bank angles) if those specific states were not explicitly explored during training.

The paper addresses how to leverage the inherent geometric symmetry of aircraft dynamics to generate additional training data without requiring further environment interaction, thereby accelerating policy convergence and improving robustness.

2. Methodology

The proposed approach integrates Symmetric Data Augmentation (SDA) into the Deep Deterministic Policy Gradient (DDPG) framework. The methodology consists of three main pillars:

A. Theoretical Foundation: Symmetry in MDPs

The authors define a symmetric dynamical system where state transitions and rewards exhibit symmetry with respect to a reference plane (typically the origin or a specific equilibrium point).

• Symmetric States: Two states $x_t$ and $x'_t$ are symmetric if $x_t + x'_t = 2x^*$ (where $x^*$ is the reference state).
• Symmetric Actions: Corresponding actions satisfy $a_t = -a'_t$.
• Symmetric Transitions: If the system dynamics $F(x)$ and $G(x)$ satisfy specific conditions (e.g., $F(x) = F(x')$ and $G(x) = G(x')$), then the resulting next states $x_{t+1}$ and $x'_{t+1}$ will also be symmetric.
• Reward Symmetry: The reward function is assumed to be invariant under these symmetric transformations.

B. Symmetric Data Augmentation (SDA)

Based on the symmetry conditions, the method generates "augmented" samples from "explored" samples:

• For every collected transition $(x_t, a_t, x_{t+1}, r_t)$, a symmetric counterpart is computed:
  • $x'_t = 2x^* - x_t$
  • $a'_t = -a_t$
  • $x'_{t+1} = 2x^* - x_{t+1}$
  • $r'_t = r_t$
• This effectively doubles the dataset size and fills in symmetric regions of the state space that the exploration policy (e.g., Ornstein-Uhlenbeck noise) might have missed.

C. DDPG with Symmetric Critic Augmentation (DDPG-SCA)

The paper identifies a drawback in simply mixing augmented and explored samples in a single batch (DDPG-SDA), as this dilutes the gradient updates from the original data. To solve this, they propose a Two-Step Approximate Value Iteration method with Dual Critics:

1. Dual Replay Buffers: Explored samples are stored in buffer $D_1$, and augmented samples in buffer $D_2$.
2. Dual Critics: Two separate critic networks ($Q_1$ and $Q_2$) are trained.
   • Step 1: Train $Q_1$ and update the Actor ($\mu$) using a minibatch from $D_1$ (explored data).
   • Step 2: Train $Q_2$ and update the same Actor ($\mu$) using a minibatch from $D_2$ (augmented data).
3. Benefit: The actor receives gradient updates from both the real exploration data and the symmetric "imagined" data in every iteration, maximizing sample utilization without increasing the batch size.

D. Action Smoothness

To ensure stable control, the authors incorporate Conditioning for Action Policy Smoothness (CAPS). This adds spatial and temporal smoothness losses to the actor's objective function, penalizing abrupt changes in control surface deflections caused by noise or rapid state changes.

3. Key Contributions

1. Symmetric Data Augmentation Method: A formalized approach to generate symmetric training samples for Markov Decision Processes (MDPs) based on the physical symmetry of aircraft dynamics.
2. DDPG-SCA Algorithm: A novel algorithmic structure featuring dual critics and two-step value iteration. This architecture allows the actor to learn from both real and augmented data simultaneously, significantly improving convergence speed compared to standard mixing strategies.
3. Validation on Fixed-Wing Dynamics: Theoretical proof and simulation verification that the lateral dynamics of a fixed-wing aircraft satisfy the required symmetry conditions, making the method applicable to real-world flight control.
4. Generalization Demonstration: Empirical evidence showing that symmetry-informed policies can successfully track references in unexplored regions (e.g., negative bank angles) where standard RL policies fail.

4. Simulation Results

The methods were tested on a fixed-wing aircraft lateral dynamics model (bank angle, roll rate, sideslip, yaw rate) with the goal of tracking a square-wave bank angle reference.

• Convergence Speed:
  • DDPG-SCA (Dual Critic) converged significantly faster than standard DDPG and DDPG-SDA (Single Critic with mixed batches).
  • In the first 500 episodes, DDPG-SCA achieved a rolling average return of -1672, compared to -2499 for standard DDPG and -2408 for DDPG-SDA.
• State Space Coverage:
  • Augmented samples compensated for gaps in the state space, particularly in regions with high angles and rates where exploration was limited.
  • Coverage rates increased from ~0.28% (explored only) to ~0.54% (explored + augmented) in asymmetric initial conditions.
• Generalization & Tracking Performance:
  • Scenario: Tracking a reference signal that included negative bank angles (unseen during training).
  • Result: Standard DDPG failed to track negative bank angles, producing erratic control actions. In contrast, DDPG-SCA and DDPG-SDA successfully tracked both positive and negative references.
  • Metrics: The Integral of Absolute Error Mean (IAEM) for the roll channel was 1.044 for DDPG-SCA vs. 5.225 for standard DDPG, demonstrating a 5x improvement in tracking accuracy for unexplored regions.

5. Significance

This paper provides a robust solution to the sample efficiency problem in offline and online RL for flight control.

• Cost Reduction: By leveraging physical symmetry, the method reduces the need for extensive exploration, saving computational resources and time.
• Safety & Robustness: It ensures that policies generalize to symmetric states (e.g., left vs. right turns) even if the agent only explored one side, which is critical for safety in aviation.
• Algorithmic Innovation: The "Dual Critic" approach offers a new paradigm for integrating augmented data into deep RL, showing that separating data sources for specialized learning can outperform simple data mixing.
• Applicability: The framework is not limited to fixed-wing aircraft; it is applicable to any mechanical system with configurational symmetry (e.g., robots, cars), provided the symmetry conditions are met.