⚡ electrical engineering

Comparative Evaluation of RL-TD3 PID Control against Hybrid and Conventional Optimizers in Renewable -Based Islanded Microgrids

This paper demonstrates that a Reinforcement Learning-based Twin Delayed Deep Deterministic Policy Gradient (RL-TD3) optimized PID controller outperforms conventional and hybrid optimization methods, such as Coati Optimization and Grey Wolf-Cuckoo Search, in enhancing load frequency control stability and disturbance rejection within renewable-based islanded microgrids.

Original authors: Dana Rajaa Shaaban, Ali Nasser Hussain, Ahmed K. Ali, Mohammed Alruwaili, Moustafa Ahmed Ibrahim

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dana Rajaa Shaaban, Ali Nasser Hussain, Ahmed K. Ali, Mohammed Alruwaili, Moustafa Ahmed Ibrahim

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a tiny, isolated island of electricity called a Microgrid. It's like a self-contained power city that doesn't connect to the big national grid. Instead, it runs on a mix of sources: some are reliable (like diesel generators), and some are as fickle as the weather (like solar panels and wind turbines).

The big problem? Keeping the lights on at the exact right speed. In the world of electricity, this speed is called frequency. If the wind stops blowing suddenly or someone turns on a massive air conditioner, the frequency wobbles. If it wobbles too much, the whole island's power system could crash.

To fix this, the island needs a Load Frequency Control (LFC) system. Think of this system as the island's autopilot. Its job is to constantly nudge the power sources to keep the frequency steady. For a long time, these autopilots used a standard recipe called a PID controller. It's like a driver who knows the rules but can't react fast enough when the road suddenly turns into a muddy, unpredictable swamp.

The New Kid on the Block: The "Super-Learner" Driver

The researchers in this paper wanted to see if they could build a smarter autopilot. They didn't just want a driver who follows rules; they wanted a driver who learns from experience.

They created a new system called RL-TD3 PID.

PID is the standard autopilot.
RL stands for Reinforcement Learning. Imagine a video game character that learns to win by trying millions of different moves, failing, and then getting better every time it tries again.
TD3 is the specific "brain" algorithm they used. It's like a super-smart coach that watches the player, corrects their mistakes, and helps them avoid overestimating how good they are (a common trap in learning).

The team trained this "Super-Learner" in a computer simulation. They threw all sorts of chaos at it: sudden spikes in power demand, random weather changes, and even "broken" parts where the system's numbers were slightly off.

The Big Showdown: Who Wins the Race?

To see if this new "Super-Learner" was actually better, the researchers pitted it against two other methods:

COA (Coati Optimization Algorithm): A method inspired by how coatis (a type of raccoon-like animal) search for food.
GWO-CS (Grey Wolf + Cuckoo Search): A hybrid method mixing the hunting strategies of wolves with the sneaky egg-laying habits of cuckoos.

These two other methods are like expert navigators who use complex maps and math to find the best route before the trip starts. They are great, but they are static; they don't learn while driving.

The Result:
In the computer simulations, the RL-TD3 PID "Super-Learner" consistently beat the other two in terms of speed and overall error, though it wasn't perfect in every single metric.

Faster Recovery: When the power demand jumped, the Super-Learner got the frequency back to normal faster (or with a very similar speed) compared to the others.
Better Overall Error: It kept the cumulative error lower over time. The paper measured this using specific "scorecards" like ITAE (a number that counts how much error happened over time). The Super-Learner got lower scores, which is the good kind of low. For example, in one test, the Super-Learner had an ITAE of 2.65E-02, while the Coati method had 8.40E-02. That's a big difference in the world of tiny numbers!
Better at Chaos: When the researchers messed with the system's settings (simulating broken or uncertain parts), the Super-Learner stayed remarkably stable. The data showed that even when system parameters were changed by ±25%, the settling time remained nearly identical (around 0.043 seconds) across all scenarios, proving the controller is robust against uncertainty.

Note on the "Wobble": Interestingly, the "Super-Learner" didn't always have the smallest initial "jolt" (overshoot or undershoot) in every single test. In some specific scenarios, it actually had slightly larger swings than the other methods before settling down. However, because it corrected itself so quickly and kept the total error low, it still performed better overall. It traded a tiny bit of initial swing for much better long-term stability.

What the Paper Doesn't Say

It's important to know what this paper doesn't claim.

It's a Simulation, Not a Real-World Win: The authors explicitly state that this was all done in a computer simulation using MATLAB. They did not build a real island and test it with real diesel engines and solar panels yet. The "Super-Learner" is a promising idea, but it hasn't been proven on real hardware.
It's Not Perfect Everywhere: In some specific, complex disturbance scenarios, the Super-Learner had slightly bigger initial swings (overshoots or undershoots) compared to the other methods. It's not a magic bullet that fixes every single metric perfectly every single time; it prioritizes rapid correction and low total error over having the absolute smallest initial dip.
It's Not a Replacement for Everything: The paper suggests this is a better approach for these specific islanded microgrids, but it doesn't claim to solve every power problem in the universe.

The Bottom Line

The paper suggests that teaching a controller to learn and adapt in real-time (using the TD3 brain) is a powerful way to keep an island's power stable, especially when the weather is unpredictable.

While the "Super-Learner" showed it could handle the chaos better than the "Expert Navigators" (COA and GWO-CS) in the computer world—specifically by minimizing total error and maintaining stability even when system parts were "broken"—the authors are careful to say this is just the first step. They admit that before we can trust this system with real electricity, it needs to be tested on actual hardware. But for now, the simulation results are a very strong hint that this "learning driver" might be the future of keeping our power grids steady.

Technical Summary: Comparative Evaluation of RL-TD3 PID Control against Hybrid and Conventional Optimizers in Renewable-Based Islanded Microgrids

Problem Statement
The integration of renewable energy sources (RES) into islanded microgrids (IMGs) introduces significant challenges to Load Frequency Control (LFC). These challenges stem primarily from reduced system inertia, unreliable generation patterns, and unpredictable load dynamics. Traditional control systems, particularly those relying on fixed-gain Proportional-Integral-Derivative (PID) controllers or standard tuning methods like Ziegler–Nichols, often fail to maintain frequency stability under nonstandard operating conditions and varying system parameters. While metaheuristic optimization algorithms (e.g., Genetic Algorithms, Particle Swarm Optimization) have been proposed to tune PID gains, they often struggle with random load disturbances and require extensive computational resources to balance hyperparameters. There is a critical need for control strategies that offer adaptability, robustness, and real-time data-driven optimization to handle the stochastic nature of renewable-based IMGs.

Methodology
The paper proposes a novel control framework that integrates a Reinforcement Learning (RL) agent, specifically the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, with a PID controller. The TD3 agent is designed to adaptively tune PID gains ( $K_p, K_i, K_d$ ) in real-time based on system interactions, rather than relying on pre-calculated optimal values.

System Modeling: A dynamic mathematical model of an IMG was developed, incorporating diverse Distributed Generation (DG) units (Diesel Engine Generator, Solar Photovoltaic, Wind Turbine Generator, Biogas Turbine, Fuel Cell) and Energy Storage Systems (Battery, Flywheel, Electric Vehicles). These components were modeled using first-order transfer functions and swing equations to simulate autonomous operation.
RL-TD3 Framework: The TD3 algorithm, an actor-critic method, was employed to minimize frequency deviations. The agent was trained offline over 1,000 episodes (90 seconds each) using a reward function designed to penalize frequency errors. The training utilized a replay buffer, dual critic networks to reduce overestimation errors, and target policy smoothing to enhance stability.
Comparative Benchmarks: The proposed RL-TD3-PID controller was compared against two metaheuristic optimization approaches:
1. COA-PID: Optimized using the Coati Optimization Algorithm.
2. CS-GWO-PID: Optimized using a hybrid of Grey Wolf Optimization and Cuckoo Search.
Simulation Scenarios: Performance was evaluated in MATLAB under two scenarios:
1. Scenario I: A two-step load change (0.06 to 0.075 p.u.).
2. Scenario II: Random step load disturbances varying between 0.02 and 0.09 p.u.
Robustness Analysis: A sensitivity analysis was conducted by varying key system parameters (inertia constant $M$ , damping coefficient $D$ , and time constants for fuel cells and diesel generators) by ±25% in both uniaxial and multiaxial configurations.

Key Contributions

Development of an Adaptive Framework: The authors developed a TD3-based RL agent framework capable of adaptively controlling PID parameters in IMGs to counteract renewable variability and load disturbances.
Dynamic Simulation Model: A comprehensive dynamic simulation model of an IMG was constructed to validate the frequency regulation capabilities of the proposed controller.
Online Adaptation: The study demonstrates the capability of the RL-TD3-PID controller to perform online, data-driven adaptation, enhancing robustness against system uncertainties without requiring a fixed mathematical model of the environment.
Comprehensive Comparative Study: The paper provides a structured comparison between the proposed RL approach and established metaheuristic methods (COA and GWO-CS), evaluating performance across dynamic response, disturbance rejection, and parameter uncertainty.

Results
Simulation results indicate that the RL-TD3-PID controller generally outperforms the COA-PID and CS-GWO-PID controllers in terms of dynamic response and disturbance rejection.

Performance Metrics: In both Scenario I and Scenario II, the TD3-PID controller achieved lower values for integral performance indices, including the Integral of Time-weighted Absolute Error (ITAE) and Integral of Time-weighted Squared Error (ITSE), compared to the optimization-based controllers.
Transient Response: The TD3-PID controller demonstrated faster settling times and better suppression of frequency oscillations. For instance, in Scenario I, the ITAE for TD3-PID was $2.65 \times 10^{-2}$ , significantly lower than $7.09 \times 10^{-2}$ for CS-GWO-PID and $8.40 \times 10^{-2}$ for COA-PID.
Robustness: Under parametric uncertainties (±25% variations), the TD3-PID controller maintained stable operation with minimal deviation in time-domain indices. While some overshoot and undershoot values increased under complex multiaxial disturbances, the controller consistently returned the system to a quasi-steady state with low steady-state error.
Convergence: The training process showed that the RL agent successfully converged to stable PID gain values, with the reward function indicating improved control performance over the 1,000 episodes.

Significance and Claims
The paper claims that the integration of deep reinforcement learning with classical PID control offers a promising approach for reliable and efficient frequency regulation in renewable-based islanded microgrids. The study highlights that the RL-TD3 approach provides superior adaptability to system uncertainties and load variations compared to static optimization methods.

The authors position the work as a significant step toward addressing the limitations of conventional and metaheuristic-tuned controllers in low-inertia systems. However, the paper maintains a modest tone regarding its limitations, acknowledging that the results are based on simulation studies and have not yet been validated through hardware-in-the-loop or real-time experimental implementation. The authors note that while the proposed method showed superior performance in most tested scenarios, the selected optimization algorithms slightly exceeded the RL-TD3-PID performance in specific disturbance cases. Future work is identified as necessary for experimental validation and extending the framework to more complex microgrid topologies and communication-delayed environments.

The New Kid on the Block: The "Super-Learner" Driver

The Big Showdown: Who Wins the Race?

What the Paper Doesn't Say

The Bottom Line

More like this