Technical Summary: Reinforcement Twinning for Hybrid Control of Flapping-Wing Drones

Problem Definition

Controlling flapping-wing micro air vehicles (FWMAVs) presents a significant challenge due to their intrinsically unstable, strongly nonlinear, and time-varying dynamics. These systems are underactuated and operate in regimes where viscous forces dominate inertial effects, leading to complex aerodynamic interactions such as leading-edge vortices. Furthermore, small-scale manufacturing introduces parametric uncertainties, and sensor data is often noisy.

Traditional control strategies face a fundamental trade-off:

• Model-Based (MB) approaches (e.g., LQR, MPC) rely on simplified linear or nonlinear models that often fail to capture time-varying uncertainties and complex kinematics, leading to poor performance in agile maneuvers.
• Model-Free (MF) approaches (e.g., Reinforcement Learning) avoid explicit modeling assumptions but suffer from poor sample efficiency, requiring extensive and potentially unsafe trial-and-error exploration to converge.

The paper addresses the need for a control framework that combines the robustness of physics-based modeling with the adaptability of data-driven learning, specifically tailored for the sample-inefficient and high-risk environment of FWMAV flight.

Methodology: Reinforcement Twinning (RT)

The authors propose a Reinforcement Twinning (RT) framework, a hybrid model-free/model-based architecture designed to facilitate bidirectional collaboration between two learning agents. The system operates through the following components:

1. Dual-Agent Architecture

Two parallel control policies are trained simultaneously:

• Model-Free Policy ($\pi_{mf}$): Trained using the Deep Deterministic Policy Gradient (DDPG) algorithm. It interacts directly with the real environment (a nonlinear time-varying, NLTV, simulation of the drone) to maximize cumulative reward.
• Model-Based Policy ($\pi_{mb}$): Trained within a Virtual Environment (a nonlinear time-invariant, NLTI, surrogate model). This policy is optimized using adjoint-based gradient descent to minimize control cost within the virtual domain.

2. Adaptive Digital Twin (Virtual Environment)

The virtual environment is not a static model but an adaptive digital twin. It utilizes a simplified NLTI formulation derived from cycle-averaged aerodynamic forces.

• Closure Law: The aerodynamic forces are represented as a polynomial function of state and control variables (specifically, a Model 3 formulation including quadratic control terms).
• Online Identification: The model parameters are continuously updated via an adjoint-based data-assimilation procedure. This process minimizes the mismatch between the virtual trajectories and real-world trajectories stored in an "assimilation buffer," allowing the model to adapt to evolving dynamics and uncertainties in real-time.

3. Cooperation Mechanisms

The two agents are coordinated by a Policy Referee that manages the interaction between the real and virtual worlds:

• Policy Arbitration: A referee selects which policy ($\pi_{mf}$ or $\pi_{mb}$) acts as the "live" controller in the real environment. This decision is based on:
  • Real-to-Virtual Trust Ratio (RVET): A metric comparing predicted improvements in the virtual environment against observed improvements in the real environment. If the virtual model is inconsistent with reality, the model-free policy is deployed.
  • Performance Comparison: If the idle policy (the one not currently deployed) consistently outperforms the live policy in the virtual environment, a switch may occur.
  • Stagnation Detection: If learning stagnates, weights are cloned from the better-performing policy to the weaker one to prevent divergence.
• Shared Experience:
  • Transitions from the live policy (whether MF or MB) populate a Replay Buffer used to train the MF critic and actor.
  • State-action trajectories are stored in an Assimilation Buffer (updated via a maximum-variance criterion to ensure diversity) to identify and update the virtual environment's closure law.

Key Contributions

The paper details three primary technical contributions:

1. Online Model Identification Strategy: The formulation of a continuous adjoint-based optimization to calibrate a simplified nonlinear time-invariant (NLTI) model from short-horizon trajectories generated by a nonlinear time-varying (NLTV) environment. This allows the model to adapt to the specific regions of the state space visited by the control policy, rather than relying on pre-collected datasets.
2. Bidirectional Learning Mechanism: Unlike standard Model-Based Reinforcement Learning (MBRL) where the model primarily supports the policy, this framework establishes reciprocal interactions. The model guides exploration in the control space, while the reinforcement learning agent enriches the data used for model identification.
3. Policy Referee and Collaboration: The design of a dynamic arbitration system that manages the deployment of agents based on real-to-virtual consistency, enabling the system to switch between model-based efficiency and model-free robustness as conditions change.

Results

The framework was evaluated on the longitudinal control of a flapping-wing drone (vertical ascent to a target state) under three distinct initialization scenarios:

1. Offline Calibration: When the virtual model was pre-calibrated on a rich dataset, the hybrid RT framework converged significantly faster and with lower variance than a standalone DDPG (model-free) approach. The model-based policy drove the early training, acting as a safeguard for exploration.
2. Fully Online Learning (Random Initialization): In scenarios where both the model and policy started with random parameters (no prior knowledge), the purely model-based approach failed to converge effectively due to initial model bias. The purely model-free approach converged but slowly. The RT framework outperformed both baselines, achieving faster convergence and higher final rewards. The system successfully adapted the surrogate model online, transitioning from model-free dominance to model-based dominance as the digital twin gained fidelity.
3. Online Learning with Model Bias: When the initial model was calibrated on data with significant biases (e.g., 33% mass increase or center-of-pressure shift), the RT framework successfully corrected the surrogate model online. It maintained efficient policy learning and achieved the target state, demonstrating robustness to parametric uncertainties and dynamic changes.

Model Selection: The study identified that a polynomial closure law including quadratic control terms (Model 3) offered the best compromise between predictive accuracy and computational complexity, achieving Pearson correlation coefficients above 0.99 for longitudinal and vertical positions.

Significance and Claims

The paper claims that the Reinforcement Twinning framework successfully mitigates the trade-off between the sample inefficiency of model-free learning and the model inaccuracies of traditional model-based control. By enabling continuous, bidirectional interaction between a data-driven policy and an adaptive physics-based model, the framework:

• Improves sample efficiency and robustness in controlling highly nonlinear, time-varying systems.
• Allows for online adaptation to evolving dynamics and parametric uncertainties without requiring extensive pre-training data.
• Provides a generalizable architecture for real-time control in dynamically evolving environments, specifically demonstrating its efficacy in the challenging context of flapping-wing drone flight.

The authors note that while the current study focuses on longitudinal control with idealized state estimation, future work will extend the framework to full 3D dynamics and validate performance against high-fidelity CFD simulations.