Experimental Demonstration of SDRL Controller for TS Wave Suppression with DBD Actuator

This paper presents an experimental demonstration of a model-free single-step deep reinforcement learning (SDRL) controller that successfully suppresses Tollmien-Schlichting (TS) waves in a flat plate boundary layer by adaptively tuning a finite-impulse-response filter to drive a dielectric barrier discharge (DBD) plasma actuator, achieving robust disturbance reduction across various flow conditions and frequency spectra.

The Gist

The Big Picture: Smoothing Out a Bumpy Ride

Imagine you are driving a car on a highway. The road is usually smooth (laminar flow), but suddenly, a construction crew starts dropping rocks, creating ripples and bumps (Tollmien-Schlichting waves) that travel down the road. If these bumps get too big, the car starts shaking violently, the tires wear out faster, and you lose fuel efficiency. In aerodynamics, this "shaking" is called turbulence, and it creates drag, which makes planes burn more fuel.

The goal of this research is to stop those bumps before they get big enough to ruin the ride. The team built a system that acts like a noise-canceling headphone for the wind.

The Setup: The "Ear," the "Brain," and the "Voice"

The researchers set up a wind tunnel with a flat metal plate to simulate an airplane wing. They placed three key components on this plate:

1. The Ear (Reference Microphone): Located upstream (at the front), this listens to the incoming "bumps" (disturbances) before they hit the main area.
2. The Brain (The AI Controller): This is the star of the show. It's a special type of Artificial Intelligence called Single-Step Deep Reinforcement Learning (SDRL).
3. The Voice (DBD Plasma Actuator): Located downstream (behind the ear), this is a device that can push or pull the air using electricity (plasma) to create a counter-wave.

How the AI Learns: The "Trial and Error" Dance

Usually, to cancel out noise, you need a complex mathematical map of exactly how the wind behaves. But wind is messy and changes constantly.

Instead of trying to map the wind, the AI acts like a child learning to juggle:

• It listens to the "Ear."
• It guesses a command for the "Voice" to push the air.
• It checks a second microphone (the "Error Mic") to see if the bumps got smaller or bigger.
• The Reward: If the bumps get smaller, the AI gets a "gold star" (reward). If they get bigger, it gets a "thumbs down."
• The Magic: The AI repeats this thousands of times in a few minutes. It doesn't need to know the physics of the wind; it just learns the pattern: "When the ear hears a bump at this specific time, I need to push the air exactly like this to cancel it out."

The Experiments: From Single Notes to a Rock Concert

The researchers tested the AI with three different levels of difficulty, like tuning a radio:

1. The Single Note (Single-Frequency): Imagine a pure musical tone (like a flute playing one note). The AI quickly learned to play the exact opposite note to cancel it out. It was very effective, reducing the "noise" by about 40%.
2. The Duet (Multi-Frequency): Now, imagine two instruments playing different notes at the same time. This is harder because the waves interfere with each other. The AI had to learn a more complex pattern to cancel both notes simultaneously. It did even better here, reducing the noise by up to 62%.
3. The Rock Concert (Broadband White Noise): Finally, they turned on a chaotic mix of all frequencies (like static on a radio or a crowd cheering). This is the hardest test. The AI had to learn to cancel a chaotic mess. Even here, it managed to reduce the disturbance energy by about 37–40%, proving it could handle real-world chaos.

The Results: A Smoother Flight

The team didn't just listen to the microphones; they used a high-speed camera (PIV) to actually see the air moving.

• Without the AI: The air was turbulent and chaotic, with big waves growing larger as they moved down the plate.
• With the AI: The waves were crushed. The air remained smooth and calm for much longer.

The most exciting part? The effect didn't stop right where the "Voice" was. The smooth air continued for a long distance downstream. This means the AI didn't just hide the problem; it actually delayed the transition to turbulence, which could eventually help planes fly further on less fuel.

Why This Matters

Previous methods required engineers to build a perfect mathematical model of the wind before the system could work. If the wind changed speed or temperature, the model would break.

This new AI is model-free. It's like a musician who doesn't need sheet music; they just listen to the room and improvise the perfect counter-melody instantly. It is fast, adaptable, and works even when the wind conditions change.

In short: The researchers taught a computer to "listen" to the wind and "sing" back the perfect counter-note to keep the air smooth, proving that AI can be a powerful tool for making flight more efficient and quieter.

Technical Summary

1. Problem Statement

The primary challenge addressed is the reduction of aerodynamic skin-friction drag by delaying the laminar-to-turbulent transition in boundary layers. This transition is primarily triggered by the amplification of Tollmien-Schlichting (TS) waves.

• Limitations of Existing Methods: Traditional active flow control (AFC) methods often rely on model-based approaches (e.g., LQG, MPC) which require accurate system identification and are sensitive to model uncertainty. Classical model-free methods (e.g., FXLMS) are robust but often limited to linear input-output mappings, resulting in slow convergence and an inability to capture complex temporal dependencies.
• The Gap: While Deep Reinforcement Learning (DRL) offers potential for handling non-linearities, full DRL pipelines are computationally expensive and impractical for real-time experimental fluid dynamics. Furthermore, previous successful applications of Single-Step Deep Reinforcement Learning (SDRL) were limited to numerical simulations. This study aims to bridge the gap by implementing SDRL in a physical wind-tunnel environment, addressing real-world challenges such as sensor noise, electromagnetic interference (EMI) from actuators, and latency.

2. Methodology

The study implements a model-free, real-time control framework in an anechoic wind tunnel at Delft University of Technology.

Experimental Setup

• Facility: Low-turbulence, zero-pressure-gradient flat-plate boundary layer ($U_\infty = 20$ m/s and $22.5$ m/s).
• Actuation: Two spanwise-uniform Dielectric Barrier Discharge (DBD) plasma actuators:
  • T-DBD (Trigger): Located upstream to artificially generate TS waves.
  • C-DBD (Control): Located downstream to suppress the waves.
• Sensing: A linear array of 14 flush-mounted electret condenser microphones.
  • Reference Sensor ($x_s$): Upstream of the control actuator.
  • Error Sensor ($e_s$): Downstream of the control actuator.
• Flow Verification: Two-component Planar Particle Image Velocimetry (PIV) was used to visualize velocity fluctuations and validate pressure-based attenuation.

Control Framework (SDRL)

The controller utilizes a Single-Step Deep Reinforcement Learning (SDRL) approach, distinct from standard multi-step DRL:

• Policy Representation: Instead of learning a complex state-dependent policy, the agent learns a Finite Impulse Response (FIR) filter that maps the upstream reference signal directly to the actuator command.
• State Independence: The policy is state-independent; the temporal dynamics of the flow are handled by the FIR filter and the signals themselves, not encoded in the neural network's state.
• Algorithm:
  • The agent samples FIR coefficients from a multivariate normal distribution $\pi_\theta(\hat{a}) = \mathcal{N}(\mu, C)$.
  • Reward Function: Based on the Root-Mean-Square (RMS) attenuation of the error signal relative to the reference signal ($r = (R_{RMS} - E_{RMS}) / R_{RMS}$).
  • Update Mechanism: Gradient ascent on the expected reward updates the distribution parameters ($\mu, C$).
• Real-Time Implementation: The system operates in a feedforward layout with a sampling rate of 5.0–5.6 kHz. The control signal is soft-clipped to prevent amplifier saturation and actuator degradation.

Test Cases

The controller was tested under progressively complex excitation scenarios:

1. Single-Frequency (Case A): Deterministic tone at $F=57$ (240 Hz).
2. Multi-Frequency (Cases B1, B2): Two-tone ($F=[57, 62]$) and three-tone ($F=[57, 62, 67]$) excitations.
3. Broadband (Cases C1, C2): White-noise forcing covering the unstable TS band (150–350 Hz).

3. Key Contributions

1. First Experimental SDRL Demonstration: This is the first successful implementation of SDRL for TS wave suppression in a physical wind tunnel, moving beyond purely numerical simulations.
2. Robustness to Real-World Constraints: The study demonstrates that SDRL can learn effective control policies despite experimental noise, EMI from high-voltage DBD actuators, and convection delays.
3. Adaptability Across Spectra: The controller successfully adapted to single-frequency, multi-frequency, and broadband disturbances without requiring re-tuning of the underlying algorithm, only adjusting filter length and action bounds.
4. Physical Interpretability: The learned FIR filters were shown to correspond to physically meaningful delay-and-gain mappings, effectively canceling waves via destructive interference.

4. Results

• Convergence: The controller achieved rapid convergence, stabilizing within 100–200 generations (approx. 1000 interactions). The reward function indicated stable learning of the amplitude-phase relationship required for opposition control.
• Attenuation Performance:
  • Single-Frequency: Achieved ~40% amplitude reduction.
  • Multi-Frequency: Achieved the highest performance, with up to 62.2% amplitude reduction (Case B2).
  • Broadband: Achieved 37–39% amplitude reduction and 65–68% spectral energy reduction across the unstable band.
• Velocity Field Validation (PIV): PIV measurements confirmed that the pressure attenuation corresponded to a genuine reduction in boundary layer velocity fluctuations.
  • In Case B1, the peak streamwise velocity fluctuation ($P_{90}\{\sigma_u\}$) decreased from 2.15% to 0.67% of $U_\infty$ (69% reduction).
  • In Case C1, fluctuations decreased from 2.73% to 1.26% (54% reduction).
• Downstream Persistence: The attenuation effect persisted several wavelengths downstream of the error sensor (up to $x=134$), confirming that the controller weakened the instability growth rate rather than just suppressing the local measurement signal. This suggests a genuine delay in transition onset.
• Velocity Robustness: The controller maintained performance when freestream velocity increased from 20 m/s to 22.5 m/s, showing robustness to moderate Reynolds number variations.

5. Significance

This work validates SDRL as a viable, computationally efficient alternative to traditional adaptive control (like FXLMS) and complex model-based DRL for fluid dynamics.

• Practicality: By using a compact, state-independent policy, the method avoids the high computational cost and data requirements of full DRL, making it suitable for real-time hardware implementation.
• Transition Delay: The experimental evidence of downstream disturbance suppression provides a strong foundation for using data-driven, model-free controllers to delay boundary layer transition, offering a pathway to significant drag reduction in aerospace applications.
• Future Outlook: The study suggests that extending this framework to multi-step DRL or multi-actuator configurations could further enhance control capabilities for complex, unsteady flow environments.