Integral modelling and Reinforcement Learning control of 3D liquid metal coating on a moving substrate

This study develops a reinforcement learning control strategy using Proximal Policy Optimization to stabilize 3D liquid metal films on moving substrates by coordinating gas jets and electromagnetic actuators, which effectively reduce interface instabilities by pushing wave crests and raising troughs.

The Gist

Imagine you are trying to paint a long, moving conveyor belt with a thick, molten metal sauce. You want the sauce to spread out into a perfectly smooth, even layer. But there's a problem: as the belt moves too fast, the sauce doesn't stay flat. Instead, it starts to ripple and wave, like a flag flapping in the wind. These ripples ruin the quality of the final product.

This paper is about a new way to smooth out those ripples using a combination of air jets and magnets, guided by a computer that learns how to fix the problem on its own.

Here is a breakdown of how they did it, using simple analogies:

1. The Problem: The "Wobbly Sauce"

In industries like making galvanized steel, a metal sheet is dipped in molten zinc and pulled out. To get the right thickness, engineers blow air jets at the wet metal to wipe off the excess. However, if the sheet moves too fast, the air and the liquid fight each other, creating unstable waves (ripples) on the surface.

2. The Map: A "Simplified GPS" for Liquid

To control these waves, you need to know exactly how the liquid will behave. Usually, simulating liquid metal with magnets is like trying to calculate the flight path of every single raindrop in a storm—it's too heavy for computers to handle in real-time.

The authors created a "Simplified GPS" (called an Integral Boundary Layer model). Instead of tracking every drop, this model tracks the "average" behavior of the liquid film. It's like looking at the traffic flow on a highway rather than counting every individual car. This allowed them to run thousands of simulations quickly to test different control strategies.

3. The Teachers: Air and Magnets

The researchers tested two tools to smooth the waves:

• The Air Jet: Think of this as a strong fan blowing on the top of the liquid. It pushes the high points (crests) of the waves down.
• The Electromagnet: This is the trickier tool. When you apply a magnetic field to moving liquid metal, it creates an invisible force (Lorentz force) that acts like a "magnetic hand." This hand pushes the liquid, but in a specific way: it tends to lift the low points (troughs) of the waves.

4. The Student: The AI Coach (Reinforcement Learning)

Instead of writing a complex manual rulebook for how to use the air and magnets, the researchers taught a computer program (an AI) to learn by trial and error. This is called Reinforcement Learning.

• The Game: The AI acts as a coach. It watches the liquid film through "eyes" (sensors) and decides whether to blow air or turn on the magnet.
• The Score: If the waves get smaller, the AI gets a "point" (reward). If the waves get bigger, it loses points.
• The Learning: The AI played this game 300 times in parallel, trying millions of different combinations of air and magnet settings. Over time, it figured out the perfect dance.

5. The Discovery: The Perfect Dance

The AI discovered a clever strategy that neither tool could do alone:

• The Air Jet acts like a flattening iron, pushing down the peaks of the waves.
• The Electromagnet acts like a lifter, pushing up the valleys of the waves.

By working together, they squeeze the wave from both the top and the bottom, flattening the liquid film much better than using just one tool. The paper calls this a "novel mechanism" where the two actuators complement each other perfectly.

6. The Catch: The "Heavy" Magnets

The study found that while the magnetic method works great in the computer simulation, it requires a very strong magnetic field to be effective in the real world. The paper notes that achieving this strength would require massive amounts of energy and could create dangerous heat (like a toaster on steroids), which might be too difficult to implement in a real factory right now.

Summary

The paper proves that by combining a simplified math model with a learning AI, we can find a way to smooth out rippling liquid metal. The AI learned that the best way to fix a wobbly wave is to push the high spots down with air and lift the low spots up with magnets, creating a perfectly flat surface. While the magnetic part is currently too energy-intensive for immediate factory use, the method proves that this "teamwork" approach is a powerful new way to think about controlling fluids.

Technical Summary

1. Problem Statement

The study addresses the challenge of controlling undulation instabilities (ripples) in liquid metal films during the hot-dip galvanizing process. In this industrial process, a steel strip is coated with molten zinc to prevent corrosion. As the strip is withdrawn from the bath at high speeds (typically >2 m/s), impinging gas jets are used to wipe off excess liquid. However, these jets often induce oscillations in the liquid film, creating traveling waves that result in uneven coating thickness and reduced surface quality.

Traditional control methods (e.g., Linear Quadratic Regulators or Model Predictive Control) struggle in this environment due to:

• High Reynolds numbers and complex fluid dynamics.
• Strong noise and measurement uncertainties.
• Unmodelled physical effects (thermal gradients, oxidation, substrate vibration).
• The computational cost of high-fidelity simulations (DNS/LES) required for model-based control.

The authors propose a model-free control strategy using Reinforcement Learning (RL) combined with a reduced-order 3D Integral Boundary Layer (IBL) model that incorporates electromagnetic actuators alongside traditional gas jets.

2. Methodology

A. Physical Modelling (3D Integral Boundary Layer)

The authors extended existing 2D IBL models to a 3D framework to capture spanwise effects and electromagnetic interactions.

• Governing Equations: The model is derived from the Magnetohydrodynamic (MHD) Navier-Stokes equations using a long-wave approximation. It tracks film thickness ($h$) and flow rates ($q_x, q_z$).
• Electromagnetic Effects: The model incorporates:
  • Lorentz Force: Acting within the bulk of the liquid metal ($f_L = j \times b$), opposing the flow.
  • Maxwell Stresses: Acting at the free surface.
• Actuators:
  • Gas Jets: Modelled using experimental correlations for pressure and shear stress distributions (impinging jets).
  • Electromagnets: Modelled as time-varying Gaussian magnetic fields inducing Lorentz forces.
• Numerical Solution: The equations are solved using a Fourier pseudo-spectral method for spatial discretization and explicit Euler integration for time. A Perfectly Matched Layer (PML) is implemented at the boundaries to simulate open-flow conditions and absorb outgoing waves.

B. Control Strategy (Reinforcement Learning)

The control problem is formulated as a Markov Decision Process (MDP) solved using the Proximal Policy Optimization (PPO) algorithm.

• State Space ($s$): Observations of the liquid film thickness at specific points upstream of the actuators.
• Action Space ($a$): Control signals for gas jet exit velocities ($U_j$) and electromagnetic field intensities ($b$).
• Reward Function ($r$): Defined as the negative exponential of the standard deviation of the film thickness within a specific "reward region." Maximizing the reward equates to minimizing the amplitude of undulation waves.
• Training: The agent interacts with 30 parallel numerical environments. It learns through trial-and-error to find an optimal policy $\pi(a|s)$ that minimizes wave amplitude.
• Policy Types: Two strategies were tested:
  1. Harmonic Policy: The agent learns amplitude, frequency, and phase for a sinusoidal control signal.
  2. Non-Harmonic Policy: The agent directly outputs continuous control signals based on instantaneous observations (fully adaptive).

3. Key Contributions

1. Novel 3D MHD-IBL Model: Development of a computationally efficient 3D reduced-order model that couples hydrodynamic film dynamics with electromagnetic forces (Lorentz and Maxwell stresses), validated against asymptotic limits and previous DNS data.
2. RL-Based Control for MHD Flows: Demonstration of using PPO to control complex magnetohydrodynamic coating flows without requiring an explicit analytical model of the system dynamics.
3. Discovery of a Novel Control Mechanism: The RL agent identified a synergistic control strategy where:
   • Gas jets act to push down the wave crests (via increased shear/pressure).
   • Electromagnets act to raise the wave troughs (via Lorentz force pushing liquid into the valleys).
4. Robustness to Uncertainty: The study highlights the superiority of RL over model-based methods in handling the high noise and unmodelled physics typical of industrial galvanizing.

4. Key Results

• Single Actuator Performance:
  • Gas Jets: Both harmonic and non-harmonic policies successfully reduced wave amplitudes. The non-harmonic policy achieved a mean reward ~24% higher than the uncontrolled case.
  • Electromagnets: The harmonic policy failed to stabilize the film (reward lower than uncontrolled). However, the non-harmonic policy discovered an effective "bang-bang" style control, acting primarily on the valleys to raise them, achieving a reward ~8% higher than the uncontrolled case.
• Combined Actuator Performance (Tandem):
  • When gas jets and electromagnets were used together, the RL agent learned the complementary mechanism (pushing crests, raising troughs).
  • This combined approach yielded the best results: a 13% increase in mean reward and a 20% reduction in standard deviation compared to the uncontrolled case, indicating superior stability and robustness.
• Physical Insights:
  • The study revealed that for the magnetic field to effectively counteract gravity and inertia in this regime, a Hartmann number ($Ha$) of approximately 16 is required. Current industrial setups ($Ha \approx 6$) are suboptimal, suggesting that while the control logic works, the physical actuation strength needs to be higher for industrial viability.
  • The magnetic field alters the phase speed of surface waves, potentially shifting the flow from convectively unstable to absolutely unstable regimes.

5. Significance and Future Perspectives

• Industrial Impact: This work provides a proof-of-concept for active control of liquid metal coatings in high-speed manufacturing. It suggests that combining pneumatic and electromagnetic actuators can significantly improve coating uniformity, reducing material waste and energy consumption.
• Computational Efficiency: The use of the IBL model allows for thousands of training episodes in a fraction of the time required for Direct Numerical Simulation (DNS), making RL training feasible for complex fluid problems.
• Future Work:
  • Hardware Implementation: Testing the control laws on physical setups, addressing the challenge of generating stronger magnetic fields (higher $Ha$) without excessive Joule heating.
  • Thermal Coupling: Incorporating thermal effects (Marangoni forces) generated by the electromagnetic field, which could further stabilize the film.
  • Hysteresis Modeling: Refining the magnetic actuator model to include core hysteresis and response delays.
  • Broader Applications: Extending this approach to other MHD applications, such as liquid metal walls in nuclear fusion reactors (tokamaks) or plasma confinement.

In conclusion, the paper successfully demonstrates that Reinforcement Learning can discover sophisticated, non-intuitive control laws for 3D liquid metal films by leveraging a reduced-order MHD model, offering a promising pathway for next-generation industrial coating processes.