Technical Summary: Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control

Problem Statement

Active Flow Control (AFC) is critical for optimizing real-world systems such as aerodynamics, energy harvesting, and turbulence reduction. While Reinforcement Learning (RL) has shown promise in this domain, progress is hindered by a fragmented research landscape. Existing studies rely on heterogeneous observation and actuation schemes, diverse numerical setups, and inconsistent evaluation protocols, making direct comparisons difficult. Furthermore, current benchmarks suffer from three primary limitations:

1. Dependency on External Solvers: Most benchmarks require coupling with external Computational Fluid Dynamics (CFD) solvers (e.g., OpenFOAM, Fluent), creating brittle software stacks that demand specialized expertise and hinder integration with standard Python RL libraries.
2. Lack of Differentiability: Existing environments are largely non-differentiable, preventing the use of gradient-based control methods like Differentiable Predictive Control (DPC) and limiting the application of modern differentiable RL techniques.
3. Limited Scope: Current benchmarks often lack support for Multi-Agent RL (MARL), which is natural for spatially distributed actuation, and are predominantly restricted to 2D scenarios, failing to capture essential 3D flow physics.

Methodology: FluidGym

To address these limitations, the authors introduce FluidGym, the first standalone, fully differentiable benchmark suite for RL in AFC.

Architecture and Design

• Single-Stack Implementation: Built entirely in PyTorch on top of the GPU-accelerated PICT solver (a differentiable, multi-block PISO solver), FluidGym eliminates the need for external CFD software. All simulations and control interfaces reside within a single Python package, allowing installation via pip and immediate compatibility with standard RL libraries (e.g., Gymnasium, PettingZoo, Stable-Baselines3, TorchRL).
• Differentiability: The framework is inherently end-to-end differentiable. This enables gradient-based control methods (like DPC) to backpropagate through the simulation to policy parameters without modifying the environment, while also supporting classical RL.
• Interaction Modes: FluidGym supports three interaction modes:
  1. Single-Agent RL (SARL): A single agent outputs a global action vector.
  2. Multi-Agent RL (MARL): Multiple agents act simultaneously at different spatial locations, receiving local observations and rewards, enabling decentralized control strategies.
  3. Gradient-Based Control: Direct optimization of policies via reward gradients propagated through the differentiable simulation.

Benchmark Environments

FluidGym provides 13 distinct environment configurations spanning four key flow scenarios, each offered in three difficulty levels (Easy, Medium, Hard) to vary turbulence intensity and flow complexity:

1. Flow Past a Cylinder: Focuses on drag reduction via synthetic jets or cylinder rotation. Includes 2D and 3D variants.
2. Rayleigh-Bénard Convection (RBC): Focuses on heat transfer enhancement by minimizing the Nusselt number via bottom-boundary heating. Includes 2D, 3D, and wide-domain variants.
3. Flow Past an Airfoil: Aims to improve aerodynamic efficiency (lift-to-drag ratio) via surface-mounted synthetic jets. Includes 2D and 3D variants.
4. Turbulent Channel Flow (TCF): Aims to reduce wall shear stress via spatially distributed blowing and suction. Includes small and large domain variants with single or dual-wall actuation.

Experimental Protocol

The authors established a unified training and evaluation protocol to ensure reproducibility:

• Data Splits: Three predefined splits (train, validation, test) containing ten randomly generated initial domains each.
• Seeding: Experiments utilize five random seeds and ten test episodes per configuration.
• Algorithms Evaluated: The study benchmarks Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), Differentiable Predictive Control (DPC), and Temporal Difference Learning for Model Predictive Control (TD-MPC).

Key Results

Algorithm Performance

• SAC vs. PPO: Across all difficulty levels and environments, SAC consistently achieved higher normalized test-set relative improvements over the baseflow compared to PPO. However, in Multi-Agent settings, PPO and SAC performance became more comparable, likely due to PPO benefiting from increased sample counts in parallelized MARL setups.
• Gradient-Based Methods: DPC demonstrated significant advantages in sample efficiency. On the CylinderJet2D-easy environment, DPC outperformed PPO and SAC by approximately one and two orders of magnitude in training speed, respectively. In more turbulent settings (RBC2D-hard), DPC remained competitive with TD-MPC and SAC, outperforming PPO.
• Multi-Agent Learning: In the RBC3D environment, MARL agents successfully learned to coordinate bottom-wall heating to form stable convection rolls, demonstrating the framework's ability to support decentralized cooperation.

Transfer Learning

• Dimensionality Transfer: Policies trained in 2D were successfully transferred to 3D cylinder environments. On easy and hard tasks, the transferred 2D policies outperformed or matched baselines trained directly in 3D, suggesting robustness in cross-dimensional transfer.
• Domain Size Transfer: Policies trained on small Turbulent Channel Flow domains generalized effectively to larger domains, outperforming policies trained directly on the large domain. This highlights the potential of MARL to learn translation-equivariant control strategies.

Runtime and Efficiency

• Training Time: DPC incurred higher wall-clock training times (1.5–2×) compared to standard RL due to the computational cost of the backward pass through the differentiable environment.
• Scalability: The framework successfully scaled to large experimental workloads, totaling over 25,000 GPU hours across the study.

Significance and Claims

The paper positions FluidGym as a foundational tool to bridge the gap between RL research and fluid dynamics. Its primary significance lies in:

1. Standardization: Providing the first standalone, plug-and-play benchmark that removes the barrier of external CFD solver dependencies, thereby lowering the entry threshold for researchers.
2. Differentiability: Establishing a fully differentiable environment that enables the systematic comparison of classical RL with gradient-based control methods, a capability previously absent in AFC benchmarks.
3. Comprehensive Scope: Offering a unified platform for both single-agent and multi-agent control across diverse 2D and 3D flow regimes, facilitating the study of spatially distributed control strategies.
4. Reproducibility: Releasing all environments, datasets, trained models, and experimental results publicly to foster reproducibility and rigorous statistical comparison in the field.

The authors note that while the benchmark is a significant step forward, current limitations include reliance on CUDA-enabled GPUs and a focus on standard hyperparameters rather than algorithm-specific tuning. Future work aims to expand the suite to magnetohydrodynamic (MHD) flows and increase the number of random seeds to further enhance statistical robustness.