ReLaTS: a Reinforcement Learning-based method for dynamically determining the coupling Time Step in multi-scale simulations of self-gravitating systems

Technical Summary: ReLaTS – Reinforcement Learning for Dynamic Time-Step Coupling in Multi-Scale N-Body Simulations

Problem Statement
Astrophysical simulations of self-gravitating systems, such as star clusters containing planetary systems, often involve multi-scale problems where spatial and temporal scales differ by orders of magnitude (e.g., planetary orbits vs. cluster relaxation times). These are typically addressed using "Bridge" schemes, which couple separate numerical solvers (e.g., a direct N-body integrator for the cluster and a symplectic integrator for the planetary system). However, the coupling relies on a fixed, manually set time step ($\Delta t_B$). Selecting an optimal $\Delta t_B$ requires significant expert knowledge: a step that is too large yields inaccurate results due to insufficient resolution of interactions, while a step that is too small incurs unnecessary computational overhead. Furthermore, because these systems are chaotic and their topology changes over time (e.g., close encounters, ejections), a constant time step is often suboptimal, leading to either energy drift or wasted computation. Validating these simulations is difficult due to the intrinsic chaos of N-body problems, making energy conservation a primary, though imperfect, diagnostic.

Methodology
The authors introduce ReLaTS (Reinforcement Learning-based method for dynamically determining the coupling Time Step), a framework that uses Deep Q-Networks (DQN) to dynamically select the Bridge time step during runtime.

• System Setup: The study utilizes an "inclusive Bridge" (iBridge) method where a host star is integrated in both the parent (cluster) and child (planetary system) domains. The parent system is integrated using the 4th-order Hermite predictor-corrector integrator (Ph4), and the child system uses the symplectic Huayno integrator.
• Reinforcement Learning Agent:
  • State ($S$): The input to the neural network consists of the gravitational potential of all cluster stars at the location of the common host star and the negative logarithm of the current relative energy error ($-\log_{10}(\Delta E)$). This design avoids the $N$-dependence of Cartesian coordinates, allowing for better generalization.
  • Actions ($A$): A discrete set of 10 possible values for the Bridge time step $\Delta t_B$, ranging from $5 \times 10^{-5}$ to $10^{-2}$ Myr.
  • Reward Function ($R$): A composite function balancing energy error and computation time. It penalizes large energy errors logarithmically and rewards larger time steps (faster computation), weighted to prioritize accuracy.
• Training Strategy: The model is trained in phases: a global search with a high learning rate on small systems (5 stars), followed by a local search with a lower learning rate. The training data is generated from simulations with varying numbers of stars (5–20) and random initial conditions.
• Hybrid Correction (H-RL): To address long-term integration errors where a single wrong action might cause a jump in energy error, a hybrid method is proposed. If the energy error increases significantly between steps, the integration step is repeated with a smaller time step (or half the current step) up to 4 times to correct the trajectory.

Key Results

• Performance vs. Fixed Time Steps: ReLaTS consistently outperforms simulations using fixed, expert-chosen time steps. In terms of the Pareto front (balancing energy error and computation time), ReLaTS achieves results that lie on or below the curve formed by fixed-step simulations, indicating superior efficiency and accuracy.
• Adaptability: The model successfully identifies dynamic events, such as close stellar encounters, and automatically selects smaller time steps to maintain accuracy. Conversely, it increases the time step during quiescent periods to save computational resources.
• Generalization:
  • Integrator Independence: The trained model generalizes to different numerical integrators (e.g., swapping Ph4 and Huayno) without retraining.
  • Scale Independence: The method works for systems with varying numbers of stars (tested up to $N=15$ in training, and validated up to $N=1,000$ using Barnes-Hut Tree codes) and different system configurations, including triple star systems with protoplanetary disks.
  • Long-term Integration: For integrations extending to 1 Myr, the H-RL variant effectively prevents large energy error jumps, maintaining stability where fixed-step methods fail or require excessive computational cost.
• Limitations: The method is less reliable for systems with infinitesimal masses (test particles), as their contribution to the total energy is negligible, making the energy error a poor diagnostic for errors in their integration. Additionally, the current implementation assumes strict separation of subsystems; objects migrating between domains (e.g., ejected planets) are not dynamically re-categorized within the iBridge framework, though the authors note this could be adapted for more complex schemes like Nemesis.

Significance and Claims
The paper claims that ReLaTS provides a robust, adaptive, and "expert-free" solution for coupling multi-scale gravitational simulations. By automating the selection of the coupling time step, ReLaTS eliminates the need for manual tuning and convergence studies, which are traditionally required to balance accuracy and speed. The authors emphasize that the method is not tied to specific numerical schemes or specific system sizes, allowing it to generalize across diverse astrophysical domains. The study demonstrates that reinforcement learning can effectively manage the trade-off between computational cost and physical fidelity in chaotic, multi-scale N-body problems, substantially reducing energy errors relative to fixed-time step baselines without incurring significant additional overhead.