Supercool with PPO: Exploring Supercooled Phase Transitions via Reinforcement Learning

Technical Summary: Supercool with PPO

Problem Statement
The identification of phenomenologically relevant benchmark points for gravitational wave (GW) signals from cosmological first-order phase transitions remains computationally challenging. While current and future GW observatories (e.g., LISA, Taiji, TianQin, PTA) offer a powerful probe for hidden sectors and Beyond the Standard Model (BSM) physics, viable and detectable signals typically occupy only a small fraction of the scanned parameter space. Traditional Monte Carlo (MC) sampling, which draws points from fixed distributions (uniform or log-uniform), is statistically well-defined for generating representative backgrounds but is highly inefficient for the specific goal of discovering detectable signals. MC scans often expend significant computational resources on parameter points that yield mathematically successful phase transitions but produce GW spectra that are either too weak or peak outside the sensitive frequency ranges of detectors. This inefficiency is exacerbated in high-dimensional parameter spaces where each model evaluation involves complex, gauge-dependent calculations of effective potentials and tunneling actions.

Methodology
The authors propose a reinforcement learning (RL) strategy based on Proximal Policy Optimization (PPO) to accelerate the search for GW signals in a minimal dark $U(1)_X$ sector. The approach frames parameter exploration as a sequential decision process where an agent proposes updates to model parameters ($v_X, g_X, \lambda_X$) and receives a reward based on the resulting GW observables and physical constraints.

1. Physical Framework:

   • Model: A minimal dark $U(1)_X$ extension of the Standard Model involving a dark Higgs, dark photon, and optional dark fermions.
   • Gauge Independence: To address the gauge dependence of finite-temperature effective potentials, the authors employ a gauge-independent formulation based on the Nielsen identity. They focus on low-temperature phase transitions ($T_p / m_{A'} \lesssim 0.5$), utilizing a derivative expansion of the effective action where the leading-order potential is explicitly gauge-independent.
   • Supercooling: The study targets supercooled phase transitions, which are known to generate stronger GW signals and are relevant for PTA and space-based detector signals.

2. Reinforcement Learning Setup:

   • Algorithm: PPO is used for its stability in continuous action spaces. It employs a policy network ($\pi_\theta$) to sample actions (parameter moves) and a value network ($V_\phi$) to estimate state values. Updates are constrained by a clipped surrogate objective to prevent large policy deviations.
   • Environment: A numerical environment maps microscopic parameters to GW observables (peak frequency $f_{GW}$ and amplitude $\Omega_{GW}h^2$).
   • Reward Designs: Four distinct reward strategies were developed to guide the agent:
     • General Scan: Rewards points with amplitudes larger than the best previously found in a local frequency neighborhood, encouraging broad frequency coverage.
     • Detector Target Scan: Incorporates detector sensitivity curves (Taiji, TianQin, LISA) as fixed thresholds, rewarding signals that exceed these bounds.
     • History-Informed Global Scan: Tracks historical frequency boundaries to encourage both broad coverage and local amplitude maximization.
     • Fixed Boundary Scan: Uses fixed reference values for frequency and amplitude to guide the search toward specific targets.

Key Contributions

• RL Framework for GW Phenomenology: The paper constructs a specific RL environment for GW physics, demonstrating how PPO can be adapted to navigate high-dimensional, computationally expensive parameter spaces with goal-directed objectives.
• Gauge-Independent Low-Temperature Scan: The implementation integrates a gauge-independent low-temperature effective action formulation directly into the RL loop, ensuring physical consistency for the supercooled transitions being targeted.
• Comparative Analysis: The study provides a rigorous comparison between PPO and conventional MC scans across narrow ($v_X \in [0.1, 1]$ GeV, $[1, 10]$ GeV) and broad ($v_X \in [0.1, 10]$ GeV) windows.
• Policy Transfer: The authors demonstrate the efficacy of "policy transfer," where an agent trained on a narrow parameter window is applied directly to a broader window, showing improved efficiency over training from scratch.

Results

• Efficiency Gains: In narrow-window scans, PPO significantly outperforms MC. For approximately the same runtime, the PPO general scan found ~2.8 times more viable low-temperature benchmark points than MC. To find the same number of detectable points, MC required ~4 times the computational time of the PPO detector target scan.
• Broad-Window Performance: In the broad window ($v_X \in [0.1, 10]$ GeV), PPO scans were generally more efficient than MC of similar duration. The Detector Target Scan showed the best overall performance, balancing the discovery of large-amplitude signals with efficient location of points within detector sensitive regions.
• Frequency Coverage: While PPO is highly efficient at finding specific targets, the History-Informed Global Scan and General Scan in broad windows did not always achieve the same breadth of frequency coverage as a sufficiently long MC scan (which ran for ~3.6 times longer in some comparisons). However, the Policy Transfer strategy (applying a narrow-window trained agent to a broad window) achieved excellent results in identifying detector-sensitive points, outperforming both the scratch-trained PPO and the long-running MC in that specific metric, though with slightly reduced coverage at the extreme high-frequency boundary.
• Macroscopic Parameters: Analysis in the $(\alpha, \beta/H)$ plane confirmed that PPO agents learn to concentrate searches in regions of parameter space corresponding to strong, supercooled transitions, whereas MC scans accumulate a denser distribution of physically inconsistent or detector-irrelevant points.

Significance and Claims
The paper claims that reinforcement learning offers a broadly applicable framework for "learning-assisted exploration" of high-dimensional scientific parameter spaces. The authors position this work not as a replacement for statistical sampling (which remains essential for global characterization and uncertainty quantification) but as a complementary, goal-directed search engine.

The primary significance lies in demonstrating that AI can accelerate the research process when a scientific problem has a clearly defined objective (e.g., finding detectable GW signals). The study highlights that the performance of the agent is intrinsically linked to the reward design; by translating scientific goals (detectability, physical consistency) into reward functions, the agent learns efficient exploration strategies that outperform blind sampling. The authors modestly frame this as a "modest starting point" to illustrate how RL can be integrated into realistic scientific workflows, suggesting that such methods will become increasingly valuable as theoretical models proliferate and detectors mature, helping to connect broad BSM landscapes to concrete observational targets.