ThousandWorlds: A benchmark for climate emulation of potentially habitable exoplanets

Technical Summary: ThousandWorlds: A benchmark for climate emulation of potentially habitable exoplanets

Problem Statement
The detection of life beyond Earth relies on interpreting faint atmospheric signatures of potentially habitable exoplanets. However, these signatures are ambiguous; for instance, an oxygen detection could indicate biology or merely the photodissociation of water. Resolving this ambiguity requires understanding the host planet's climate, including temperature, circulation, clouds, and heat transport. Currently, this understanding is derived from Global Climate Models (GCMs), which are computationally expensive (requiring $10^4$–$10^6$ core-hours per run) and require significant expert time to configure. This cost limits studies to small ensembles of hand-picked configurations, creating a bottleneck for large-scale parameter sweeps and uncertainty quantification. While machine learning (ML) emulators offer a potential solution, progress has been stalled by the absence of a curated, multi-model dataset. Existing simulations are scattered across incompatible formats, vertical grids, and output variables, preventing the training of robust emulators that can generalize across different simulators.

Methodology and Dataset Construction
The authors introduce ThousandWorlds, an ML-ready benchmark designed for exoclimate emulation and the broader regime of low-data, multi-simulator, parameter-to-field regression.

• Dataset Composition: The dataset comprises approximately 1,800 simulations from five distinct GCMs: the Unified Model (UM), ExoCAM, ExoCAM-pre-2022, LFRic, and ExoPlaSim. These simulations cover a physical domain of tidally locked waterworlds (ocean-covered rocky planets in the habitable zone).
• Inputs: Each simulation is defined by eight continuous planet parameters: radius, surface gravity, rotation period, surface pressure, CO$_2$ volume fraction, CH$_4$ volume fraction, incident stellar flux, and stellar temperature. A discrete label identifies the source GCM.
• Outputs: The target is the time-mean atmospheric state represented as 53 fields on a $32 \times 64$ latitude–longitude grid with 10 pressure levels. Variables include temperature, humidity, winds (zonal and meridional), cloud fraction, surface temperature, outgoing longwave radiation (OLR), and absorbed shortwave radiation (ASR).
• Data Challenges: The dataset intentionally incorporates three nested subsets to stage increasing difficulty:
  1. Single-complete: Regression using only the UM GCM (no multi-simulator transfer, no missingness).
  2. Multi-complete: Regression using all five GCMs with complete observations (introduces cross-GCM transfer).
  3. Multi-partial: The full dataset using all five GCMs and 53 fields, introducing structured missingness (different vertical grids, missing variables, and pressure levels).
• Evaluation Protocols:
  1. Standard Protocol: Uses a larger test set to rank methods against one another using area-weighted Root Mean Squared Error (RMSE).
  2. Shared-Planets Protocol: Restricts evaluation to planets simulated by both target GCMs (UM and ExoCAM). Emulator error is measured relative to the disagreement between the two GCMs themselves. This metric assesses whether the emulator performs better than the inherent epistemic uncertainty of the physics models.

Baseline Methods
The authors evaluated seven baselines spanning simple methods, deep learning, and Gaussian processes (GPs):

• Simple: Train-mean and k-Nearest Neighbors (kNN).
• Deep Learning: Coord-MLP (pointwise MLP), Coord-DeepONet (DeepONet architecture), and PCA-MLP (PCA on spectral coefficients followed by MLP regression).
• Gaussian Processes: PPCA-ICM (Probabilistic PCA compression followed by Intrinsic Coregionalization Model GP regression) and GPLFR (Gaussian Process Latent Factor Regression, an end-to-end optimization of the latent representation and GP kernel).

Key Results

• Performance: GP-based methods emerged as the strongest baselines. GPLFR achieved the lowest RMSE on the majority of variables across all subsets, followed closely by PPCA-ICM.
• Deep Learning Limitations: Among deep learning methods, PCA-MLP performed best. Notably, Coord-DeepONet and Coord-MLP underperformed relative to GP methods, particularly on the Single-complete subset. This suggests that the low-data, parameter-to-field regime of exoclimate emulation presents challenges for standard off-the-shelf deep learning approaches.
• Variable Specifics: kNN was competitive with learned methods for cloud fraction, winds, and ASR, suggesting these variables are dominated by local spatial templates or inter-GCM variability rather than smooth global trends. Conversely, temperature fields showed the clearest benefit from complex nonlinear models.
• Scientific Utility: Under the shared-planets protocol, the best baselines (GPLFR, PPCA-ICM, PCA-MLP) achieved relative RMSEs below 1 for most variables. This indicates that these emulators can predict individual GCM outputs more accurately than the GCMs agree with each other, a threshold of practical relevance for exoplanet astronomers. However, significant headroom remains in absolute error terms (e.g., surface temperature RMSEs around 10 K).
• Data Composition: Auxiliary GCM data (lower fidelity models) benefited GP methods but was neutral or detrimental to deep learning methods, suggesting GPs are better at exploiting cross-GCM information. Data from outside the strict target physical domain was beneficial across methods, likely helping to anchor predictions near domain boundaries.

Significance and Claims
The paper claims that ThousandWorlds fills a critical gap in scientific ML by providing the first large, curated, multi-GCM dataset for emulating potentially habitable exoplanet climates. Its significance lies in:

1. Benchmarking a Challenging Regime: It exposes a regime (low-data, parameter-to-field, multi-simulator) where standard deep learning struggles and where Gaussian processes currently outperform, offering a testbed for novel surrogate methods.
2. Scientific Utility: It provides a framework to measure emulator performance not just against a single ground truth, but against the spread of high-fidelity physics models, directly addressing the epistemic uncertainty inherent in climate science.
3. Community Resource: By releasing the dataset and code, the authors aim to accelerate the field of exoplanet climate research, enabling large parameter sweeps and principled uncertainty quantification that were previously computationally prohibitive.

The authors remain modest, noting that while emulators are approaching the accuracy of inter-GCM disagreement, they are not yet perfect substitutes for full GCM runs, and the benchmark is currently limited to tidally locked waterworlds, excluding other planet classes like dry planets or eccentric orbits.