A public dataset of Ariel simulated observations for developing exoplanetary atmosphere data reduction pipelines

Technical Summary: A Public Dataset of Ariel Simulated Observations for Developing Exoplanetary Atmosphere Data Reduction Pipelines

Problem Statement
The characterization of exoplanet atmospheres faces significant challenges due to atmospheric signals often being comparable to residual noise and instrumental or astrophysical systematics. While spectral features range from a few parts per million (ppm) for small planets to $\sim10^3$ ppm for hot giants, high-quality time-series spectroscopy (e.g., from JWST) typically achieves noise levels of 10–50 ppm. The upcoming ESA Ariel mission aims to survey $\sim1000$ exoplanet atmospheres, necessitating scalable, automated, and robust data reduction pipelines capable of detrending systematic effects without introducing bias. A critical bottleneck is the lack of public benchmark datasets with known "ground truth" to validate these pipelines, particularly for machine learning (ML) approaches which are prone to overfitting and performance degradation when faced with distribution shifts (domain shift) between training and real-world data.

Methodology
The authors generated a comprehensive public dataset of simulated Ariel observations using a coupled simulation framework:

1. Simulation Tools: The dataset was created using ExoSim2 (an end-to-end instrument simulator) coupled with TauREx3 (for atmospheric forward modeling) and PAOS (for optical point-spread functions).
2. Instrument Configuration: Simulations utilized two specific channels: the FGS1 photometer (0.6–0.8 $\mu$m) and the AIRS-CH0 spectrometer (1.95–3.90 $\mu$m, $R=100$).
3. Noise and Systematics: The simulations incorporated realistic noise sources derived from JWST NIRSpec calibration products (dark current, read noise, flat fields, bad pixels) and modeled specific systematic effects including:
   • Pointing Jitter: Simulated using a 10-hour timeline sampled at 1 kHz, inducing time-correlated fluctuations due to intra-pixel response variations.
   • Gain Drift: Modeled as a 5th-order polynomial function of time and wavelength (1–5% amplitude).
   • Detector Effects: Pixel non-linearity, saturation, and Correlated Double Sampling (CDS) readout modes.
4. Target Selection: The dataset includes 1,435 simulated observations across four host stars (KELT-11, HD 17156, HD 209458, HD 149026) and synthetic planets with varying radii, masses, and atmospheric compositions (including trace gases like H$_2$O, CH$_4$, CO$_2$, NH$_3$, etc., and cloud decks).
5. Data Partitioning: To rigorously test generalization, the dataset was partitioned into:
   • Training Sets: Including standard and extended configurations.
   • Test Cases: Designed with explicit "out-of-distribution" shifts, including different detector models, new host stars, broader planetary radii, and entirely different atmospheric compositions (e.g., HCN, H$_2$S) not present in the training set.
   • Calibration Cases: Featureless spectra to assess systematic biases.
6. ML Baseline: The authors implemented two Convolutional Neural Network (CNN) baselines to demonstrate data reduction:
   • 3D-CNN: Processes full spatial-spectral-time cubes to predict mean transit depth and spectral fluctuations.
   • 2D-CNN: Processes binned 2D images (time vs. wavelength) after summing spatial flux, focusing on spectral fluctuations around a mean value predicted by a 1D-CNN.

Key Contributions

• Public Benchmark Dataset: The release of a 7 TB simulated dataset (with a sanitized 174 GB public subset) featuring realistic noise, systematics, and known ground truth spectra. This dataset is hosted on Kaggle as part of the Ariel Data Challenge 2024.
• Stress-Testing Framework: A data partition strategy explicitly designed to expose ML models to domain shifts (different detectors, stars, and atmospheric chemistries) to evaluate robustness against overfitting.
• Baseline Performance: Implementation and evaluation of deep learning baselines for end-to-end data reduction, providing a reference point for the community.

Results

• Mean Transit Depth: Both CNN baselines successfully extracted mean transit depths from noisy data.
• Spectral Feature Recovery: The models struggled to generalize to the test sets. The 3D-CNN tended to converge on a single spectral shape (a compromise across the dataset), failing to capture diverse atmospheric features. The 2D-CNN showed better diversity in validation but failed to generalize to the test set.
• Distribution Shift Impact: Performance degraded significantly on test data with different atmospheric compositions or detector models, highlighting the risk of "data shift" where models memorize training patterns rather than learning general denoising principles.
• Uncertainty Quantification: The models, using Monte Carlo Dropout, failed to provide calibrated uncertainties for out-of-distribution data. Predicted error bars did not scale with actual prediction errors, leading to overconfident predictions on unseen data.
• Mitigation: Adding a small fraction of test data (e.g., 50 samples) to the training set significantly improved Mean Squared Error (MSE), suggesting that adaptive or continuous learning strategies could enhance performance. However, uncertainty calibration (GLL scores) remained poor.

Significance and Claims
The paper claims that this dataset serves as a critical resource for the exoplanet community to develop, compare, and stress-test scalable data reduction pipelines ahead of the Ariel mission launch. The authors emphasize that while their ML baselines demonstrate the potential for automated reduction, they also reveal fundamental limitations: current non-parametric models are susceptible to overfitting and struggle with the domain shifts inherent in survey missions where target properties are unknown. The dataset is intended to foster the development of robust, adaptable algorithms capable of handling diverse observational scenarios, ensuring the scientific return of the Ariel mission. The authors explicitly state that their neural network architectures are proof-of-concept and not yet optimized for scientific use, noting that future work must address domain adaptation and improved uncertainty quantification.