Interpretability of deep-learning methods applied to large-scale structure surveys

Technical Summary: Interpretability of Deep-Learning Methods Applied to Large-Scale Structure Surveys

Problem Statement
Deep learning, particularly Convolutional Neural Networks (CNNs), has emerged as a powerful tool for cosmological analysis of large-scale structure (LSS) surveys, offering performance comparable to or exceeding classical methods based on fixed summary statistics. However, a significant barrier to their adoption as primary analysis tools is their "black-box" nature. Unlike classical methods where the decision process is transparent, CNNs optimize millions of parameters with no direct physical meaning. This lack of interpretability obscures potential limitations and biases, making it difficult to trust the results or optimize future surveys specifically for these algorithms. While interpretability is an active field in general machine learning, its application to cosmology remains nascent.

Methodology
The authors propose a novel interpretability approach distinct from "a posteriori" analysis (analyzing a trained network on normal data). Instead, they investigate the network's behavior by systematically degrading the input data before training and observing the impact on constraining power.

• Data and Architecture: The study utilizes a simplified version of the DeepLSS architecture (Kacprzak & Fluri 2022). It employs tomographic weak lensing convergence ($\kappa_g$) maps generated from N-body simulations of a flat $\Lambda$CDM universe. The dataset covers 900 deg$^2$ across four redshift bins. Two network architectures are trained:

  1. A ResNet-based CNN that takes raw pixel maps as input.
  2. A Power Spectrum (PS) neural network that takes auto- and cross-spectra of the maps as input, serving as a baseline for comparison.
     Both networks are trained to predict cosmological parameters $\Omega_m$ and $\sigma_8$ (and their covariances) using a negative log-likelihood loss function.

• Degradation Strategy: To understand which features drive the network's predictions, the authors insert preprocessing layers to remove or alter specific information types before the first convolutional layer. The degradations include:

  • Scale modifications: Changing Gaussian smoothing scales ($R = 2, 4, 8, 16$ Mpc/h) and isolating specific scale ranges via Starlet transforms.
  • Convergence thresholds: Separating maps into low, mid, and high convergence regions ($\kappa_g$ thresholds).
  • Transformations: Converting data to Fourier space (real/imaginary or amplitude/phase) or applying Starlet transforms.
  • Information removal: Shuffling redshift bins (removing tomographic order), summing redshift bins (removing tomographic depth), and full pixel shuffling (removing all spatial information).

• Performance Metrics: Network performance is quantified using two indicators calculated from Markov Chain Monte Carlo (MCMC) chains:

  1. The standard deviation of the constraint on $S_8$ ($\sigma_{S_8}$).
  2. The information entropy ($H$) of the posterior distribution.

Key Results
The study reveals that the CNN's prediction process relies on a specific mix of Gaussian and non-Gaussian information, with a distinct emphasis on scales at the transition between linear and non-linear regimes.

• Scale Dependence: The network performs best when data includes scales around 8 to 12 Mpc/h, which corresponds to the transition between linear and non-linear regimes at the survey's redshifts.

  • Removing small scales (dominated by noise) has little effect.
  • Providing only large scales ($>16$ Mpc/h) significantly degrades performance, bringing it down to the level of the Power Spectrum network.
  • Providing only small scales ($<10$ Mpc/h) results in slightly worse performance than the reference.
  • Interestingly, removing the smallest ($<5$ Mpc/h) and largest ($>20$ Mpc/h) scales slightly improves performance, likely by allowing the network to focus on the most informative intermediate scales.

• Convergence Regions:

  • Mid-convergence regions (intermediate values, ~60% of pixels) contain the least information on their own.
  • Extreme regions (low convergence/voids and high convergence/peaks) are highly informative.
  • However, combining mid-convergence regions with either extreme significantly improves performance compared to extremes alone. This suggests that while extremes provide non-Gaussian information, they lack large-scale connected structures found in mid-regions.
  • Combining high-convergence regions (peaks) with the Fourier amplitude (Gaussian information) recovers performance equivalent to the reference model, suggesting that a combination of peak counts and power spectrum is a robust strategy.

• Transformations and Spatial Information:

  • Starlet transforms (decomposing images into multi-scale channels) do not degrade performance and may slightly improve it.
  • Fourier space inputs (real/imaginary or amplitude/phase) significantly degrade performance, indicating the CNN architecture is optimized for real-space images.
  • Redshift information: Summing redshift bins (removing tomographic depth) surprisingly causes no significant performance loss in this simplified baseline. However, shuffling the redshift order degrades performance, suggesting the network relies on the correct presentation of data.
  • Spatial information: Full pixel shuffling (removing all spatial structure) causes significant performance loss, reducing the network to a level similar to the Power Spectrum approach, yet the network still produces relevant predictions.

Significance and Claims
The paper claims that its primary contribution is a novel, intuitive method for interpreting deep learning models in cosmology by analyzing the impact of data degradation on constraining power. The authors conclude that:

1. Hybrid Information Requirement: Like classical summary statistics, CNNs require both Gaussian (large-scale) and non-Gaussian (small-scale, peak/void) information to achieve optimal constraining power.
2. Critical Scales: The most critical information for the network lies in structures at the limit between linear and non-linear regimes (8–12 Mpc/h).
3. Validation of Combined Probes: The results support the hypothesis that combining power spectrum analysis (Gaussian) with peak counts or Minkowski functionals (non-Gaussian) is an effective strategy that minimizes information loss.
4. Architectural Sensitivity: The CNN is highly sensitive to the representation of data (preferring real space over Fourier space) and the presence of spatial correlations.

The authors note that these findings are based on a simplified physical model (lacking astrophysical uncertainties like intrinsic alignment, which Appendix A shows can alter results regarding redshift binning). Nevertheless, the study provides a foundational understanding of how deep learning extracts cosmological information from weak lensing surveys, addressing the "black box" concern by identifying the specific physical features the network deems most valuable.