Euclid preparation. LXXXIX. Accurate and precise data-driven angular power spectrum covariances

1. Problem Statement

The Euclid Wide Survey aims to map billions of galaxies to measure clustering and weak-lensing angular power spectra. The resulting data vector is expected to contain approximately $10^5$ elements (combining auto- and cross-spectra for clustering and E/B modes across multiple tomographic bins).

• The Challenge: Estimating the covariance matrix for such a high-dimensional vector is computationally prohibitive using standard external mock catalogs (requiring $>10^5$ simulations). Internal data-driven methods, specifically Jackknife resampling, are preferred because they require no assumptions about cosmology or galaxy bias.
• Limitations of Standard Jackknife:
  1. Singularity: With a data vector length $N_{data} \approx 10^5$, a standard Jackknife requires $N_{JK} > N_{data}$ to produce a non-singular (invertible) covariance matrix. In practice, computational limits restrict $N_{JK}$ to $\sim 10^2$, leading to singular matrices that cannot be inverted for likelihood analysis.
  2. Bias: Jackknife estimates of variance are known to be biased high (Efron & Stein 1981), leading to overestimated error bars.
  3. Partitioning Issues: Traditional sky partitioning methods (e.g., k-means clustering) produce segments with significant area disparities (up to 16% standard deviation), introducing unwanted variance dependent on segment size.
  4. Partial-Sky Effects: Removing segments alters the survey footprint, introducing mode coupling that biases the angular power spectra ($C_\ell$) if not corrected.

2. Methodology

The authors propose a comprehensive pipeline called DICES (Debiased Internal Covariance Estimation with Shrinkage) to address these issues.

A. Sky Partitioning: Binary Space Partitioning (BSP)

To replace k-means clustering, the authors adapted the Binary Space Partitioning (BSP) algorithm for the unit sphere.

• Algorithm: The survey footprint (HEALPix map) is recursively split by hyperplanes.
  1. Compute the barycenter of the current segment.
  2. Rotate the segment so the barycenter is at the pole.
  3. Identify the "longest side" (maximum angular extent).
  4. Rotate to align this side with the equator.
  5. Find a dividing plane that splits the segment area as equally as possible based on a target weight ratio.
• Advantage: This ensures Jackknife segments are equal in area to high precision (standard deviation of area reduced from 16% in k-means to 0.11% in BSP), minimizing area-dependent variance.

B. Partial-Sky Correction

When a Jackknife segment is removed, the effective footprint changes, altering the mode coupling.

• Correction: The authors transform the pseudo-$C_\ell$ into real-space correlation functions ($C(\theta)$).
• Damping: They apply a multiplicative correction using the ratio of the full-sky mask correlation to the Jackknife mask correlation. To prevent numerical instability where the Jackknife mask correlation approaches zero, a logistic damping function is applied to suppress signals in regions with no baselines.

C. Linear Shrinkage

To handle the singularity issue caused by $N_{JK} \ll N_{data}$:

• Technique: The noisy empirical covariance ($C_{est}$) is shrunk towards a well-conditioned target covariance ($C_{tar}$).
• Target: The target is the Gaussian prediction of the covariance (block-diagonal structure based on the measured power spectra).
• Shrinkage Intensity ($\lambda$): The optimal scalar shrinkage intensity is calculated to minimize the mean squared error between the shrunk estimator and the true covariance. The authors found scalar shrinkage (a single $\lambda$ for the whole matrix) to be the most stable and accurate.
• Result: This produces a non-singular, invertible covariance matrix, though it retains the Jackknife bias in the diagonal elements.

D. Bias Correction (Delete-2 Jackknife)

To correct the high bias in the diagonal elements (variances):

• Technique: The authors implement a delete-2 Jackknife correction. This involves computing statistics where two segments are removed simultaneously.
• Formula: A correction term is derived based on the difference between the full sample, delete-1 samples, and delete-2 samples.
• Limitation: While this successfully debiases the diagonal elements (standard deviations), the off-diagonal elements (correlations) of the pure delete-2 estimator remain extremely noisy and singular.

E. The DICES Synthesis

The final DICES estimator combines the strengths of the previous steps:
$$C_{DICES}^{(ij)} = C_{shr}^{(ij)} \times \frac{\sqrt{C_{debias, (ii)}^{JK} C_{debias, (jj)}^{JK}}}{\sqrt{C_{shr}^{(ii)} C_{shr}^{(jj)}}}$$

• It takes the correlation structure (off-diagonals) from the Shrunk covariance (which is non-singular and low-noise).
• It replaces the diagonal elements (variances) with the Debiased Jackknife standard deviations.

3. Key Contributions

1. SkySegmentor Package: Development of a public Python package implementing the BSP algorithm for creating equal-area Jackknife segments on arbitrary survey footprints.
2. DICES Framework: A novel hybrid estimator that solves the singularity problem (via shrinkage) and the bias problem (via delete-2 correction) simultaneously.
3. Partial-Sky Correction: A robust method for correcting Jackknife $C_\ell$ estimates using real-space masking and logistic damping to handle numerical instabilities.
4. Validation: Rigorous testing on 1,000 synthetic Euclid-like lognormal catalogues (GLASS simulations) mimicking the Euclid DR1 footprint.

4. Results

• Accuracy: DICES produces accurate, non-singular covariance estimates.
  • Relative error in the covariance matrix improved by 33% compared to standard Jackknife estimates.
  • Relative error in the correlation structure improved by 48%.
• Singularity: The method successfully produces invertible matrices even with $N_{JK} = 74$ (far fewer than the data vector length of $\sim 210$ in the test, and orders of magnitude fewer than required for a full Euclid vector).
• Bias Removal: The diagonal elements (standard deviations) are unbiased and match the sample covariance (ground truth), whereas standard Jackknife estimates were biased high.
• Signal-to-Noise Ratio (SNR):
  • DICES yields SNR values consistent with the sample covariance for weak lensing (E and B modes).
  • For clustering, DICES removes the bias seen in standard Jackknife estimates, though a slight overestimation remains, likely due to noise in the correlation matrix estimation at low $N_{JK}$.
• Dependence on $N_{JK}$: Increasing the number of Jackknife segments improves the precision of the covariance and the stability of the shrinkage intensity, but DICES remains robust even at lower $N_{JK}$.

5. Significance

• Model Independence: DICES provides a way to estimate covariances directly from data without relying on specific cosmological models or galaxy bias assumptions, which is critical for validating systematic effects in the Euclid survey.
• Feasibility for Large Surveys: It enables the use of internal covariance estimation for massive data vectors ($10^5$ elements) where generating sufficient external mocks is computationally impossible.
• Validation Readiness: The method allows for rigorous validation of Euclid's weak lensing and clustering measurements against observational systematics (e.g., testing correlations with systematic effects via angular cross-spectra).
• Future Applications: The methodology is applicable to other large-scale structure surveys (e.g., LSST, Roman) and can be refined for future data releases with larger sky coverage. The authors note that while the current implementation uses scalar shrinkage, future work may explore non-linear shrinkage or better targets as data volume increases.

In summary, DICES represents a critical advancement in statistical data analysis for cosmology, providing a robust, unbiased, and computationally feasible solution for covariance estimation in the era of big-data astronomy.