🔭 astrophysics

Denoising clustering covariance matrices with Rotational Invariant Estimators

This paper introduces the Rotational Invariant Estimator (RIE) as a superior method for denoising covariance matrices in galaxy clustering analyses, demonstrating that it significantly stabilizes best-fit recovery and reduces bias compared to standard sample covariance and NERCOME, particularly in Fourier space when the number of mock catalogs is limited.

Original authors: Antonio Farina, Massimo Guidi, Alfonso Veropalumbo, Claudio Guida

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonio Farina, Massimo Guidi, Alfonso Veropalumbo, Claudio Guida

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a cosmic mystery: What is the true nature of the universe? To do this, you look at how galaxies are clustered together. But here's the catch: your "evidence" (the data) is noisy, and you don't have a perfect map of how that noise behaves.

In statistics, this "map of noise" is called a Covariance Matrix. It tells you how much the data points wiggle and how they wiggle together. If you get this map wrong, your detective work (estimating the universe's properties) will lead you to the wrong conclusions.

The Problem: The "Too Few Clues" Dilemma

Usually, to build this noise map, astronomers run thousands of computer simulations (mocks) to see how the universe could look. They compare these simulations to their real data.

However, modern telescopes are so powerful that they collect massive amounts of data (a huge "data vector"). Sometimes, the number of simulations you can afford to run is almost the same size as your data.

The Analogy: Imagine trying to guess the average height of everyone in a stadium by measuring only 10 people. If the stadium has 10,000 seats, your guess will be wild and full of errors.
The Scientific Term: This is the "curse of dimensionality." When the number of data points ( $D$ ) gets close to the number of simulations ( $N$ ), the standard math used to calculate the noise map breaks down. It becomes "noisy" and biased, leading to unstable answers.

The Solution: Three Different "Noise Cleaners"

The authors of this paper tested three different ways to fix this noisy map:

The Standard Method (Sample Covariance): This is the "naive" approach. It just averages the simulations.
- The Flaw: When data is scarce, it's like trying to hear a whisper in a hurricane. The result is shaky. Sometimes it says the answer is very certain, and other times it says it's very uncertain, even if the truth hasn't changed. This is known as the Dodelson-Schneider effect.
NERCOME (The Mixer): This method tries to fix the noise by mixing and matching different subsets of simulations. It's like taking a few blurry photos, cutting them up, and reassembling them to find the clearest picture.
- The Result: It helps, but it's a bit of a gamble. Depending on the type of data, it sometimes makes the uncertainty look too small (overconfident) or too big (underconfident).
RIE (The Rotational Invariant Estimator): This is the star of the show. It's a fancy mathematical tool based on Random Matrix Theory (a branch of math that studies patterns in huge, random numbers).
- The Analogy: Imagine you have a very noisy radio signal. The RIE doesn't just turn up the volume; it listens to the structure of the static. It knows that true signals have a specific "shape" (spectrum), while random noise looks like static. It mathematically "denoises" the signal, stripping away the random static while keeping the true cosmic melody intact.

The Experiment: Configuration vs. Fourier Space

The authors tested these methods on two different ways of looking at the galaxy data:

Configuration Space (2PCF): Looking at the distances between galaxies (like measuring how far apart chairs are in a room).
Fourier Space (Power Spectrum): Looking at the "waves" of density in the universe (like analyzing the sound waves of a song rather than the individual notes).

The Results:

In the "Room" (Configuration Space): Both the mixer (NERCOME) and the denoiser (RIE) helped stabilize the results, but they sometimes made the "uncertainty bars" look too tight, as if they were too confident in their answers.
In the "Sound Waves" (Fourier Space): This is where RIE shined.
- The standard method was all over the place.
- NERCOME was okay but still a bit shaky.
- RIE was a champion. Even when they had very few simulations (almost as few as the data points), RIE produced results that were almost identical to what you would get if you had infinite simulations. It stabilized the "best fit" (the detective's final conclusion) perfectly.

The Big Takeaway

The paper concludes that RIE is the best tool for the job, especially when analyzing complex, wave-like data (Fourier space).

Why it matters: Future telescopes (like the Euclid mission or DESI) will generate data so huge that we won't be able to run enough simulations to use the old, standard methods.
The Future: RIE allows astronomers to get accurate answers even when they are "short on clues." It's like having a super-powered noise-canceling headset that lets you hear the universe clearly, even when the background static is overwhelming.

In short: The authors found a new, mathematical "noise filter" that lets cosmologists solve the universe's mysteries with high precision, even when they don't have enough computer simulations to do it the old-fashioned way.

1. Problem Statement

In precision cosmology, inferring parameters from galaxy clustering (e.g., via the Two-Point Correlation Function or Power Spectrum) relies heavily on the accuracy of the covariance matrix ( $C$ ) and its inverse, the precision matrix ( $C^{-1}$ ).

The Dimensionality Curse: In practice, $C$ $C$ is estimated from a finite number of mock catalogs ( $N$ $N$ ). When the dimension of the data vector ( $D$ $D$ ) becomes comparable to $N$ $N$ (i.e., the ratio $q = N/D$ $q = N / D$ is small), the sample covariance estimator suffers from severe issues:
1. Bias in Precision: The naive inverse of the sample covariance is biased (overestimating precision), requiring corrections like the Hartlap factor.
2. Stochastic Shifts (Dodelson–Schneider Effect): Even with bias correction, sampling noise causes random shifts in the location of best-fit parameters and inflates the scatter of these estimates.
3. Uncertainty in Error Volumes: The estimated volume of the posterior distribution (Figure of Merit, FoM) often deviates from the truth.
Current Limitations: Existing solutions include theoretical models (limited to linear regimes), data compression, or non-linear shrinkage estimators like NERCOME (Non-parametric Eigenvalue Regularised Covariance Matrix Estimator). However, the performance of these shrinkage methods varies significantly depending on the statistical probe (configuration vs. Fourier space) and the specific structure of the covariance matrix.

2. Methodology

The authors introduce the Rotational Invariant Estimator (RIE), proposed by Bun et al. (2016), to the field of cosmological galaxy clustering for the first time.

Theoretical Framework:
- RIE: Unlike NERCOME, which mixes subsets of data, RIE leverages Random Matrix Theory (RMT) to analytically denoise the eigenvalue spectrum of the covariance matrix. It assumes the true covariance shares an eigenvector basis with the noisy sample but corrects the eigenvalues based on the "overlap" between noisy and true eigenvectors.
- Implementation: The authors apply RIE to the correlation matrix first (to isolate the regularization of the correlation structure) and then rescale it to the original variances.
Experimental Setup:
- Data: Synthetic datasets for the 2PCF (configuration space) and Power Spectrum (Fourier space) based on a DESI-like cosmology ( $z=1.317$ ).
- Ground Truth: The true covariance matrices are known analytically (Gaussian), allowing for a controlled benchmark.
- Variables: The study varies the mock-to-dimension ratio $q = N/D$ (from 1.2 to 16) and the data-vector size $D$ (90, 180, 270 bins).
- Comparators: The RIE is benchmarked against the standard Sample Covariance (corrected with Hartlap and Percival factors) and NERCOME.
Inference & Metrics:
- Bayesian inference is performed using an EFT-based model and the Nautilus sampler.
- Performance is quantified using:
  - Figure of Bias (FoB): Deviation of best-fit parameters from true values.
  - Figure of Merit (FoM): Inverse of the posterior volume (precision of constraints).

3. Key Contributions

First Application of RIE in Cosmology: This work pioneers the use of RIE for large-scale structure parameter inference, demonstrating its potential to outperform traditional shrinkage methods.
Probe-Dependent Performance Analysis: The paper provides a rigorous comparison showing that the effectiveness of denoising estimators is highly dependent on whether the analysis is performed in configuration space (2PCF) or Fourier space (Power Spectrum).
Stabilization of Best-Fit Recovery: The study demonstrates that RIE significantly suppresses the stochastic shifts in best-fit parameters caused by the Dodelson–Schneider effect, even when the number of mocks barely exceeds the data dimension ( $q \approx 1.2$ ).

4. Key Results

A. Configuration Space (2PCF)

Best-Fit Stability: Both NERCOME and RIE reduce the scatter in best-fit parameters compared to the sample estimator.
Uncertainty Bias: Both shrinkage estimators tend to underestimate the posterior volume (overestimate constraints), leading to an inflated FoM. This bias grows with the data-vector dimension $D$ .
Conclusion: In 2PCF, while RIE stabilizes the best-fit location, it produces overly tight constraints that do not match the true uncertainty volume.

B. Fourier Space (Power Spectrum)

Best-Fit Stability: RIE shows markedly superior performance. It maintains low FoB (bias) across all $q$ values, significantly outperforming both the sample estimator and NERCOME.
Uncertainty Bias:
- NERCOME: Overestimates constraining power (underestimates uncertainties), consistent with previous findings in weak lensing.
- RIE: Exhibits only a mild underestimation of the FoM (slightly broader posteriors than the truth) but remains remarkably close to the reference.
Robustness: Even at $q=1.2$ (where $N$ is only slightly larger than $D$ ), RIE reproduces the reference posterior contours in both shape and position, whereas sample and NERCOME estimators show significant degradation.

C. Mechanism of Success

The authors hypothesize that the superior performance of RIE in Fourier space is due to the structure of the covariance matrix. Power spectrum data vectors typically have a smoother, more diagonally dominated covariance structure, which aligns well with the shrinkage model assumptions of RME. In contrast, the 2PCF covariance is more complex, leading to larger biases in the estimated volume.

5. Significance and Future Outlook

High-Dimensional Statistics: As cosmology moves toward higher-order statistics (e.g., bispectrum, 3PCF), the data-vector dimension $D$ explodes (often $O(10^2)$ bins). Obtaining enough mocks ( $N \gg D$ ) for a stable sample covariance becomes computationally prohibitive.
Practical Application: RIE emerges as the most effective tool for these regimes, capable of providing reliable parameter inference even when $q \approx 1$ .
Future Work: The authors suggest that while RIE stabilizes the location of the best-fit, the volume of the posterior must be carefully calibrated for each specific statistic. Future work should test RIE on non-Gaussian covariances, survey geometry effects, and joint analyses of 2-point and higher-order statistics.

Conclusion: The Rotational Invariant Estimator (RIE) is a powerful denoising technique that stabilizes cosmological parameter inference in the "low-mock" regime. While its impact on posterior volumes requires careful validation, it is particularly effective in Fourier space, offering a robust solution for the high-dimensional challenges of next-generation surveys like Euclid and DESI.