Two Point Correlation Function Estimation with Contaminated Data

1. Problem Statement

The two-point correlation function (2PCF) is a fundamental statistic in cosmology used to quantify the clustering of galaxies and constrain models of dark matter, dark energy, and structure formation. The standard method for estimating the 2PCF is the Landy–Szalay (LS) estimator, which relies on counting pairs of objects in data-data (DD), data-random (DR), and random-random (RR) configurations.

The Core Challenge:
In modern wide-field imaging surveys (e.g., LSST, Euclid, Roman), the "working catalog" used for analysis is constructed from noisy measurements and automated pipelines. This leads to two critical issues:

1. Contamination: Non-target objects (e.g., stars, quasars, or artifacts) are misclassified as target galaxies.
2. Incompleteness: True target galaxies are missed due to measurement errors or selection cuts.

These errors are rarely spatially uniform; they correlate with survey depth, observing conditions, and foreground structures. Consequently, standard LS estimators applied to noisy catalogs produce biased 2PCF measurements (spurious large-scale power or suppressed clustering).

Limitations of Existing Solutions:

• Spectroscopic-only samples: Using only high-fidelity spectroscopic data removes bias but suffers from massive variance due to small sample sizes.
• Forward modeling: Explicitly modeling systematics is computationally expensive and requires unverifiable assumptions about error distributions.
• Cross-correlation decontamination (CCD): Requires access to a "pure" contaminant catalog and precise purity estimates, which are often unavailable.
• Global corrections: Simple global purity factors fail to correct for spatially varying systematic errors.

2. Methodology: Prediction-Powered Landy–Szalay (PP-LS)

The author proposes the Prediction-Powered Landy–Szalay (PP-LS) estimator. This method integrates Prediction-Powered Inference (PPI) with the standard LS framework to correct for label noise without requiring explicit modeling of the contamination mechanism.

Key Assumptions:

• Source Catalog ($S$): A large catalog of objects with noisy inclusion labels $\tilde{Y}_i \in \{0, 1\}$ (or soft probabilities).
• Gold-Standard Subset ($L$): A small, simple random sample ($m \ll n$) of objects with exact, error-free true labels $Y_i$ (e.g., from high-resolution spectroscopy).
• Random Catalog ($R$): A synthetic catalog representing the survey geometry and selection function (standard LS requirement).

The Algorithm:
The method decomposes the true pair count into components based on the residual between the true label ($Y$) and the noisy label ($\tilde{Y}$).

1. Residual Definition: For labeled objects $i \in L$, define the residual $\Delta_i = Y_i - \tilde{Y}_i$. For unlabeled objects, $\Delta_i = 0$.
2. Decomposition: The true pair product $Y_i Y_j$ is expanded as:
   $$Y_i Y_j = \tilde{Y}_i \tilde{Y}_j + \Delta_i \tilde{Y}_j + \tilde{Y}_i \Delta_j + \Delta_i \Delta_j$$
3. Horvitz–Thompson Scaling: The terms involving the residuals ($\Delta$) are estimated using the labeled subset $L$ and scaled by the inverse of the sampling probability ($n/m$ or $n(n-1)/m(m-1)$).
   • $T_0$: Counts pairs where both have noisy labels (computed on full sample).
   • $T_1$: Cross-terms involving one residual and one noisy label (computed on $L \times S$).
   • $T_2$: Terms involving two residuals (computed on $L \times L$).
4. Reconstruction: The PP-LS estimator reconstructs the unbiased DD, DR, and RR counts by combining these scaled terms, effectively "de-biasing" the pair counts using only the small labeled subset.

Advantages:

• No Calibration Required: Does not require known misclassification rates, class priors, or calibrated probabilities.
• No Spatial Model: Does not require an explicit model of how contamination varies across the sky.
• Computational Efficiency: Requires only a few additional weighted pair counts, making it compatible with existing pipelines like TreeCorr and Corrfunc.

3. Key Contributions

1. Theoretical Framework: Formalizes the estimation of 2PCF under noisy inclusion labels using a design-based approach. Proves that under simple random sampling of the labeled subset, the PP-LS estimator is design-unbiased and consistent with the "oracle" estimator (which would use true labels for all objects).
2. Algorithmic Innovation: Adapts PPI to pairwise U-statistics, creating a self-contained correction mechanism that preserves the geometric advantages of the Landy–Szalay estimator.
3. Robustness: Demonstrates that the method works even when contamination and misclassification are spatially structured (e.g., gradients, hotspots), a scenario where naive estimators fail.
4. Implementation Guide: Provides a step-by-step algorithm (Algorithm I) showing how to implement PP-LS using standard pair-counting engines with minimal code modification.

4. Results

The paper validates the method using controlled synthetic simulations (Thomas processes for clustering and inhomogeneous Poisson processes for contaminants).

• Bias Reduction:

  • Naive LS: Exhibits significant bias (up to $O(10^{-1})$) at small scales due to clustered contaminants and large-scale bias due to gradients.
  • PP-LS: Successfully removes this bias, closely tracking the oracle (true-label) 2PCF across all scales.
  • Cross-Correlation Decontamination (CCD): Performs well only when a pure contaminant catalog is available; otherwise, it is less robust than PP-LS.

• Variance and Efficiency:

  • Spectroscopic-Only: Unbiased but has extremely high variance (orders of magnitude worse than PP-LS) due to small sample size.
  • PP-LS: Achieves a variance significantly lower than spectroscopic-only methods (by an order of magnitude or more in low-label regimes) while maintaining unbiasedness. It effectively interpolates between the high-variance spectroscopic limit and the low-variance (but biased) photometric limit.
  • Label Error Sensitivity: PP-LS remains robust even with moderate classification errors in the labeled subset, though variance increases as error rates rise.

• Scaling: The variance of PP-LS scales favorably with the size of the labeled subset, offering substantial gains even when only $\sim 1\%$ of the data is labeled.

5. Significance and Impact

• Enabling Next-Generation Surveys: As upcoming surveys (LSST, Euclid, Roman) generate massive photometric datasets with limited spectroscopic follow-up, PP-LS provides a statistically principled way to utilize the full dataset without sacrificing accuracy.
• General Applicability: While framed in the context of galaxy clustering, the formalism applies to any astrophysical population defined by a binary inclusion rule (e.g., quasars, clusters) suffering from noisy labels.
• Paradigm Shift: Moves away from complex forward modeling of systematics toward a "correction-by-design" approach that leverages small, high-fidelity subsets to debias large, noisy datasets.
• Practicality: The method is lightweight and integrates directly into existing cosmological analysis pipelines, lowering the barrier to adoption for the community.

In conclusion, the PP-LS estimator offers a robust, unbiased, and computationally efficient solution for measuring the two-point correlation function in the presence of spatially structured contamination, bridging the gap between the statistical power of large photometric surveys and the fidelity of spectroscopic data.