Calibrating redshift distributions at $z>2$ with Lyman-$\alpha$ forest cross-correlations

This paper demonstrates that cross-correlating photometric galaxies with Lyman-$\alpha$ forest data using a novel theoretical framework can reliably calibrate the redshift distribution of high-redshift ($z>2$) galaxy samples, achieving a mean redshift precision of $\sigma_z/(1+\bar{z}) = 0.006$ at $\bar{z}=2$ in simulations of future DESI and LSST surveys.

The Gist

Imagine you are trying to map a vast, foggy city at night. You have a list of addresses for millions of people (photometric galaxies), but the list only gives you a rough idea of where they are, not exactly how far away they are. In astronomy, knowing the exact distance (redshift) to these objects is crucial; without it, your map of the universe is blurry, and you can't measure how the universe is expanding or what dark energy is doing.

This paper is about a clever new way to cut through that fog for the most distant, hardest-to-see parts of the city (galaxies at a distance corresponding to $z > 2$).

The Problem: The "Foggy" High-Altitude View

Astronomers usually try to guess distances by looking at the color of the light from galaxies. It's like guessing how far away a streetlight is by how orange or blue it looks. But for very distant, faint galaxies, this method gets messy. The "fog" (uncertainty) gets so thick that the map becomes unreliable.

To fix this, scientists usually use a "reference map" made of objects whose distances are known perfectly (spectroscopic galaxies). They look at how the fuzzy galaxies cluster around the sharp ones to figure out the fuzzy ones' distances. However, for the most distant galaxies, there are very few "sharp" reference objects available. It's like trying to map a remote mountain range, but you only have a few lighthouses to guide you, and they run out of steam before you reach the peak.

The Solution: The "Forest" of Absorption

The authors propose using a different kind of reference map: the Lyman-alpha Forest.

Imagine looking at a distant lighthouse (a quasar) through a thick forest. The trees (hydrogen gas clouds) block some of the light, creating a pattern of shadows on the beam.

• The Quasar: The bright lighthouse in the distance.
• The Forest: The gas clouds between us and the lighthouse.
• The Shadows: The specific wavelengths of light that get absorbed by the gas.

Because the universe is expanding, the "shadows" from gas clouds at different distances appear at different colors (wavelengths) in the lighthouse's beam. By analyzing this "forest" of shadows, astronomers can create a 3D map of the gas distribution in the universe. This gas acts as a giant, invisible net that catches the light from the distant galaxies.

The Innovation: Cleaning the Lens

The tricky part is that the "lighthouse" itself (the quasar) has a natural brightness curve that changes over time. To see the "shadows" of the forest clearly, you have to subtract the lighthouse's natural glow.

• Old Method (Picca): Imagine trying to guess the lighthouse's natural glow by looking at the whole beam, including the shadows. You might accidentally erase some of the important shadow patterns, making the map blurry.
• New Method (LyCAN): The authors use a new AI tool called LyCAN. It's like a smart filter that looks only at the part of the beam before the forest starts to predict what the lighthouse's natural glow should look like. This allows them to subtract the glow perfectly without erasing the important shadow patterns.

The Experiment: Building a Virtual Universe

To test if this works, the team didn't just look at real data; they built a virtual universe (simulations) using supercomputers.

1. They created a fake universe with millions of fake galaxies and quasars.
2. They simulated the "shadows" in the light, including realistic noise and errors (like bad weather or a dirty lens).
3. They applied their new AI method (LyCAN) to this fake data to see if they could recover the true distances of the fake galaxies.

The Results: A Clearer Picture

The results were impressive:

• Signal Strength: They detected the connection between the galaxies and the gas forest with extremely high confidence (24 times stronger than random noise).
• Precision: They could pin down the average distance of the galaxy group with a precision of about 0.6%. This is good enough to satisfy the strict requirements for future major surveys like the Rubin Observatory (LSST).
• The "Best" Angle: They found that looking at the clustering on specific scales (about 10 arcminutes across the sky) gave the best results. It's like finding the perfect zoom level on a camera to see the pattern clearly.

Why This Matters

This paper proves that we can use the "forest" of gas clouds to calibrate the distances of the most distant galaxies in the universe.

• Before: We were flying blind in the high-redshift zone, unsure if our maps were accurate.
• Now: We have a reliable "GPS" using the Lyman-alpha forest.

This is a game-changer for Stage IV cosmology (the next generation of massive sky surveys). It means that when we look at the edge of the observable universe to study dark energy or the early formation of galaxies, we won't be guessing their distances anymore. We'll know exactly where they are, allowing us to build a much more accurate map of our cosmic history.

In short: The authors found a way to use the "shadows" of ancient gas clouds, cleaned up by a smart AI, to act as a ruler for measuring the most distant galaxies in the universe.

Technical Summary

Here is a detailed technical summary of the paper "Calibrating redshift distributions at $z> 2$ with Lyman-$\alpha$ forest cross-correlations" by Hang et al. (2026).

1. Problem Statement

Photometric galaxy surveys (e.g., LSST, Euclid) rely on accurate redshift distributions ($n(z)$) for cosmological probes like weak gravitational lensing. While photo-$z$ estimates and self-organizing maps work well for lower redshifts, they struggle with the high-redshift tail ($2 < z < 3$) due to a lack of spectroscopic training data and incomplete spectral energy distribution (SED) templates.

• The Gap: Traditional "clustering redshift" methods, which cross-correlate photometric galaxies with spectroscopic reference samples, fail at $z > 2$ because the number density of standard spectroscopic tracers (like Emission Line Galaxies and Quasars) drops sharply, and magnification bias complicates the signal.
• The Opportunity: The Lyman-$\alpha$ (Ly$\alpha$) forest—absorption features in quasar spectra caused by intergalactic neutral hydrogen—provides a dense, continuous tracer of the large-scale structure at $z > 2$.
• The Challenge: Using Ly$\alpha$ for clustering redshifts is difficult because the standard continuum fitting methods (e.g., Picca) suppress large-scale radial modes, distorting the correlation signal and rendering the standard clustering redshift estimators invalid.

2. Methodology

Simulations and Data

The authors utilized CoLoRe simulations to generate mock catalogs for:

• Reference Sample: DESI 5-year Ly$\alpha$ forest data (quasar sightlines).
• Unknown Sample: Rubin Observatory LSST 10-year photometric galaxies (specifically the highest redshift tomographic bin).
• Setup: 10 simulation boxes covering a joint area of 5,492 deg$^2$, with galaxies in the range $z \in [0, 3]$ and Ly$\alpha$ forests in $z \in [1.8, 3]$.

Continuum Fitting Strategies

The paper compares three approaches to estimating the transmitted flux fluctuation $\delta_F$:

1. Raw/True Continuum: The ground truth from simulations (no noise or fitting errors).
2. Picca: The standard code used in DESI/eBOSS BAO analysis. It fits a universal template but introduces distortions by suppressing large-scale modes.
3. LyCAN: A machine-learning-based method (Turner et al. 2024) that predicts the continuum using only flux red-ward of the Ly$\alpha$ emission line. Crucially, it preserves large-scale modes, making it suitable for clustering redshifts.

Theoretical Framework

The authors developed a new theoretical framework to model the angular cross-correlation $w_{gF}(\theta)$ directly, rather than relying on the conventional $n(z) \propto w_{ru}/\sqrt{w_{rr}}$ estimator.

• Modeling: They derived the expected cross-correlation including Redshift-Space Distortions (RSD) for the Ly$\alpha$ forest, which are significant at these scales.
• Bias Mitigation: Since the cross-correlation constrains the product of bias and redshift distribution ($b(z)n(z)$), they tested two models:
  • Shift Mean: Assumes the shape is known, only the mean redshift ($\delta_z$) is unknown.
  • Gaussian Process (GP): A non-parametric approach allowing the shape of $b(z)n(z)$ to vary around a template.

Measurement

• Estimator: Used a mean-subtracted estimator with a random catalog to remove additive biases caused by the survey footprint.
• Scales: Measured angular cross-correlations in logarithmic bins from $\theta = 1$ to $50$ arcmin.
• Inference: Used a likelihood analysis with Jackknife covariance matrices to constrain the redshift distribution parameters.

3. Key Contributions

1. Feasibility of Ly$\alpha$ Clustering Redshifts: Demonstrated that Ly$\alpha$ forests can serve as a high-fidelity reference sample for calibrating photometric galaxies at $z > 2$, a regime previously inaccessible to clustering methods.
2. Critical Role of Continuum Fitting: Showed that standard continuum fitting (Picca) destroys the large-scale signal required for clustering redshifts, reducing the Signal-to-Noise Ratio (SNR) by ~40%. Conversely, the ML-based LyCAN method preserves the signal, enabling high-precision measurements.
3. Theoretical Formalism: Provided a rigorous derivation for modeling angular cross-correlations between Ly$\alpha$ forests and galaxies, explicitly accounting for RSD and the specific weighting of Ly$\alpha$ pixels, which differs from standard galaxy-galaxy clustering.
4. Optimization of Analysis Parameters: Identified that:
   • Small angular scales ($\theta \sim 10-15$ arcmin) maximize SNR.
   • Redshift bin sizes ($\Delta z$) of 0.1 are sufficient; finer binning does not significantly improve constraints due to the featureless nature of the high-$z$ tail.
   • Contamination (DLAs, BALs) has a minor impact (~5% error increase) if the effective bias is known.

4. Key Results

• Detection Significance: In the baseline scenario (LyCAN mocks, $\Delta z = 0.1$, $\theta \approx 10.5-13.6$ arcmin), the method achieves a 24$\sigma$ detection of the cross-correlation signal.
• Redshift Calibration Precision:
  • Shift Mean Model: Constrains the mean redshift to $\sigma_z / (1+\bar{z}) = 0.014$ (assuming no knowledge of $z<2$).
  • Gaussian Process Model: Assuming the distribution shape is known for $z<2$, the constraint tightens to $\sigma_z / (1+\bar{z}) = 0.006$ at $\bar{z}=2$. This approaches the LSST Dark Energy Science Collaboration (DESC) requirement of 0.001.
• Impact of Continuum Fitting:
  • Picca: Reduced SNR to 14$\sigma$ and introduced shape distortions.
  • LyCAN: Maintained high SNR (24$\sigma$) and accurate signal shape.
  • True Continuum: Achieved the theoretical maximum of 30$\sigma$.
• Contamination: Including astrophysical contaminants (DLAs, BALs, metals) degraded the SNR by only ~10% (22$\sigma$), provided the bias change is accounted for.
• Scaling: The SNR scales with the square root of the survey area ($\sqrt{A}$) and is dominated by Ly$\alpha$ forest noise rather than galaxy shot noise, provided the galaxy density is above a certain threshold.

5. Significance and Future Outlook

• Stage IV Surveys: This method offers a robust solution for calibrating the high-redshift tails of LSST and Euclid, which are critical for constraining dark energy and neutrino masses. Without this, the uncertainty in the mean redshift of the highest bins would degrade cosmological parameter constraints.
• Beyond Ly$\alpha$: The paper suggests that future improvements could come from combining Ly$\alpha$ with other high-$z$ tracers (e.g., metal forests, Lyman-break galaxies) or using DESI-II to increase quasar density.
• Limitations: The current study is a proof-of-concept. Future work must address:
  • Simulation limitations at small scales (currently cutting off at $\theta > 10$ arcmin).
  • The use of synthetic data vectors due to unexpected fluctuations in mocks.
  • Joint analysis with lower-redshift tracers to break degeneracies between bias and redshift distribution.

Conclusion: The paper establishes that Ly$\alpha$ forest cross-correlations, when paired with advanced continuum fitting (LyCAN), are a reliable and promising tool for calibrating photometric redshifts at $z > 2$, potentially reducing mean redshift uncertainties to levels required for next-generation cosmology.