The miniJPAS survey quasar selection V: combined algorithm

Technical Summary: The miniJPAS Survey Quasar Selection V. Combined Algorithm

Problem Statement
Photometric surveys generate vast datasets requiring efficient object classification and redshift estimation to support spectroscopic follow-up and cosmological studies, such as Baryon Acoustic Oscillations (BAO) using Lyman-$\alpha$ absorption. While individual machine learning algorithms have been developed to distinguish quasars from stars and galaxies using miniJPAS data, they often yield slightly different catalogues. The challenge lies in optimally combining these diverse algorithmic outputs to maximize both the purity (minimizing contamination) and completeness (maximizing detection) of the final quasar catalogue, while also improving photometric redshift estimates.

Methodology
This work presents a "combined algorithm" that acts as a meta-classifier, integrating the outputs of eight previously developed machine learning classifiers and two redshift estimators.

1. Data Sources:

   • miniJPAS Data: The analysis utilizes the miniJPAS Data Release, covering $\sim1$ deg$^2$ with 56 narrow-band filters plus 4 SDSS broadband filters. The sample is restricted to $r$-band magnitudes $17.0 < r \le 23.5$ (extended to 24.3 for some tests) and focuses primarily on point-like sources ($ERT \ge 0.1$).
   • Simulated Data (Mocks): Used for training and initial validation. These mocks are based on SDSS spectra convolved with J-PAS filters, containing equal numbers of stars, galaxies, and quasars to facilitate training.
   • Real Data: The DESI Early Data Release (EDR) cross-matched with miniJPAS is used as an independent truth table to validate performance against real observations.

2. Input Features:
   The combined model does not use raw photometric data directly. Instead, it ingests the confidence scores and redshift estimates from eight individual classifiers (CNN1, CNN1NE, CNN2, RF, LGBM, ANN1, ANN2, and SQUEZE) and two template-fitting redshift estimators (QPz and LePhare).

   • Classification: A Random Forest Classifier is trained to predict the final class (star, galaxy, low-$z$ quasar, high-$z$ quasar) based on the confidence outputs of the individual algorithms.
   • Redshift Estimation: A Random Forest Regressor (implemented via XGBoost) is trained to refine redshift estimates. Notably, the training metric is the Normalized Median Absolute Deviation ($\sigma_{NMAD}$) rather than Mean Squared Error, to better handle the non-Gaussian nature of photometric redshift errors. For bright objects ($r < 22$), the QPz estimates are retained as they outperform the ensemble.

3. Evaluation Metrics:
   Performance is assessed using the F1 score (harmonic mean of purity and completeness), purity, completeness, and $\sigma_{NMAD}$ for redshift precision.

Key Contributions

• Ensemble Learning: The paper introduces a second-layer machine learning approach that optimally combines the outputs of eight distinct algorithms, rather than relying on a single model.
• Feature Importance Analysis: The study quantifies the relative importance of the input algorithms, revealing that the CNN1 algorithm dominates the combined decision, with CNN2 and ANN1 providing significant secondary contributions.
• Redshift Refinement: A dedicated regressor is developed to improve redshift estimates, specifically targeting the reduction of catastrophic outliers (line confusion) by leveraging alternative estimates from SQUEZE and others, even when they are less precise than QPz for bright objects.
• Real-Data Validation: Unlike previous works in the series that relied solely on mocks, this paper validates the combined algorithm against the DESI EDR, providing a critical reality check on the simulation-based performance.

Results

• Performance on Mocks:

  • The combined algorithm outperforms all individual classifiers on the mock test sets.
  • For high-redshift quasars ($z \ge 2.1$) at $r \le 23.5$, the combined algorithm achieves an F1 score of 0.88 (Purity: 0.91, Completeness: 0.85).
  • For low-redshift quasars ($z < 2.1$) at $r \le 23.5$, the F1 score is 0.79 (Purity: 0.79, Completeness: 0.79).
  • Redshift precision on mocks yields $\sigma_{NMAD} = 0.11$ overall, with $\sigma_{NMAD} = 0.03$ for high-$z$ and $0.19$ for low-$z$ quasars.

• Performance on Real Data (DESI EDR):

  • When applied to the DESI cross-match, the performance dynamics shift significantly. The combined algorithm does not outperform the individual classifiers; in fact, its performance drops relative to the mock results.
  • Specifically, the CNN1 algorithm, which drives the combined model, shows a significant drop in purity on real data compared to mocks.
  • Conversely, algorithms like SQUEZE and Random Forest (RF), which performed poorly on mocks, show substantial performance gains on real data.
  • Redshift precision on real data is unexpectedly high, with $\sigma_{NMAD} = 0.02$, significantly better than the mock prediction of 0.11.

• Catalogue Production:

  • Two catalogues are released: one for point-like sources (784 quasars) and one including extended sources (11,487 quasars).
  • The authors note that the number of extended sources is unreasonably high, suggesting the combined algorithm is not robust for non-point-like objects.
  • To align with theoretical expectations (Palanque-Delabrouille et al. 2016), a confidence cut of $\ge 0.67$ is recommended for point-like sources, yielding ~298 quasars down to $r=23$.

Significance and Limitations
The paper concludes that while the combined algorithm successfully optimizes the integration of diverse machine learning methods, the discrepancy between mock and real data performance indicates that the simulated mocks are not realistic enough. The authors attribute this to the mocks being based on SDSS spectra and not fully capturing the properties of fainter objects or the specific noise characteristics of the DESI/miniJPAS overlap.

The significance of this work lies in the demonstration that:

1. Ensemble methods can improve quasar selection on simulated data.
2. Current mock data may lead to overfitting or misestimation of algorithm importance when transitioning to real data.
3. A new, more refined version of the mocks is required to fully validate the combined approach and to properly account for extended sources.

The authors modestly state that the data sample is not yet large enough for full statistical consideration and that the current catalogue should be used with the understanding that the performance metrics derived from mocks may not fully translate to real-world applications without further mock refinement.