🔭 astrophysics

Machine Learning Closure Audits for LSST Photometric Supernova Cosmology

This paper introduces a supervised machine learning closure audit using LightGBM models to reveal significant structured residuals in LSST supernova simulations and DES real data that traditional one-dimensional diagnostics miss, thereby providing a robust diagnostic protocol to identify non-closure before unblinding cosmological parameters.

Original authors: Ayan Mitra

Published 2026-07-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ayan Mitra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a cosmic mystery: Why is the universe expanding faster than we thought? To do this, you use "Type Ia supernovae"—exploding stars that act as standard "candles" to measure vast distances.

The problem is that your measurements aren't perfect. There are tiny errors, biases, and "noise" in your data. Scientists have developed a complex set of rules (a "pipeline") to clean up these errors. Usually, they check if the pipeline works by looking at the big picture: "Does the average error look like zero?"

This paper argues that looking at the average is like checking the temperature of a whole room and ignoring the cold draft coming from a specific window. Even if the room feels "average," that draft could ruin your experiment.

Here is a simple breakdown of what the authors did:

1. The Problem: The "Invisible Draft"

The authors looked at simulations of supernovae (computer-generated data that mimics what the LSST telescope will see). They applied the standard cleaning rules.

The Old Check: They looked at the data in simple slices (like sorting stars by how far away they are). In these slices, the errors looked tiny and random. It looked like the cleaning rules worked perfectly.
The New Idea: The authors asked, "What if we use a smart computer (Machine Learning) to look at everything at once? Can the computer predict the remaining errors just by looking at the details of the star's light and its surroundings?"

2. The Test: The "Sherlock Holmes" Audit

They built a "Closure Audit." Think of this as a lie detector test for your data cleaning process.

They fed the computer all the details: how bright the star looked, how noisy the image was, the color of the star, the type of galaxy it lived in, and the quality of the telescope's view.
They asked the computer: "Based on these details, can you guess what the error in our distance measurement is?"

3. The Shocking Result

The Old Check: The simple slices said the errors were random noise (less than 1% predictable).
The New Check: The smart computer said, "I can predict 98% of the remaining errors!"
What this means: The errors weren't random noise. They were structured, organized patterns that the standard cleaning rules missed. It's like the computer found a hidden code in the "mistakes" that the old method completely ignored.

4. The Real-World Check

They didn't just test this on fake computer data. They ran the same test on real data from the Dark Energy Survey (a real telescope project).

Even with real, messy data, the computer could predict 72% of the errors.
This proves that real-world measurements also have these hidden, structured patterns.

5. What Caused the Errors? (The "Clues")

The authors used a tool called SHAP (which acts like a magnifying glass to see which clues the computer used most).

The Top Clues: The most important factors weren't complex physics theories. They were simple things like how bright the star looked and how clear the image was (Signal-to-Noise).
The Analogy: Imagine you are trying to guess how far a car is. The computer realized that if the car looks very dim and the photo is grainy, the distance calculation is likely off. It's not a mystery of the universe; it's a limitation of how the camera sees faint objects.

6. The Solution: A "Scorecard" for the Future

The authors propose a new tool called a Scorecard.

Before: Scientists would just check if the average error was zero.
Now: Before they look at the final, unblinded results of the LSST telescope, they will run this "Scorecard."
How it works: They will compare the "Scorecard" results of the real telescope data against the computer simulations.
- If the real data looks like the simulation, they are good to go.
- If the real data looks different (like a different pattern of errors), they know something is wrong with their cleaning rules before they make any big claims about the universe.

Summary

This paper introduces a smart, high-tech quality control check for supernova astronomy. It shows that even when data looks "clean" on the surface, there can be deep, hidden patterns of error. By using Machine Learning to find these patterns early, scientists can fix their tools before they try to measure the expansion of the universe, ensuring their final answers are truly accurate.

Technical Summary: Machine Learning Closure Audits for LSST Photometric Supernova Cosmology

Problem Statement
Modern and next-generation supernova cosmology analyses, particularly those targeting the Vera C. Rubin Observatory LSST, rely on end-to-end simulations to train classifiers, characterize selection effects, and calibrate distance bias corrections. Standard closure tests typically evaluate whether a pipeline recovers the input cosmology using global Hubble diagram summaries (e.g., redshift-binned means). However, these global metrics can mask multivariate structures in post-correction residuals that remain uncorrected. The paper addresses the question of whether residuals that appear "closed" in one-dimensional redshift summaries remain predictable when a machine learning (ML) model utilizes the full suite of measured observables, including light curve parameters, host galaxy properties, and survey quality metrics.

Methodology
The authors introduce a supervised machine learning closure audit designed to test if measured observables can predict bias-corrected Hubble residuals ( $\Delta\mu$ ). The audit is applied to three distinct datasets:

M23 Mock Data: Independent spectroscopic and photometric redshift samples from Mitra et al. (2023), fitted with SALT2.
M25 Mock Data: A predominantly photometric LSST-like simulation from Mitra et al. (2025), fitted with SALT3, incorporating host galaxy photometric redshifts and the SCONE classification algorithm.
DES 5YR Real Data: The Dark Energy Survey 5-Year spectroscopic Type Ia supernova sample (Vincenzi et al. 2024), fitted with SALT3.

The methodology employs LightGBM models trained on held-out data using a five-fold cross-validation scheme to prevent information leakage. The audit evaluates predictive performance using the out-of-fold coefficient of determination ( $R^2$ ) and Root Mean Square Error (RMSE).

Baseline Metrics: The audit compares ML performance against a null baseline (constant mean) and a redshift-binned mean.
Feature Groups: Features are categorized into four groups to prevent circular reasoning: (1) Target-adjacent standardization features (e.g., $x_1, c, m_B, z_{HD}$ ), (2) Physical environment diagnostics (e.g., host mass), (3) Fit quality and selection limit tracers (e.g., $\chi^2$ , signal-to-noise), and (4) Analysis stage outputs (e.g., bias corrections).
Interpretation: SHAP (SHapley Additive exPlanations) values are computed to rank feature attribution. The authors emphasize that high predictability is a diagnostic indicator of structured residuals, not a proposed distance correction.

Key Results

Predictability of Residuals: Standard redshift-binned diagnostics explain less than 1% of the residual variance ( $R^2 < 0.01$ ) across all datasets, suggesting apparent closure. In contrast, full ML audit models recover up to 98.2% of the variance in the M25 simulations ( $R^2 = 0.982$ ) and 72.5% in the real DES 5YR sample ( $R^2 = 0.725$ ).
Feature Hierarchy: The predictive signal is dominated by apparent magnitudes and signal-to-noise ratios rather than a single unique problem in one passband. In the M25 simulation, the LSST $Y$ -band apparent magnitude is the top-ranked feature, acting as a compact tracer for brightness, distance, and proximity to selection limits.
Consistency Between Simulation and Reality: Independent models trained on M25 and DES 5YR (restricted to 13 shared features) exhibit highly consistent SHAP feature attribution rankings (Spearman $\rho = 0.802$ ). Both models are dominated by apparent magnitude, redshift, color, and stretch.
Ablation Studies: Removing "brightness and measurement quality" features causes the largest drop in predictive performance ( $\Delta R^2 \approx 0.12$ ). Conversely, removing explicit redshift features has negligible impact, as apparent magnitudes and signal-to-noise ratios serve as empirical proxies for redshift in the presence of selection limits.
Robustness: The audit results remain stable across 19 systematic analysis variants (e.g., zero-point shifts, wavelength shifts), with feature rankings remaining consistent (Spearman $\rho \geq 0.99$ ).

Significance and Claims
The paper posits that the closure audit scorecard serves as a diagnostic protocol for comparing mock ensembles with real observations. Its primary significance lies in identifying residual non-closure before cosmological parameters are unblinded. The authors claim that a simulation can recover acceptable global summaries while retaining scientifically important conditional residual structures that standard tests miss.

The proposed operational framework suggests that if a real data metric differs from the mock ensemble mean by more than twice the mock-to-mock scatter ( $> 2\sigma_{mock}$ ), it should trigger a "closure failure" flag. This flag would initiate controlled, one-at-a-time simulation checks (e.g., varying dust models or host mass evolution) rather than immediate machine learning corrections to the Hubble diagram. The authors explicitly state that this audit is a diagnostic layer, not a replacement for distance estimators, and that high predictability indicates structured residuals requiring targeted modeling checks.