Computational references are not experiments: pre-registered validation of machine-learned sodium-cathode voltages

Technical Summary: Pre-registered Validation of Machine-Learned Sodium-Cathode Voltages

Problem Statement
The field of computational materials discovery, particularly for battery cathodes, relies heavily on machine learning (ML) models trained and validated against computed reference voltages (typically GGA or GGA+U values from databases like the Materials Project). A critical, often unaddressed issue is that these reference scales carry systematic errors relative to experimental electrochemical measurements. Consequently, an ML model can achieve high accuracy against its training references while failing to predict real-world voltages. This paper investigates this "reference error" problem by subjecting a specific, closed-loop materials screening pipeline to a pre-registered validation against experiment-anchored literature values.

Methodology
The study employs a "pre-registered" scientific protocol to eliminate post-hoc rationalization. Before any data analysis, the authors committed to specific verdict thresholds, a primary metric, and defined failure modes.

1. System Under Test (QME): A closed-loop pipeline operating on a single desktop, consisting of:

   • Generator: Creates candidate Na-ion cathodes via prototype substitution into known frameworks.
   • Screen: A Siamese multi-head Graph Neural Network (GNN) predicting average intercalation voltages with Monte-Carlo dropout uncertainty. The model is trained primarily on Li-ion data (2814 graphs) with only 0.7% Na chemistry, requiring extrapolation.
   • Triage: A "prior-art" layer that flags candidates if their composition matches known literature or databases.
   • Verification: A local DFT bench (PBE+U in Quantum ESPRESSO) used to verify top picks and fine-tune the GNN.

2. Validation Protocol:

   • Reference Upgrade: Instead of comparing predictions to the Materials Project (MP) computed voltages, the authors curated a set of Na-ion cathodes with "quote-anchored" experimental average voltages from primary literature.
   • Data Integrity: A human audit verified the phase identity of candidates against the specific crystal structures used by the model. Compounds with phase mismatches (e.g., amorphous vs. crystalline) were excluded.
   • Metrics: The primary metric was the upper 95% confidence bound of the bias-corrected Mean Absolute Error (MAE) on a held-out set.
   • Failure Modes: Pre-defined conditions (F1–F5) would trigger if errors were non-additive, sample sizes were too small, or if the experimental set was insufficiently curated.

Key Results

1. Failure of the Screen Against Experiment:

   • On a curated set of 6–7 known Na-ion cathodes, the raw Mean Absolute Error (MAE) was 0.67 V.
   • The pre-registered primary metric (upper 95% confidence bound of the bias-corrected MAE) was 1.09 V, far exceeding the 0.50 V threshold for "screening-grade" performance.
   • Non-Additive Error: The prediction error was strongly voltage-dependent ($r = -0.94$). The model over-predicted low-voltage compounds and under-predicted high-voltage cobalt polymorphs. This structure rendered simple additive calibration invalid; applying a bias correction actually increased the error.

2. Dominance of Reference Error:

   • A three-way comparison (Model Prediction vs. MP Reference vs. Experiment) on two NaCoPO$_4$ polymorphs revealed that the MP PBE+U reference values were approximately 0.54 V lower than experimental measurements.
   • The model's deviation from the experiment was largely driven by its fidelity to the erroneous reference scale. The model successfully reproduced the reference error, inheriting a systematic deficit of ~0.4–0.5 V against real measurements.

3. Prior-Art Saturation:

   • The "discovery" space targeted by the generator was found to be saturated. Of 78 chemically valid Na substitution candidates, 70.5% were flagged as already published.
   • A similar audit on a sample of the GNoME stable-materials release showed that while title-level matching found few duplicates, the pipeline's own triage layer (checking composition and structure) identified significant overlap with known literature, suggesting that "novelty" in generated libraries is often an artifact of incomplete database coverage rather than true discovery.

4. Reference Data Integrity:

   • The curation process uncovered significant identity errors in existing databases, including namespace collisions and phase misidentifications (e.g., matching a literature voltage for an amorphous phase to a model predicting a crystalline phase). This highlighted that reference data integrity is a binding constraint on validation, often more critical than model architecture.

Significance and Claims
The paper explicitly claims no discovered materials, no validated screen, and no calibrated scale. Its primary contribution is discipline and protocol:

• Validation Discipline: It demonstrates a rigorous, pre-registered framework where negative results are reported without renegotiation. Every quantitative claim traces to a committed artifact (commit hashes, audit trails).
• Systemic Warning: It provides evidence that ML models trained on computed voltages inherit systematic errors from those references, particularly in transition-metal phosphate chemistries. The "benchmark" accuracy against databases like Materials Project may significantly understate true error against experiment.
• Redefining Novelty: It argues that automated novelty claims should be bounded by "absence within stated coverage" rather than absolute discovery, acknowledging that current databases are incomplete and that high base-rates of known materials exist in targeted substitution spaces.
• Self-Reflection: The authors retire their Na-ion screen, acknowledging it is not screening-grade. They have pre-registered a calibration audit of their own DFT bench against Li-ion couples, with the verdict pending, to determine if their internal scale can be corrected.

In summary, the paper serves as a "negative result" case study, arguing that the field's binding constraint has shifted from generating candidates to deciding which claims deserve trust, and that current validation practices often fail to expose systematic errors inherited from computational references.