AtomBench: A Benchmarking Framework for Generative Crystal Reconstruction Models in Conventional Superconductors

Technical Summary: AtomBench

Problem Statement
The paper addresses a critical gap in the benchmarking of generative models for materials science: the lack of a standardized, fair framework for evaluating crystal reconstruction (inverse design) tasks. While many models claim success in de novo generation (assessing stability, uniqueness, and novelty), the ability to accurately reconstruct a known crystal structure from partial information is often overlooked.

A primary issue identified is that existing comparisons frequently confound architectural performance with information asymmetry. Different models are often evaluated under conditions where they possess unequal amounts of information about the target crystal (e.g., some receive only composition, while others receive composition plus a latent embedding of the full structure). This makes direct architectural comparisons invalid. Furthermore, standard metrics like "match rate" and "matched RMSD" rely on hard thresholds (e.g., pymatgen StructureMatcher), creating a discontinuous evaluation where models are only scored on the subset of structures they happen to match, potentially masking systematic failures in the majority of predictions.

Methodology
The authors introduce AtomBench, an extensible, model-agnostic framework designed to benchmark generative models on a well-defined crystal reconstruction task using conventional superconductors.

1. Datasets: The study utilizes two DFT-generated datasets:

   • JARVIS Supercon-3D: 1,058 crystal/$T_c$ pairs, dominated by O, Al, and Ti, with diverse crystal systems.
   • Alexandria DS-A/B: 8,253 crystal/$T_c$ pairs, dominated by Ti, Rh, and Al, primarily cubic and tetragonal.
   • Both datasets use Density Functional Perturbation Theory (DFPT) with specific protocols (e.g., $\mu^*$ values, pseudopotentials).

2. Models Benchmarked: Four distinct generative architectures were trained and evaluated:

   • AtomGPT: A fine-tuned large language model (Mistral-7B) treating crystals as text.
   • CDVAE: A diffusion variational autoencoder using SE(3)-equivariant graph neural networks.
   • FlowMM: A Riemannian flow-matching network.
   • MatterGen: A joint-diffusion model with equivariant score networks.
   • Conditioning Variants: For models supporting it (AtomGPT, MatterGen), variants were trained with and without conditioning on the superconducting critical temperature ($T_c$).

3. Evaluation Protocol:

   • Data Split: 10% of each dataset was held out for testing.
   • Information Regimes: Models were grouped by the information available prior to prediction:
     • Full-structure embedding: CDVAE (receives a latent vector of the full target).
     • Composition + $T_c$: AtomGPT/MatterGen (Tc-conditioned).
     • Composition only: FlowMM, AtomGPT, MatterGen (unconditioned).
   • Metrics:
     • Lattice Parameters: Mean Absolute Error (MAE) and Kullback-Leibler Divergence (KLD) for distributions.
     • Atomic Coordinates:
       • Matched RMSD: Average error over structures successfully matched by StructureMatcher.
       • Continuous Corrected RMSD (ccRMSD): A novel metric introduced in this work. It uses the Average Minimum Distance (AMD) descriptor and Chebyshev distance to measure local geometric fidelity continuously across all test structures, regardless of whether a discrete match is found. This eliminates the bias of discarding "hard-to-match" structures.

Key Contributions

1. AtomBench Framework: An open-source Python package (atombench) that standardizes the evaluation of inverse design models, supporting direct submission to the JARVIS-Leaderboard.
2. ccRMSD Metric: The introduction of a continuous, match-rate-independent metric that provides a more robust assessment of local geometric fidelity than traditional matched RMSD.
3. Information-Theoretic Benchmarking: A rigorous grouping of models by "conditional entropy" regimes (amount of known information) to ensure fair architectural comparisons.
4. Systematic Analysis: The first large-scale comparison of these specific architectures on superconductor reconstruction, analyzing the impact of $T_c$ conditioning and crystal system symmetry.

Results

• Performance by Metric:
  • Atomic Coordinates: MatterGen achieved the best reconstruction fidelity (lowest ccRMSD and matched RMSD), followed by AtomGPT. CDVAE and FlowMM showed significantly higher errors.
  • Lattice Parameters: CDVAE was the most accurate in reconstructing lattice parameters (lowest MAE and KLD), followed by AtomGPT and MatterGen. FlowMM was the least accurate.
  • Speed vs. Accuracy: FlowMM was the fastest at inference but least accurate. MatterGen offered the highest coordinate accuracy but at a higher computational cost.
• Impact of $T_c$ Conditioning: Conditioning on the critical temperature ($T_c$) did not consistently improve reconstruction fidelity for either AtomGPT or MatterGen, suggesting that a single scalar property adds little structural information beyond composition in this context.
• Crystal System Trends: Reconstruction error varied by crystal system. Cubic systems (high symmetry, fewer degrees of freedom) exhibited the lowest errors, while monoclinic and hexagonal systems showed higher errors, particularly in the $c$ lattice parameter.
• Metric Discrepancy: The study highlighted that matched RMSD can be misleading; for example, FlowMM had a very low match rate (<10%), making its matched RMSD statistically unreliable, whereas ccRMSD provided a meaningful error estimate for the entire dataset.

Significance and Claims
The paper claims that AtomBench provides a necessary correction to the field by exposing how information asymmetry can artificially inflate performance metrics. By introducing ccRMSD, the authors argue for a shift away from "match-or-discard" evaluation toward continuous fidelity measures that account for the full distribution of predictions.

The authors modestly state that their findings do not universally rank architectures but rather illustrate how reconstruction fidelity varies under different inference information regimes. They emphasize that the choice of model should depend on the specific application:

• FlowMM for high-throughput sampling where speed is paramount.
• CDVAE for tasks requiring high lattice-parameter accuracy.
• MatterGen for tasks requiring high-fidelity atomic coordinate placement.

The work concludes that while current models (CDVAE, AtomGPT, MatterGen) reach accuracy levels suitable for initializing DFT calculations, the field requires continuous metrics like ccRMSD to properly assess and improve generative models for materials discovery.