🔬 materials science

How to verify the precision of density-functional-theory implementations via reproducible and universal workflows

This paper establishes general recommendations for verifying the precision of density-functional-theory implementations by presenting a universal, reproducible workflow based on AiiDA that generates a comprehensive reference dataset of 960 equations of state across the periodic table to cross-check all-electron codes and improve pseudopotential-based approaches.

Original authors: Emanuele Bosoni, Louis Beal, Marnik Bercx, Peter Blaha, Stefan Blügel, Jens Bröder, Martin Callsen, Stefaan Cottenier, Augustin Degomme, Vladimir Dikan, Kristjan Eimre, Espen Flage-Larsen, Marco Forna

Published 2026-09-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emanuele Bosoni, Louis Beal, Marnik Bercx, Peter Blaha, Stefan Blügel, Jens Bröder, Martin Callsen, Stefaan Cottenier, Augustin Degomme, Vladimir Dikan, Kristjan Eimre, Espen Flage-Larsen, Marco Fornari, Alberto Garcia, Luigi Genovese, Matteo Giantomassi, Sebastiaan P. Huber, Henning Janssen, Georg Kastlunger, Matthias Krack, Georg Kresse, Thomas D. Kühne, Kurt Lejaeghere, Georg K. H. Madsen, Martijn Marsman, Nicola Marzari, Gregor Michalicek, Hossein Mirhosseini, Tiziano M. A. Müller, Guido Petretto, Chris J. Pickard, Samuel Poncé, Gian-Marco Rignanese, Oleg Rubel, Thomas Ruh, Michael Sluydts, Danny E. P. Vanpoucke, Sudarshan Vijay, Michael Wolloch, Daniel Wortmann, Aliaksandr V. Yakutovich, Jusong Yu, Austin Zadoks, Bonan Zhu, Giovanni Pizzi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of materials science, researchers often rely on powerful computer simulations to predict how new substances will behave before they are ever built in a lab. These simulations are based on a fundamental theory of physics called density-functional theory, which uses the laws of quantum mechanics to calculate the properties of matter using only its chemical composition and crystal structure. It is a tool that has accelerated the discovery of everything from better batteries to stronger metals. However, because these calculations are so complex, scientists must make a series of choices about how to solve the equations, such as which mathematical shortcuts to use or how finely to divide the space around atoms. Different computer programs, or codes, make these choices in slightly different ways. For a long time, it was unclear whether these different approaches would lead to the same answer, or if the variations in the software itself were introducing hidden errors into the results.

A massive international collaboration has now settled this question with a level of precision never seen before. The researchers set out to verify the accuracy of these different computer codes by testing them against a vast and diverse set of materials. They did not just look at a few simple examples; instead, they created a reference library covering every element in the periodic table, from hydrogen to curium. For each of these 96 elements, they modeled ten different crystal structures: four made of just that single element, and six different oxides where the element is bonded to oxygen. This resulted in a total of 960 distinct systems to test. To ensure the results were trustworthy, the team first used two of the most advanced and rigorous computer codes available, which treat every single electron in the atom explicitly, to generate a "gold standard" set of data. These two codes agreed with each other so closely that their results could be treated as the correct answer for the purpose of comparison.

With this high-precision reference in hand, the team then ran the same 960 simulations using nine other popular computer codes that are widely used in the scientific community. These other codes use a different, faster method that simplifies the calculation by focusing only on the outer electrons involved in bonding, rather than every electron. The researchers found that while many of these popular codes performed very well, there were noticeable differences in their results compared to the gold standard. Some codes struggled more than others with specific types of elements, particularly those with complex electron behaviors like the rare earth metals. The study did not simply point out these flaws; it used the discrepancies to guide improvements. By adjusting the settings and the underlying data libraries within these codes, the researchers were able to significantly reduce the errors, bringing the faster, approximate methods much closer to the rigorous, all-electron results.

To make this process repeatable and transparent for the entire scientific community, the team built a fully automated system that runs these thousands of calculations without human intervention. This system ensures that every code is tested under exactly the same conditions, eliminating the chance that a human error in setting up the experiment could skew the results. The team also developed new ways to measure the differences between the codes, creating metrics that focus on physically measurable quantities like the volume of the crystal and its resistance to compression, rather than just abstract energy numbers. They found that for most materials, the different codes now agree within a very tight margin of error, but they also identified specific cases where the differences were still too large to ignore.

The work serves as a crucial checkpoint for the field, proving that while the different software tools generally produce reliable results, they are not identical. The study demonstrates that by using these rigorous verification protocols, scientists can identify exactly where a code needs improvement and then refine it. The researchers have made their entire dataset, along with the automated tools used to generate it, freely available to anyone. This allows other scientists to check their own work against this universal benchmark, ensuring that the simulations driving the discovery of new materials are built on a foundation of verified precision. The ultimate goal is not just to find the most accurate code, but to provide a clear path for all codes to reach a level of reliability where scientists can trust their predictions with confidence.

Technical Summary: Verification of Density-Functional-Theory Implementations via Reproducible and Universal Workflows

Problem Statement
Density-functional theory (DFT) has become a cornerstone of condensed matter physics and materials science, enabling the prediction of material properties with minimal experimental input. However, the reliability of these predictions depends heavily on the precision of the computational codes and the specific numerical choices made by users (e.g., basis sets, pseudopotentials, k-point meshes, and smearing techniques). While a systematic assessment of DFT precision was conducted in 2016 (the " $\Delta$ -project") on 71 elemental crystals, the field lacked a comprehensive verification framework covering the full periodic table, diverse chemical environments, and multiple computational approaches. Furthermore, the lack of standardized, reproducible workflows and curated reference datasets hinders the ability to quantify uncertainties and improve pseudopotential libraries systematically.

Methodology
The authors developed a highly automated, reproducible infrastructure to generate a massive reference dataset for verification purposes.

Dataset Scope: The study covers 960 cubic crystal structures representing all elements from Hydrogen ( $Z=1$ $Z = 1$ ) to Curium ( $Z=96$ $Z = 96$ ). For each element, the dataset includes:
- Unaries: Four mono-elemental cubic structures (FCC, BCC, Simple Cubic, Diamond) covering coordination numbers 12, 8, 6, and 4.
- Oxides: Six cubic oxides ( $X_2O$ , $XO$ , $X_2O_3$ , $XO_2$ , $X_2O_5$ , $XO_3$ ) designed to span formal oxidation states from +1 to +6.
Reference Generation: The "ground truth" was established using two independent, state-of-the-art all-electron (AE) codes: FLEUR and WIEN2k. Both utilize the linearized augmented plane wave (LAPW) method with local orbitals. Calculations were performed using the PBE exchange-correlation functional, scalar-relativistic treatment, and specific numerical parameters (Fermi–Dirac smearing of 0.0045 Ry, specific k-point meshes).
Automation: The entire workflow was managed using AiiDA and the aiida-common-workflows (ACWF) package. This ensured uniform input parameter selection, automatic error handling, and full data provenance, allowing for the generation of thousands of calculations with identical protocols across different codes.
Metrics: The authors introduced two new metrics to quantify discrepancies between computational approaches:
- $\varepsilon$ : A dimensionless, renormalized metric based on the root-mean-square difference of energy-volume curves, insensitive to bulk modulus magnitude.
- $\nu$ : A metric based on the relative differences of fitted Birch–Murnaghan parameters ( $V_0$ , $B_0$ , $B_1$ ), weighted by their sensitivity to numerical noise ( $w_{V_0}=1, w_{B_0}=1/20, w_{B_1}=1/400$ ).
Comparison: The AE reference dataset was used to verify nine pseudopotential-based approaches (ABINIT, BigDFT, CASTEP, CP2K, GPAW, QUANTUM ESPRESSO, SIESTA, SIRIUS, VASP) employing various basis sets (plane waves, Gaussians, wavelets, atomic orbitals).

Key Contributions

Curated Reference Dataset: A FAIR (Findable, Accessible, Interoperable, Reusable) dataset of 960 equations of state (EOS) generated with two AE codes, serving as a high-precision benchmark.
Standardized Protocols: The definition and implementation of the verification-PBE-v1 protocol, which specifies exact numerical parameters (smearing, k-point density, fitting algorithms) to ensure comparability across different codes.
New Metrics: The proposal of $\varepsilon$ and $\nu$ metrics to provide physically meaningful, quantitative measures of agreement between codes, moving beyond simple energy differences.
Pseudopotential Improvement: The dataset was used to identify deficiencies in existing pseudopotential libraries, leading to the creation of improved versions for several codes (e.g., PseudoDojo v0.5, SSSP v1.3, CASTEP C19 updates, and VASP lanthanide potentials).
Workflow Infrastructure: Demonstration that automated, universal workflows can guarantee uniformity and reproducibility in large-scale computational materials science.

Results

AE Code Agreement: The two all-electron codes (FLEUR and WIEN2k) showed excellent agreement. For 93% of structures, the relative difference in equilibrium volume ( $V_0$ ) was below 0.1%. Discrepancies in bulk modulus ( $B_0$ ) and its pressure derivative ( $B_1$ ) were larger but generally within acceptable limits, with outliers primarily in noble gas oxides.
Pseudopotential Performance: Significant variations in precision were observed among pseudopotential approaches.
- Plane-wave codes generally showed better agreement with the AE reference than localized basis set codes, though the latter are computationally more efficient.
- Specific issues were identified, such as the need for f-projectors in PseudoDojo for elements around the 4f block and the sensitivity of results to smearing types (e.g., in Er diamond structure).
- After applying the authors' recommendations and updating pseudopotentials (e.g., adding f-projectors, adjusting core radii, changing libraries), the agreement with the AE reference improved significantly for many codes.
Thresholds: The study established indicative thresholds for "excellent" agreement ( $\varepsilon \lesssim 0.06$ or $\nu \lesssim 0.1$ ) and "good" agreement ( $\varepsilon \lesssim 0.2$ or $\nu \lesssim 0.33$ ).
Limitations: The authors note that while the dataset is excellent for verifying EOS and total energies within a single material, it should not be directly used to calculate formation energies between different materials without ensuring consistent simulation parameters (e.g., core/valence separation) across those different systems.

Significance
The paper positions itself as a critical step toward controlling the numerical aspects of electronic structure calculations. Its significance lies in:

Reliability Assessment: Providing a rigorous method to quantify the precision of DFT implementations, allowing users to select codes and parameters with known uncertainties.
Community Standardization: Offering a "universal" workflow and dataset that can be reused to test new functionals, codes, or pseudopotentials, thereby accelerating the development of more accurate computational tools.
Pseudopotential Optimization: Demonstrating how curated reference data can drive the iterative improvement of pseudopotential libraries, which are essential for efficient high-throughput calculations.
Reproducibility: Highlighting the necessity of automated, transparent workflows to ensure that computational results are reproducible and comparable, a prerequisite for the integration of DFT data into machine learning models and large-scale materials discovery pipelines.

The authors explicitly state that this work focuses on verification (code precision relative to the DFT formalism) rather than validation (accuracy relative to experimental results), though they acknowledge the latter as a necessary, separate step. They conclude by recommending future extensions to cover magnetic systems, relativistic effects, and additional functionals, as well as the development of protocols that optimize computational cost for target accuracies ("good-enough data").

More like this