🤖 AI

ED-CSP: Crystal Structure Prediction from Electron Diffraction

This paper introduces ED-CSP, a machine learning framework that leverages a relational set encoder and periodic flow generator to predict 3D crystal structures from sparse, unindexed electron diffraction patterns and chemical composition, achieving state-of-the-art performance on a newly constructed dataset of 4.85 million simulated structures.

Original authors: Germain Poloudenny, Yaël Frégier, Arnaud Demortière

Published 2026-08-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Germain Poloudenny, Yaël Frégier, Arnaud Demortière

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Invisible Puzzle: Why Crystals Matter and How We See Them

Imagine trying to build a house, but you can't see the blueprints, and the bricks are so tiny you can't even hold them. This is the daily reality for scientists studying crystals—materials like the silicon in your phone, the salts in your food, or the minerals in your bones. To understand how these materials work, scientists need to know exactly how every single atom is arranged in 3D space. Usually, they use X-rays, which are great for big chunks of material, but they struggle with tiny, microscopic crystals that are too small to see with a standard microscope.

Enter Electron Diffraction (ED). Think of this as a super-powerful flashlight made of electrons instead of light. When you shine this "flashlight" at a tiny crystal, the electrons bounce off the atoms and create a pattern of dots on a detector, much like how a shadow puppet show creates shapes on a wall. However, there's a catch: the crystal is so small and the pattern so sparse (full of gaps) that it's incredibly hard to reverse-engineer the 3D house from just a few scattered shadows. For a long time, scientists could only guess the crystal's identity by matching these shadows to a giant, pre-made library of known shapes. But what if the crystal is something brand new, or the library is missing the right page? That's the big question this paper tackles: Can we teach a computer to look at these messy, incomplete shadows and invent the 3D structure from scratch, without needing a pre-existing library?

The Paper: Teaching AI to "Dream" Crystals from Shadows

This paper introduces a new AI model called ED-CSP, which acts like a master architect that can reconstruct a 3D crystal building just by looking at a few scattered, blurry snapshots of its shadow.

The Challenge: The "Unindexed" Mystery
Usually, to figure out a crystal's structure, scientists have to first "index" the diffraction pattern. Imagine trying to solve a jigsaw puzzle where the pieces are scattered on the floor, and you don't even know which box they came from or what the final picture looks like. That's what "unindexed" means: the computer sees a list of dots (spots) on a detector but doesn't know which atom made which dot or where the crystal is pointing. Previous AI methods could only guess the name of the crystal or pull a matching structure out of a finite database. They couldn't actually build the 3D coordinates of the atoms if the answer wasn't already in their memory.

The Solution: ED-CSP
The authors built ED-CSP to fill this gap. Instead of just looking up answers, this model learns to generate them. Here is how it works, using a playful analogy:

Imagine you are trying to describe a unique, complex sculpture to a friend over the phone, but you can only send them a few blurry photos taken from different angles.

The Recipe (Composition): You tell your friend, "The sculpture is made of 10 gold atoms and 20 silver atoms." This is the known composition. The AI knows exactly what "ingredients" (atoms) and how many of them it needs to use.
The Clues (ED Spots): You send the blurry photos. These are the sparse multi-view ED spots. The AI looks at the patterns of dots in these photos to understand the shape and orientation of the sculpture.
The Builder (The Generator): The AI uses a special "periodic flow generator." Think of this as a 3D printer that starts with a random cloud of atoms and slowly sculpts them into the correct shape, guided by the photos and the ingredient list. It doesn't just guess; it mathematically "flows" the atoms into the most likely arrangement that would create those specific dot patterns.

The Big Test: The CHILI-100K Benchmark
To see if ED-CSP actually works, the researchers tested it on a massive dataset called CHILI-100K, which contains 100,000 known crystal structures. They simulated what the electron diffraction photos would look like for these crystals and then asked the AI to rebuild them.

The Results: When the AI was trained only on the 100,000 structures, it successfully rebuilt the correct 3D structure for 57.5% of the test cases (when allowed to try 5 different guesses).
The "Secret Sauce": The researchers found that the AI's performance skyrocketed when they gave it a "warm start" using a massive library of 4.85 million simulated structures (called ED-CS) before testing it on the 100,000. With this huge head start, the success rate jumped to 66.3%.
Verifying the method: To make sure the AI wasn't just memorizing the answers, they ran a test where they swapped the "photos" for a different crystal that had the exact same ingredients but a different shape. The AI's performance dropped significantly, proving it was actually using the specific dot patterns from the photos to figure out the shape, not just guessing based on the ingredients.

What It's NOT (and What It Rules Out)
The paper is very careful to say what this model doesn't do yet.

It's not a magic wand for real-world experiments (yet): The results are based on simulated data. The computer generated the "photos" using physics equations, not from a real electron microscope in a lab. The authors admit that real-world experiments have messy issues like background noise and missing data that the simulation didn't include.
It's not just a library lookup: The paper explicitly shows that if you try to just look up the answer in a database (retrieval), you fail for about half the cases where the exact formula isn't in the training set. ED-CSP succeeds where the database fails because it generates new structures rather than searching for old ones.
It's not a solved problem: The authors note that while the AI is good, it's not perfect. They found that if they took the AI's generated structures and "relaxed" them using energy calculations (letting the atoms settle into their most comfortable positions), the accuracy improved even more. This suggests the AI is a great first draft, but it still needs a little polishing to be perfect.

Why This Matters
This paper is a major step forward because it moves the field from "finding the answer in a book" to "solving the puzzle from scratch." By proving that an AI can learn to translate sparse, messy electron diffraction spots into 3D atomic maps, it opens the door to discovering new materials that have never been seen before. It suggests that in the future, we might be able to identify the structure of a tiny, unknown crystal in seconds, simply by feeding its shadow patterns into a computer that knows how to dream up the 3D world behind the dots.

Technical Summary: ED-CSP – Crystal Structure Prediction from Electron Diffraction

Problem Statement
Recovering a periodic 3D crystal structure from sparse, unindexed electron diffraction (ED) observations is a challenging generative inverse problem. While Powder X-ray Diffraction (PXRD) is well-established for bulk powders, it averages orientation information into a 1D radial profile. In contrast, ED collects structural signals from individual nanocrystals, preserving detector-plane relationships among scattering vectors and orientation-dependent reciprocal space information. Existing machine learning approaches for ED largely focus on predicting crystallographic labels (e.g., space groups) or retrieving structures from finite databases. There is a gap in methods capable of predicting full lattice and fractional atomic coordinates directly from unindexed, multi-view ED spot lists given a known chemical composition.

Methodology: ED-CSP
The authors introduce ED-CSP, a machine learning model designed to generate crystal structures conditioned on known composition and multiple sparse ED views. The architecture consists of three primary components:

Relational Set Encoder: Each ED view is processed as a variable-length list of detector-plane spots (coordinates and log-intensities). The model employs a shared relational set encoder, adapted from the PE-AG-GMoE backbone, which uses graph-attention mechanisms to encode raw spot lists. This encoder is permutation-invariant and handles the unindexed nature of the data.
Multi-View Aggregation: Information from $K$ different ED views is aggregated using a mean-max pooling head. This creates a unified conditioning representation that captures geometric information across orientations.
Periodic Flow Generator: The aggregated ED state, combined with the known composition (fixing atom types and counts), conditions a periodic flow generator. This generator, based on a six-layer CSPNet-style periodic graph decoder, predicts the lattice vectors and fractional atomic coordinates by modeling vector fields over the periodic lattice.

Training and Data Construction
To train ED-CSP, the authors constructed ED-CS (Electron Diffraction Crystal Structures), a curated dataset of 4.85 million structures.

Source: Structures were drawn from seven major repositories (AFLOW, Alexandria, COD, GNoME, Materials Project, OQMD, JARVIS-DFT).
Filtering: The dataset was deduplicated using a strict structural matcher and explicitly filtered to exclude any structures present in the CHILI-100K benchmark set, ensuring a disjoint training/test split.
Simulation: For each structure, multi-view ED patterns were simulated using the py4DSTEM pipeline (dynamical diffraction at 300 keV). Structures were retained only if they yielded at least 10 valid views with sufficient spot intensity.
Pretraining Strategy: The model utilizes a "registry-scale" transfer approach. A precursor model was trained on a 1-million-structure subset before being warm-started for fine-tuning on the full ED-CS corpus and the CHILI-100K benchmark.

Key Contributions

Formulation of Unindexed Multi-View ED CSP: The paper defines the task of predicting crystal geometry from unindexed, sparse multi-view ED spot lists given composition, instantiated by the ED-CSP architecture.
ED-CS Dataset: The release of a 4.85-million-structure corpus with simulated multi-view ED, designed specifically to be disjoint from the CHILI-100K benchmark to prevent data leakage.
Controlled Signal Analysis: The study employs input interventions (e.g., removing ED spots, swapping donor ED patterns) and a converged composition-only reference to rigorously test whether the model utilizes the specific diffraction signal beyond just composition priors.
Benchmarking: A comprehensive evaluation framework comparing ED-CSP against state-of-the-art PXRD-conditioned models (PXRDGen, XRDSol, deCIFer) and finite-library retrieval baselines.

Results
Evaluated on 2,075 held-out CHILI-100K materials:

Performance: The registry-initialized ED-CSP achieves a structural match rate (MR) of 66.3% at 5 candidates (MR@5). This outperforms PXRDGen (52.9% MR@5), which uses the same generator architecture but is conditioned on PXRD profiles.
Impact of Dataset Scale: Warm-starting the model with a 1-million-structure precursor significantly improves performance, raising MR@5 from 57.5% (CHILI-only initialization) to 66.3%.
Signal Use: Removing ED spots entirely drops MR@1 from 51.66% to 17.35%. Furthermore, swapping target ED observations with those from a non-isomorphic same-formula donor reduces MR@5 by 22.1 percentage points, confirming the model relies on query-specific diffraction geometry.
Generalization: On 1,024 queries where the exact reduced formula is absent from the training library, the registry-initialized model retains a 53.5% MR@5, demonstrating generative capability where exact-formula lookup fails.
Post-Processing: Applying target-free interatomic potentials (e.g., ORB-v3, MACE-MPA-0) to relax and rank generated candidates improves top-1 recovery by ~13 percentage points, though a performance ceiling remains, suggesting room for ED-aware refinement.

Significance and Claims
The paper positions ED-CSP and the ED-CS dataset as a controlled benchmark for generative inference from sparse, simulated ED data. The authors claim that:

ED-CSP successfully bridges the gap between label prediction/retrieval and full structure generation for ED.
The model demonstrates that sparse, unindexed multi-view ED contains sufficient learnable signal to recover crystal structures, outperforming PXRD-conditioned counterparts in this specific modality.
The results support a workflow where generative recovery complements, rather than replaces, finite-library lookups and indexed-reflection reconstruction.
The study highlights that while current performance is strong, the transition from calibrated simulated data to experimental 3D ED remains a significant challenge due to factors like detector calibration, background noise, and thickness variations.

The authors maintain a modest stance, noting that their comparisons evaluate complete systems (encoder + generator) rather than isolating the diffraction modality alone, and that future work is needed to address the "external-validity gap" between simulation and experiment.

The Invisible Puzzle: Why Crystals Matter and How We See Them

The Paper: Teaching AI to "Dream" Crystals from Shadows

Technical Summary: ED-CSP – Crystal Structure Prediction from Electron Diffraction

More like this