🔬 applied physics

High Throughput Analysis of Nanobeam Electron Diffraction Datasets using Unsupervised Clustering

This paper demonstrates that unsupervised clustering algorithms can effectively automate the high-throughput analysis of nanobeam electron diffraction datasets by decomposing diffraction peak vectors to identify crystalline, amorphous, and minor components in both polycrystalline and single-crystal systems.

Original authors: Ian MacLaren, Ala Al-Afeef, Trevor Almeida, Rantej Bali, Shriyar Tariq, Emily Wackan, Luke Daly, Joshua F. Einsle

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ian MacLaren, Ala Al-Afeef, Trevor Almeida, Rantej Bali, Shriyar Tariq, Emily Wackan, Luke Daly, Joshua F. Einsle

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a massive, chaotic library where every single book represents a tiny snapshot of a material's atomic structure. In the past, trying to find specific stories (like a specific type of crystal or a hidden defect) in this library was like searching for a needle in a haystack by reading every page of every book one by one. It was slow, tedious, and you might miss the needles entirely.

This paper introduces a new, super-fast librarian who uses a clever trick called Unsupervised Clustering to organize this library instantly. Here is how it works, broken down into simple concepts:

The Problem: Too Much Data

Scientists use a powerful microscope technique called 4D-STEM to look at materials. Instead of taking one picture, the microscope takes thousands of "diffraction patterns" (think of these as unique atomic fingerprints) as it scans across a sample.

The Old Way: Traditionally, scientists looked at these patterns like regular photos, trying to spot differences by eye. This is like trying to find a specific song by listening to the static noise of a radio.
The New Way: The authors realized that instead of looking at the "noise" (the whole image), they could just list the "notes" (the specific bright spots in the pattern). This turns a giant, heavy image file into a simple, lightweight list of coordinates.

The Solution: The "Grouping" Game

Once the data is converted into a list of these "notes," the authors use a computer algorithm (a type of machine learning) to play a game of "find your friends."

Level 1: Sorting the Notes
Imagine you have a pile of mixed-up musical notes from different songs. The computer looks at the pitch and timing of each note and groups them.

If two notes are very similar in position, they get put in the same pile.
If a note is unique, it gets its own pile.
Result: The computer instantly separates the "fingerprint" of one crystal from the "fingerprint" of another, even if they are right next to each other.

Level 2: Finding the Band Members
Now that the notes are sorted, the computer realizes that notes from the same song (or the same crystal grain) often come from the same physical location on the sample.

It groups the "note piles" together based on where they appear on the map.
Result: It creates a clear map showing exactly where each individual crystal is, separating them from their neighbors.

Level 3: Finding the "Static"
Sometimes, parts of the sample aren't crystalline at all; they are amorphous (like glass) or made of tiny, messy nanocrystals. These don't make sharp "notes"; they make a fuzzy hum.

The computer has a special setting to find these fuzzy hums. It groups all the "messy" data together.
Result: It can draw a picture showing exactly where the "glassy" parts are and where the "crystalline" parts are, even if they are mixed together.

Real-World Examples from the Paper

The authors tested this "smart librarian" on two very different samples:

The Partially Frozen Film (Iron-Vanadium):
- The Scene: A thin film that was partially melted and then frozen by a beam of ions. Some parts turned into crystals, while others stayed messy and amorphous.
- The Result: The algorithm instantly separated the "frozen" crystals from the "messy" parts. It created a colorful map where every crystal got its own color, and the messy parts were shown in gray. It even measured how big the crystals were without the scientist having to count them manually.
The Meteorite Mystery (Pallasite):
- The Scene: A piece of a meteorite containing a mix of iron-nickel minerals. Some parts were the main "matrix" rock, and others were tiny, needle-like crystals (precipitates) hiding inside.
- The Result: Standard microscopes missed many of these tiny needles. The clustering algorithm, however, spotted them immediately. It discovered that some of these needles had a different internal structure than the rock they were sitting in. It even revealed a complex story: the needles formed a shell around a core that had a different orientation, a detail too small and complex for traditional imaging to see clearly.

Why This Matters

Speed: What used to take hours of manual tweaking can now be done in seconds on a standard laptop.
Discovery: It finds things the human eye would miss because it doesn't have "expectations." It just looks at the data and groups what belongs together.
Simplicity: By turning complex images into simple lists of numbers, the computer doesn't need to be a supercomputer to do the heavy lifting.

In short, this paper shows that by changing how we look at the data (from "pictures" to "lists of points") and using a smart grouping algorithm, scientists can automatically map out the hidden structures of materials with incredible speed and clarity.

Technical Summary: High Throughput Analysis of Nanobeam Electron Diffraction Datasets using Unsupervised Clustering

Problem Statement
Nanobeam electron diffraction (NBED) within 4-dimensional scanning transmission electron microscopy (4DSTEM) generates massive datasets where diffraction patterns are recorded at every point in a 2D scan. While Virtual Dark Field (VDF) imaging and traditional dark field techniques allow for the visualization of specific crystallographic features, they often struggle with variable contrast due to sample bending, defects, and the difficulty of segmenting numerous grains or identifying minor, unexpected phases. Furthermore, standard imaging methods rely on raw intensity summation, which includes diffuse background noise and requires manual intervention to define regions of interest. The authors identify a gap in existing machine learning applications for electron microscopy: prior clustering studies have typically operated on raw diffraction patterns (treating them as images) or derived properties like crystal orientation, rather than clustering directly on sparse lists of diffraction peak positions that jointly encode reciprocal-space position, real-space scan position, and intensity.

Methodology
The proposed approach utilizes unsupervised clustering algorithms, specifically DBSCAN (Density-Based Spatial Clustering of Applications with Noise), applied directly to "pointsarrays." These arrays are generated by detecting diffraction disks in 4DSTEM data, resulting in a sparse list of vectors for each detected peak containing reciprocal space coordinates ( $Q_x, Q_y$ ), intensity ( $I$ ), and real-space scan coordinates ( $R_x, R_y$ ).

The workflow employs a hierarchical, multi-level clustering strategy:

Data Preprocessing: Diffraction patterns are converted into pointsarrays. The primary beam is excluded, and data is normalized.
Level 1 (L1) Clustering: Clusters are formed based on the similarity of diffraction peak vectors.
- For polycrystalline systems, clustering is performed in a 4D space ( $sQ_x, sQ_y, R_x, R_y$ ), where a scaling factor $s$ weights reciprocal space positions more heavily than real space. This separates peaks with similar reciprocal space coordinates but originating from different physical locations.
- For single-crystal systems (with precipitates), clustering is performed in 2D reciprocal space ( $Q_x, Q_y$ ) to identify distinct diffraction spot families.
Level 2 (L2) Clustering: This level groups L1 clusters to identify distinct physical objects (e.g., individual grains or crystal variants).
- For polycrystals, L2 clustering groups L1 components based on the intensity-weighted Center of Mass (COM) of their real-space Digital Dark Field (DDF) images.
- For single crystals, L2 clustering uses a precomputed Jaccard distance matrix (based on image overlap of DDF masks) to group L1 clusters into families of crystals with similar spatial and diffraction signatures.
Level 3 (L3) Clustering: Applied to unclustered points from L1 to identify amorphous or deeply nanocrystalline components. This is performed on reciprocal space coordinates ( $Q_x, Q_y$ ) with looser parameters to capture diffuse ring patterns.

The method was validated on two distinct samples: a partially crystallized Fe $_{60}$ V $_{40}$ thin film (polycrystalline) and a plessite region from an Imilac meteorite (single crystal with precipitates).

Key Results

Polycrystalline FeV Film: The hierarchical approach successfully segmented the film into 52 distinct crystalline grains (L2 clusters) and a separate amorphous/nanocrystalline component (L3 cluster). The L3 clustering identified a diffuse BCC 110 ring corresponding to the uncrystallized matrix and surface oxides, which were invisible to standard L1 clustering. The resulting composite Digital Dark Field image clearly visualized the partial recrystallization process, distinguishing crystalline grains from the underlying amorphous layer.
Meteorite Plessite: In a single-crystal matrix containing precipitates, L1 clustering identified over 40 distinct diffraction spot families, revealing needle-like features and fine structures not apparent in standard Annular Dark Field (ADF) imaging. L2 clustering, using image similarity, grouped these into seven distinct families. This analysis revealed complex structural relationships: large lamellae were found to have Ni-rich rims (tetrataenite/taenite) and differently oriented Fe-rich cores (kamacite), a detail missed by cursory EELS scans. The clustering confirmed a precipitation sequence where taenite forms within kamacite, followed by further segregation in larger lamellae.
Performance: The analysis is computationally efficient. On a standard laptop, the process from data loading to L2 clustering and visualization took approximately 10 seconds for the plessite dataset, with peak memory usage under 4 GB. The sparse vector representation significantly reduces memory requirements compared to processing raw 4DSTEM data cubes.

Significance and Claims
The paper claims that applying unsupervised clustering directly to sparse diffraction peak lists offers a high-throughput, automated method for decomposing complex NBED datasets. The authors assert that this approach:

Automates Imaging and Analysis: It removes the need for manual aperture placement and reduces user bias, automatically generating composite images of crystalline and amorphous components.
Enhances Sensitivity: It improves the visibility of weak features and minor components (such as small precipitates or amorphous regions) that are difficult to detect using conventional TEM dark field or VDF methods.
Facilitates Segmentation: It provides a straightforward way to segment individual crystals in polycrystalline materials, enabling direct calculation of grain size and shape statistics without overlap issues.
Reveals Hidden Structures: In single-crystal systems, it uncovers nanoscale structural details and phase relationships (e.g., orientation relationships between matrix and precipitates) that are not immediately obvious from standard imaging.

The authors position this method as a preliminary step that can streamline the analysis of nanobeam diffraction data, making it easier to identify features of interest before undertaking more detailed crystallographic or crystal size/shape distribution analyses. They note that while DBSCAN was used, the framework is adaptable to other algorithms, and the choice of distance metrics (e.g., Jaccard vs. intensity-weighted) remains an area for optimization depending on the specific dataset characteristics.

The Problem: Too Much Data

The Solution: The "Grouping" Game

Real-World Examples from the Paper

Why This Matters

More like this