Predictive drift compensation of multi-frame STEM via live scan modification

This paper presents a predictive drift compensation method for scanning transmission electron microscopy (STEM) that analyzes past frames to dynamically adjust future scan grids at both pixel and frame levels, thereby mitigating sample drift and preserving image fidelity during multi-frame acquisition without relying on post-processing registration.

The Gist

Imagine you are trying to take a high-resolution photograph of a tiny, intricate snowflake using a camera that moves across the snowflake line by line, like a scanner. This is essentially what a Scanning Transmission Electron Microscope (STEM) does, but instead of light, it uses a beam of electrons to see atoms.

The problem? The snowflake (the sample), the table (the stage), or even the camera itself might be slightly wobbly. In the real world, we call this "drift." It's caused by things like the room getting warmer, vibrations, or the machine settling down after being turned on.

If you try to take a picture while everything is wobbling, your photo comes out blurry, stretched, or torn. Usually, scientists take many quick photos and try to stitch them together later on a computer to fix the blur. But there's a catch: because the sample moved while they were taking the photos, the "common area" where all the photos overlap gets smaller and smaller. It's like trying to make a collage from photos where the subject kept walking out of the frame; you end up throwing away most of the pictures.

The New Solution: Predicting the Future

This paper introduces a clever new method called Predictive Drift Compensation. Instead of waiting for the photo to get blurry and then fixing it later, the microscope learns to predict where the sample is going to move before it happens, and adjusts the camera in real-time to follow it.

Here is how it works, using some everyday analogies:

1. The "Dance Partner" Analogy (Frame-by-Frame)

Imagine you are dancing with a partner who keeps drifting away from you.

• Old Way: You dance for a bit, stop, look at where they went, and then try to shuffle your feet to get back in sync. By the time you move, they've already moved again.
• New Way: You watch your partner's movement pattern. If they are drifting slowly to the left, you don't wait for them to leave your view. You predict they will be two steps to the left in the next second, so you step there before they get there. You are essentially "leading" the dance to stay perfectly in sync.

In the microscope, after taking the first few frames, the computer calculates the speed and direction of the drift. It then tells the electron beam, "Hey, the sample is moving left, so let's aim the beam slightly to the right for the next frame." This keeps the sample perfectly centered without needing to crop the image later.

2. The "Stretchy Rubber Sheet" Analogy (Pixel-by-Pixel)

Sometimes, the drift isn't just a simple slide; it's a wobble or a twist. Imagine the image is drawn on a rubber sheet. If the sheet stretches unevenly, the picture gets warped (like a funhouse mirror).

• The Innovation: This paper doesn't just move the whole picture; it predicts how every single pixel (every tiny dot of the image) will be distorted. It's like having a magic hand that stretches and pulls the rubber sheet back into shape while the picture is being drawn. This fixes "warping" and "shearing" that usually ruins high-magnification images.

Why This is a Big Deal

The authors tested this in two main ways:

1. The "Waiting Game" (Solving the Settling Time):
   Usually, when a scientist puts a sample into the microscope, they have to wait 30 to 60 minutes for the machine to stop shaking and the sample to stop drifting. It's like waiting for a car engine to warm up.

   • With this new method: You can start taking pictures almost immediately. The microscope compensates for the "warming up" drift as it happens. This saves hours of waiting time and means you don't waste precious sample time.

2. The "Melting Gold" Experiment (In-Situ Imaging):
   They heated gold nanoparticles until they melted and changed shape. This is a chaotic process where the sample changes drastically.

   • The Result: Even as the gold melted and the sample physically moved around, the microscope kept the camera locked onto the exact same spot. It was like filming a melting ice sculpture but keeping the camera perfectly focused on one specific droplet, even as the whole sculpture shifted.

The Bottom Line

Think of this technology as giving the microscope superhuman reflexes. Instead of reacting to mistakes after they happen, it anticipates them.

• No more wasted time: You don't have to wait for the machine to settle.
• No more lost data: You keep the entire field of view, so you don't have to throw away parts of your image.
• Sharper pictures: By fixing the drift during the scan, the final image is crisp, not blurry.

The authors have even made the software open-source (free for anyone to use), meaning this "predictive dance" can be taught to other microscopes and even other types of cameras, helping scientists everywhere see the tiny world more clearly.

Technical Summary

1. Problem Statement

Scanning Transmission Electron Microscopy (STEM) is a critical tool for materials characterization, but it is highly susceptible to drift (displacement of the sample, stage, or beam) caused by thermal expansion, mechanical instability, electrical noise, or charging.

• Consequences: Drift manifests as image distortions (bending, tearing, affine shearing) and causes the region of interest (ROI) to move out of the field of view (FoV) during multi-frame acquisitions.
• Limitations of Current Methods:
  • Post-acquisition alignment: While effective for restoring image quality, it requires cropping the final image to the "common area" shared by all frames, resulting in a loss of data, wasted electron dose, and reduced resolution.
  • Physical stage correction: Moving the physical stage is often imprecise due to mechanical backlash, slip-stick movement, and limited travel range (especially for piezoelectric stages).
  • Auxiliary signals: Some methods use extra scan areas or signals (e.g., ADF) for drift tracking, which is dose-inefficient and time-consuming.

2. Methodology

The authors propose a live, predictive drift compensation method that modifies the scan grid before the next frame is acquired, rather than correcting images after the fact. The system operates on two time-scales and two length-scales.

Core Workflow

1. Image Registration:

   • After acquiring a frame, the system calculates the drift relative to previous frames using cross-correlation (specifically phase-correlation or mutual-correlation) in Fourier space.
   • To handle low signal-to-noise ratios (SNR) and periodic structures (avoiding "unit-cell hopping"), the correlation is constrained to the first unit cell using a smooth-edged mask.
   • A holistic least-squares solution is used on a matrix of pairwise displacements to create a noise-resilient drift estimate.

2. Drift Prediction:

   • The system fits a mathematical function to the history of measured drifts to predict the displacement at the start time of the next frame ($t_{n+1}$).
   • Time Modeling: The prediction accounts for non-uniform time intervals between frames by recording the absolute hardware clock time ($t_n$) for each frame start.
   • Fitting Models:
     • Polynomial Fitting: Used for general cases; assumes drift evolves smoothly (e.g., thermal settling).
     • ARIMA (AutoRegressive Integrated Moving Average): Used for time-series with periodic variations or complex trends (though polynomial was preferred for flexibility with variable time steps).
     • Weighting: Recent observations are weighted more heavily (e.g., exponential decay) to adapt to changing drift rates.

3. Live Scan Modification (Compensation):

   • Framewise (Rigid) Compensation: The entire scan grid for the next frame is offset by the predicted rigid translation ($d_x, d_y$). This prevents the ROI from drifting out of the FoV.
   • Pixelwise (Non-Rigid) Compensation: The system calculates the predicted drift for the specific acquisition time of each individual pixel within the frame. This corrects intra-frame distortions (shear, warping) caused by non-linear drift occurring during the scan.
   • Hardware Implementation: The method utilizes the scan deflectors (beam blanking/deflection coils). It operates within safe coil limits by utilizing the "flyback" margin (the unused deflection range at the start of each line).

3. Key Contributions

• Pre-emptive Correction: Unlike traditional post-processing, this method prevents drift corruption before it happens, preserving the full field of view and maximizing dose efficiency.
• Dual-Scale Correction:
  • Macro-scale: Rigid frame offsets prevent loss of the ROI.
  • Micro-scale: Pixelwise offsets correct intra-frame shear and warping, enabling distortion-free atomic resolution imaging even with significant drift.
• Generalizability: The framework is compatible with various scan patterns (raster, serpentine, interlaced, rotating STEM) and does not require additional fiducial markers or auxiliary scan areas.
• Open Source: The algorithm is implemented in Python and made available as open-source code, ensuring reproducibility.

4. Results

The method was validated through synthetic testing, atomic-resolution imaging, and challenging in-situ experiments:

• Synthetic Testing: Using experimental data to simulate drift, the method achieved an average prediction error of 2.4 pixels (ARIMA) vs. 3.4 pixels (polynomial), closely matching ground truth.
• Settling Time Reduction: Simulations showed that live compensation reduces the required "settling time" for a sample to stabilize from 40–60 minutes (for classical imaging) to just 2.5–10 minutes, significantly accelerating workflow.
• Atomic Resolution (YAG):
  • In a 40-frame acquisition of Yttrium Aluminium Garnet (YAG), uncompensated frames resulted in a 29% loss of common area after alignment.
  • Live compensation retained >98% of the area with minimal residual blurring (average under/over-compensation of 0.66 Å).
• Non-Rigid Compensation (Shear Correction):
  • In rotating STEM acquisitions, rigid compensation left visible shear artifacts.
  • Pixelwise compensation successfully eliminated shear, producing uniform images with accurate lattice angles, validated against rotating STEM ground truth.
• In-Situ Melting (Gold):
  • During a 25-minute heating experiment (100°C to 1000°C) involving massive morphological changes (melting, dewetting) and a physical chip rupture, the system successfully tracked drift.
  • It reduced total drift from 451 nm (313 pixels) to <0.7 nm (<0.5 pixels) on average, maintaining the FoV despite drastic sample changes and focus variations.

5. Significance

This work represents a paradigm shift in STEM data acquisition:

• Efficiency: It eliminates the trade-off between acquisition speed and image quality, allowing for fast, low-dose imaging of beam-sensitive samples without data loss.
• Automation: It removes the need for manual stage adjustments or long waiting periods for thermal stabilization, facilitating fully automated in-situ experiments.
• Data Integrity: By preserving the full field of view and correcting non-linear distortions in real-time, it ensures that quantitative measurements (e.g., interatomic distances, strain mapping) are accurate and reliable without relying on post-processing assumptions.
• Broad Applicability: The open-source nature and hardware-agnostic design allow for potential adaptation to other microscopy techniques (SEM, AFM, Confocal) and future high-speed scanning modalities.