Continuous three-dimensional imaging of nanoscale dynamics by in situ electron tomography

This paper presents a novel dynamic electron tomography framework that combines continuous tilting with self-supervised deep learning to enable continuous, dose-efficient 3D imaging of nanoscale structural transformations under operating conditions, overcoming the limitations of traditional static reconstruction methods.

The Gist

Imagine you are trying to watch a movie of a melting ice cream cone, but you can only take one photo every few seconds, and the camera is very slow. If you try to take enough photos to make a smooth video, the ice cream might melt so much that by the time you finish, it's just a puddle. Worse, if you use a bright flash for every photo, the heat from the flash might melt the ice cream even faster, ruining the movie you wanted to capture.

This is the exact problem scientists face when trying to watch tiny objects (nanomaterials) change shape inside an electron microscope. These objects are so small that we need powerful electron beams to see them, but the beam itself can damage or change the object while we are trying to film it.

The Old Way: The "Stop-and-Go" Movie
Traditionally, to get a 3D movie of a changing object, scientists used a "stop-and-go" method.

1. They would stop the experiment (like pausing a movie).
2. They would take a full set of photos from different angles to build one 3D snapshot.
3. They would restart the experiment, wait a bit, stop again, and take another set of photos.

The Problem: This takes forever. By the time they get the next snapshot, the object has changed so much that the "movie" has huge gaps. Also, stopping and starting the experiment (like heating and cooling) can mess up the natural process, and the repeated electron beams act like a hammer, breaking the delicate object before the movie is finished.

The New Solution: DIP-STER (The "Magic Time-Lapse" Camera)
The authors of this paper, led by Timothy Craig and Sara Bals, invented a new way to film these tiny movies called DIP-STER. Think of it as a magic camera that can take a continuous, slow-motion video of the object changing, but only needs a fraction of the photos to create a crystal-clear 3D movie.

Here is how it works, using a simple analogy:

1. The Golden Ratio Spin (The Camera Movement)

Instead of taking photos in a boring, predictable order (like 0°, 10°, 20°), the microscope spins the sample using a special pattern called the Golden Ratio.

• Analogy: Imagine a dancer spinning around. Instead of stopping at every 10 degrees to take a picture, she spins continuously, but the camera clicks at random, mathematically perfect intervals. This ensures that every single photo captures a slightly different angle and a slightly different moment in time, without ever stopping the dance.

2. The Self-Taught AI (The Magic Editor)

This is the real breakthrough. Usually, to turn 2D photos into a 3D movie, you need a huge library of pre-existing 3D movies to teach the computer what to look for. But here, the computer teaches itself.

• Analogy: Imagine you have a blurry, jumbled stack of photos of a melting ice cream cone. You don't have a reference photo of what the cone should look like. Instead, you give the photos to a very smart AI editor (the neural network).
• The AI says: "I know these photos are all of the same cone, just at different times and angles. I also know that ice cream doesn't teleport; it melts smoothly. So, I will guess what the cone looks like at every single second, and I will keep guessing until my guess, when turned back into a 2D photo, matches the blurry photo I was given."
• The AI uses Self-Supervised Learning. It doesn't need a teacher; it learns the rules of physics and smoothness just by looking at the data it has.

3. The Result: A Smooth, High-Speed 3D Movie

Because the AI is so smart and the camera keeps spinning continuously, they can reconstruct a full 3D movie of the object changing, second by second, from just one single continuous spin.

Why is this a big deal?

• Less Damage: Because they don't need to stop and start, and they don't need to take as many photos, the electron beam hits the object much less. It's like taking a photo with a gentle candlelight instead of a blinding camera flash. This means the object stays true to its natural self.
• Real-Time Dynamics: They can watch things happen that were previously impossible to see, like gold stars melting or silver and gold mixing together (alloying) in real-time.
• Speed: What used to take hours of "stop-and-go" filming now happens in a continuous 35-minute session.

In Summary:
The scientists built a system that combines a clever spinning camera with a self-teaching AI. This allows them to film the "life story" of tiny, fragile nanomaterials in 3D without breaking them or missing the action. It turns a blurry, jumbled mess of data into a high-definition, time-resolved 3D movie of the nanoworld.

Technical Summary

1. Problem Statement

Traditional Electron Tomography (ET) is a powerful tool for 3D characterization of nanomaterials but faces critical limitations when applied to dynamic processes (e.g., heating, alloying, beam damage):

• Static Assumption: Conventional reconstruction algorithms (e.g., SIRT, FBP) assume the specimen remains static during the acquisition of a tilt series. If the sample evolves during acquisition, the result is a "motion-blurred" average, losing temporal resolution.
• The "Stop-and-Go" Limitation: Current methods for dynamic ET rely on a "stop-and-go" approach, where the stimulus is paused to acquire a full tilt series, then restarted. This is logistically difficult, time-consuming (often >8 hours), and introduces artifacts. Prolonged electron beam exposure during these pauses can alter the sample (e.g., transforming surface ligands into protective carbon layers), preventing the observation of true dynamics.
• Dose Efficiency: Acquiring multiple full tilt series for different time points results in excessive electron doses, which can destroy beam-sensitive specimens or induce artificial changes.
• Continuous Acquisition Trade-off: While "continuous" tilting schemes exist, standard reconstruction methods cannot handle them because they require a complete, static angular range for each 3D frame.

2. Methodology: DIP-STER

The authors propose DIP-STER (Deep Image Prior in Space-Time Environment Reconstruction), a self-supervised deep learning framework that enables continuous 3D imaging from a single, continuous tilt series.

• Data Acquisition (GRS Scheme):

  • Instead of a linear incremental tilt, the authors use a Modified Golden Ratio Scanning (GRS) scheme. This continuously subdivides the tilt range (e.g., ±70°) irrationally, ensuring that projections are acquired at unique, non-repeating angles over time.
  • This allows for continuous tilting and data acquisition while the specimen is evolving (e.g., being heated).

• Reconstruction Algorithm (Self-Supervised Learning):

  • Architecture: The method combines Deep Image Priors (DIP) with Manifold Learning. It uses a Convolutional Neural Network (CNN) and a mapping network (MAPNET).
  • Input: The network takes a latent vector derived from a 3D manifold coordinate system: Time ($t_i$), Spatial Position ($y_k$), and Projection Angle ($\theta_i$).
  • Process:
    1. The network generates a 2D orthoslice (a slice of the 3D volume).
    2. A forward projection model ($H$) simulates the projection of this slice at the specific angle $\theta_i$.
    3. The simulated projection is compared to the actual experimental projection at that specific time and angle.
    4. Loss Function: The network is optimized using the Geman-McClure loss function (robust to noise/outliers) combined with Total Variation (TV) regularization to suppress noise.
  • Key Innovation: Unlike supervised learning, DIP-STER requires no external training dataset. It learns the structure and dynamics directly from the single tilt series by enforcing consistency between the generated 3D volume and the acquired 2D projections across time and space.

• Output: The trained network can reconstruct a full 3D volume at any arbitrary time point within the acquisition window, generating a "volume-time series" (4D data).

3. Key Contributions

• Continuous 3D Imaging: Demonstrated the first framework to recover dynamic 3D structures from a single continuous tilt series without pausing the experiment.
• Dose Efficiency: Achieved a significant reduction in electron dose (order of magnitude lower than "stop-and-go" methods) by sharing projections across multiple time-resolved reconstructions.
• Self-Supervised Approach: Eliminated the need for large, labeled training datasets by utilizing the self-supervised nature of the DIP architecture adapted for spatio-temporal manifolds.
• Quantitative Dynamics: Enabled the quantification of dynamic properties (e.g., alloying percentages, surface-area-to-volume ratios) with high temporal resolution.

4. Results

The method was validated through simulations and experimental studies on gold (Au) and silver-gold (Au@Ag) nanoparticles under heating:

• Synthetic Validation:

  • Au Nanoshell (Beam Damage/Etching): DIP-STER accurately reconstructed the thinning of walls and formation of gaps over time. It outperformed conventional SIRT (using moving time windows), which suffered from severe motion blur and missing wedge artifacts.
  • Au@Ag Nanorod (Alloying): The method successfully tracked the diffusion of Au into the Ag shell. Quantitative metrics (RMSE < 8%, SSIM > 0.9) showed high fidelity to ground truth, whereas SIRT failed to resolve the internal mixing dynamics clearly.

• Experimental Validation:

  • Au Nanostars (220°C): Tracked the morphological evolution where branches shrank and material redistributed to the core. The results showed full redistribution, contrasting with previous "stop-and-go" studies where ligand degradation prevented reshaping.
  • Au@Ag Nanocubes (350°C–400°C): Monitored the alloying process. The silica shell maintained the particle shape while the internal core-shell structure blended. The method quantified the alloying progression (from ~10% to >50%) by analyzing intensity histograms, revealing an inflection point at ~15 minutes.
  • Beam Sensitivity: The continuous approach used significantly lower accumulated doses ($3.5 \times 10^3$ and $5.7 \times 10^3$ e-/Ų) compared to the estimated $6 \times 10^4$ e-/Ų for equivalent "stop-and-go" experiments, preventing beam-induced artifacts.

5. Significance

• Unlocking True Dynamics: DIP-STER allows researchers to observe nanoscale transformations (melting, alloying, growth) as they happen, without the artifacts introduced by pausing experiments or excessive beam damage.
• Beam-Sensitive Materials: The low-dose capability makes it feasible to study highly sensitive materials (e.g., MOFs, halide perovskites) that were previously impossible to image in 3D under operating conditions.
• Efficiency: It drastically reduces experiment time and increases throughput, moving dynamic ET from a "quasi-in situ" technique to a true "in situ" capability.
• Generalizability: The modular nature of the neural network allows for the incorporation of additional priors (e.g., periodic behavior, specific material constraints), making it a versatile tool for various dynamic electron microscopy applications.

In conclusion, this work establishes a practical, accurate, and dose-efficient framework for time-resolved 3D characterization, fundamentally advancing the ability to study nanomaterial evolution under operating conditions.