Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy

1. Problem Statement

Transmission Electron Microscopy (TEM) generates vast amounts of data, yet over 90% of it remains unpublished and unused, often discarded to save storage space. These "leftover" datasets are rich in detail and variation but are rarely utilized because:

• Data Heterogeneity: Materials science TEM involves diverse samples, operating modes, and acquisition settings, making it difficult to train models that generalize across different conditions.
• Noise and Dose Efficiency: Modern TEM prioritizes dose efficiency (minimizing electron damage), resulting in very noisy data. Current machine learning (ML) denoising models are typically trained from scratch for specific noise levels or samples, lacking robustness to changes in instrument parameters (e.g., magnification, dwell time, beam current).
• Lack of Paired Data: Existing open-source datasets are either small, synthetic, or lack the rich, paired acquisition metadata necessary to train models that understand the relationship between image appearance and instrument state.

The core challenge is to leverage these unused, metadata-rich archives to create generalizable ML workflows that can understand instrument states, perform style transfer, and denoise images without requiring new paired training data for every parameter change.

2. Methodology

The authors propose Contrastive Image-Metadata Pre-training (CIMP), a framework inspired by CLIP (Contrastive Language-Image Pre-training), adapted for the specific domain of High-Angle Annular Dark-Field Scanning TEM (HAADF-STEM).

A. Dataset Construction

• Source: 7,330 HAADF-STEM images collected from a single Titan microscope at an anonymized materials research lab.
• Content: Covers >30 distinct material systems (primarily heterogeneous catalysts like Pt/CeO2, Pd/ZrO2) with >130 samples.
• Metadata: Each image is paired with seven acquisition parameters:
  1. Pixel size
  2. Dwell time
  3. Beam convergence angle
  4. Beam current
  5. Detector gain
  6. Detector offset
  7. Inner collection angle
• Evaluation Sets: Specific series of images taken of the same sample with varied single parameters (dwell time, gain, offset) were reserved for qualitative evaluation, ensuring no data leakage during training.

B. CIMP Architecture

• Objective: Learn a joint embedding space where HAADF-STEM images and their metadata vectors are aligned via contrastive loss.
• Encoders:
  • Image Encoder: A ResNet-18 (trained from scratch) was found to outperform or match ImageNet-pretrained Vision Transformers (ViT), likely due to the significant domain gap between natural images and STEM micrographs.
  • Metadata Encoder: A Multi-Layer Perceptron (MLP) processing the 7 acquisition parameters.
• Training: Uses an InfoNCE contrastive loss. The model learns to match an image to its correct metadata vector among a batch of negative samples.

C. Downstream Applications

The learned embeddings serve as a foundation for two main tasks:

1. Linear Probe Recovery: Testing if specific acquisition parameters can be linearly decoded from the frozen visual embeddings.
2. Generative Style Transfer: A GAN-based generator (U-Net with FiLM conditioning) uses the CIMP embeddings to translate an experimental image into the style it would have under different acquisition parameters.
   • Conditioning: The generator takes a source image and two embeddings (source metadata $e_{id}$ and target metadata $e_{tgt}$).
   • Loss Functions: Adversarial loss ($L_{LSGAN}$), Cycle consistency ($L_{cyc}$), Identity loss ($L_{id}$), and a perceptual embedding loss ($L_{emb}$) ensuring the output matches the target style in the CIMP space.

3. Key Contributions

1. Dataset Release: The first large-scale (7,330 images), paired image-metadata dataset for experimental HAADF-STEM, covering diverse materials and instrument settings.
2. CIMP Framework: A novel pre-training method that learns a joint embedding space for TEM images and metadata, treating these embeddings as a foundational layer for downstream tasks rather than a task-specific model.
3. Physics-Informed Denoising: Demonstrating that the style-transfer network, when conditioned on virtual increases in dwell time and beam current, acts as a superior denoiser compared to self-supervised baselines like Noise2Void.
4. Linear Decodability: Proving that the contrastive objective, without direct supervision of metadata values, results in embeddings where seven distinct physical parameters are individually recoverable.

4. Results

A. Embedding Learning & Retrieval

• Hyperparameters: Optimal performance was achieved with 512x512 image crops, a ResNet-18 backbone, and an effective batch size of 512.
• Retrieval Accuracy: The model achieved a Top-1 retrieval accuracy of 0.844 (matching an image to its correct metadata vector).
• Backbone Comparison: ResNet-18 (trained from scratch) outperformed ImageNet-pretrained ViT (84.4% vs 82.8% Top-1), highlighting the domain specificity of STEM data.

B. Metadata Recovery (Linear Probing)

• Using Ridge regression on frozen embeddings, the model successfully recovered all seven parameters.
• Performance:
  • Detector Gain: Best recovered (SMAPE ≈ 5.3%, $R^2 > 0.82$).
  • Log-transformed parameters (Pixel size, Dwell time, Beam current): Showed lower $R^2$ (0.66–0.78) and higher SMAPE (28–47%) due to the multiplicative nature of errors after re-exponentiation, but were still linearly decodable.
• Significance: This confirms the embeddings capture a "factored" representation of physical parameters rather than collapsing into a single "noise vs. clean" axis.

C. Style Transfer & Denoising

• Qualitative Trends: The generator successfully mimicked physical trends (e.g., higher dwell time = smoother/less noisy; higher beam current = brighter).
  • Note: The model learned dataset biases (e.g., convergence angle correlated with beam current due to aperture changes) rather than pure physics in some cases.
• Quantitative Metrics: On held-out evaluation sets, the CIMP GAN outperformed the "no-edit" baseline in Gain transfer (SSIM, PSNR, LPIPS) and matched the baseline for Dwell time.
• Denoising Performance:
  • Compared to Noise2Void, the CIMP GAN produced more realistic images.
  • Noise2Void suffered from checkerboard artifacts and hallucinated background content.
  • CIMP GAN preserved structural details while effectively reducing noise by virtually increasing the electron dose.

5. Significance and Impact

• Paradigm Shift: Moves TEM data analysis from "train from scratch for every specific condition" to "pre-train on archives to learn generalizable instrument physics."
• Data Utilization: Validates that "leftover" experimental data is a valuable resource for training robust ML models, potentially unlocking the 90% of unused TEM data.
• Generalizability: The CIMP embeddings can serve as conditioning signals for segmentation, analysis, or retrieval tasks, making ML workflows robust to variations in microscope settings.
• Community Resource: The release of the dataset and code lowers the barrier for other labs to adopt data-driven methods, encouraging a culture of data sharing in electron microscopy.

6. Limitations

• Boundary Conditions: The model struggles with extreme parameter values (e.g., image clipping/saturation) because such "bad data" is typically discarded by operators and absent from the training set.
• Dataset Scale: While the largest of its kind in materials TEM, 7,330 images is small compared to computer vision datasets (e.g., ImageNet's 1.28M images), limiting the scale of model training.
• Dataset Bias: The model sometimes learns correlations present in the specific dataset (e.g., aperture size affecting both convergence angle and beam current) rather than pure physical laws.

In conclusion, this work establishes a new foundation for materials TEM by demonstrating that contrastive pre-training on metadata-rich archives creates powerful, physics-aware representations that enable advanced generative tasks and robust denoising.