Technical Summary: STEMGYM

Problem Statement

Autonomous scientific imaging in Scanning Transmission Electron Microscopy (STEM) operates under a fundamental constraint: every measurement deposits electron dose, which causes irreversible damage to the specimen through knock-on displacement, radiolysis, and heating. This is particularly critical for beam-sensitive materials like 2D chalcogenides. The prevailing hypothesis in the field posits that adaptive navigation (e.g., Bayesian optimization, reinforcement learning) is the primary lever for improving dose efficiency by concentrating measurements on high-information regions. However, progress has been difficult to track due to a lack of standardized benchmarks. Existing studies use incompatible specimens, instruments, dose budgets, and task definitions, making it impossible to isolate whether performance gains stem from the navigation strategy, the perception (analysis) pipeline, or the planning logic. Furthermore, no community-standard benchmark exists for STEM analogous to CryoBench for Cryo-EM.

Methodology

The authors introduce STEMGYM, an open-source Gymnasium-compatible benchmark designed to decouple and evaluate the components of the autonomous acquisition pipeline under a unified dose budget.

1. Environment and World Generation

• Simulation: The benchmark utilizes PRISM multislice simulation (via abTEM) to generate 15 physics-simulated STEM worlds.
• Materials: Five material systems spanning four crystal structure families:
  • Perovskites: SrTiO₃ (grain-boundary vacancies) and BaTiO₃ (ferroelectric phase boundaries).
  • Semiconductors: Si/Ge (compositional gradients) and GaN (InGaN quantum wells).
  • Metals: Pt nanoparticles on amorphous carbon.
• Difficulty Levels: Three tiers (Easy, Medium, Hard) defined by vacancy rates, phonon configurations, and intended dose levels (ranging from $10^3$ to $10^4$ e Å⁻²).
• Task Definitions: Four characterization tasks:
  • Defect Census: Counting and locating vacancies/substitutions (F1 score).
  • Phase Mapping: Identifying crystal phases (IoU).
  • Targeted Characterization: Locating specific defects (Spatial F1).
  • Particle Census: Detecting and sizing nanoparticles (Composite score).

2. Metric: Dose-Efficiency Curve Area (DEC-AUC)

To evaluate performance, the authors propose DEC-AUC, a single scalar metric capturing the Pareto frontier between information (task score) and dose.

• The agent operates under a hard dose budget ($B$).
• The Dose-Efficiency Curve $s(f)$ plots the expected task score against the consumed dose fraction $f$.
• DEC-AUC is the area under this curve, integrating both final accuracy and the speed of convergence. This avoids the "order-discontinuity" of time-to-threshold metrics where agents may fail to reach a high score within the budget.

3. Agent Configurations

The study evaluates 33 agent configurations to perform a causal decomposition of the acquisition pipeline:

• Naïve Baselines: Random sampling, Raster scanning, and GP-UCB (Gaussian Process Upper Confidence Bound) navigation, all paired with a simple heuristic analyst.
• Equipped Baselines: The same navigation policies paired with a trained neural analyst (U-Net ensemble for atom finding, CNN for defect classification, ResNet for phase identification).
• Complex Agents: STEMAgent (a multi-agent system with GP-UCB navigation, neural analyst, and a Finite State Machine planner), Reinforcement Learning agents (DQN, PPO, SAC), and Vision-Language Model (VLM) agents (e.g., Claude, Gemini, Llama).
• Control Variants: Ablations of STEMAgent (removing planner, uncertainty estimation) and compressed sensing baselines.

Key Results

1. Perception Dominates Navigation

The most significant finding is that the analyst (perception) is the dominant determinant of dose efficiency, not the navigator.

• Magnitude of Gain: Pairing a trained CNN analyst with a trivial Raster scan increases the DEC-AUC by 5.5× compared to a raster scan with a naïve heuristic (0.287 vs. 0.052 on defect census).
• Navigation Insignificance: Once a capable analyst is provided, substituting the raster scan with adaptive strategies (Bayesian GP-UCB or FSM-based planning) yields no statistically significant gain. The DEC-AUC values for Raster+Analyst, GP-UCB+Analyst, and STEMAgent are statistically indistinguishable (p > 0.05).
• Variance Source: The variance in performance is driven by cross-world material heterogeneity and analyst capability, not by the stochasticity of the navigation policy.

2. Generalization and Specialization

• Material Specificity: The trained analyst performs well on perovskites (SrTiO₃, BaTiO₃) but fails on non-perovskite materials (SiGe, GaN) and nanoparticles when not specifically trained for them.
• VLM Performance: Production-tier Vision-Language Models (VLMs) underperform task-specific CNNs by approximately 13× on crystallographic defect analysis. However, VLMs excel on Pt nanoparticle census (where CNNs score 0.000 due to architectural specialization), suggesting a need for hybrid perception routers rather than a single universal model.
• Reinforcement Learning: RL agents (DQN, PPO, SAC) trained without a trained analyst perform near-randomly. When equipped with the analyst, SAC approaches the performance of Raster+Analyst, while PPO collapses, indicating that algorithm choice matters less than the presence of a competent perception module.

3. Robustness

• Budget Sensitivity: The superiority of the "Raster + Trained Analyst" configuration holds across a 50× dose budget range (1,000 to 50,000 e Å⁻²).
• Real-Data Validation: A sanity check on real Sm-doped BiFeO₃ acquisitions confirms that the "equipped vs. naïve" direction is preserved, even though absolute scores drop due to the sim-to-real domain gap.

Significance and Claims

The paper claims to reframe the investment of machine learning efforts in autonomous electron microscopy:

1. Shift in Focus: The community's focus on developing complex adaptive navigation strategies is misplaced if the perception pipeline is weak. The "principal lever" for dose efficiency is analysis, not navigation.
2. Benchmarking Infrastructure: STEMGYM provides the first standardized protocol to isolate the contributions of perception, navigation, and planning, addressing a primary bottleneck for ML adoption in microscopy identified by the Microscopy Hackathon.
3. Open-Loop Limitation: The results specifically apply to open-loop navigators (where acquisition decisions do not condition on analyst predictions). The authors note that closing the loop (using analyst uncertainty to guide navigation) remains an open direction for future gains, but current evidence suggests that simply "looking everywhere" with a smart analyst is currently more effective than "looking selectively" with a dumb analyst.

The authors conclude that for autonomous STEM, the priority should be developing robust, generalizable perception models and hybrid architectures that route to the appropriate analyst, rather than optimizing navigation policies in isolation.