Neural Field-Based 3D Surface Reconstruction of Microstructures from Multi-Detector Signals in Scanning Electron Microscopy

This paper introduces NFH-SEM, a neural field-based hybrid framework that leverages multi-view and multi-detector SEM signals integrated with a learnable physics-informed forward model to achieve high-fidelity, self-calibrated 3D surface reconstruction of microstructures, effectively overcoming limitations in textureless regions and shadowing artifacts.

The Gist

Imagine you have a tiny, intricate sculpture, like a pollen grain or a microscopic gear, and you want to know exactly what it looks like in 3D. You put it under a powerful microscope called a Scanning Electron Microscope (SEM).

The problem? The SEM is like a very smart, high-tech camera that only takes flat, 2D black-and-white photos. It sees the surface, but it doesn't "know" how deep a hole is or how high a bump stands. It's like looking at a shadow puppet show; you see the shape, but you can't tell if the puppet is flat or has depth.

For years, scientists have tried to turn these flat photos into 3D models, but the old methods are like trying to guess the shape of a mountain just by looking at a foggy photo. They get confused by smooth surfaces, get tricked by shadows, and often produce models that look like melted wax.

Enter NFH-SEM. Think of this new method as a super-smart, self-correcting 3D sculptor that uses a special kind of "digital clay" called a Neural Field.

Here is how it works, broken down into simple steps:

1. The "Flashlight" Trick (Multi-Detector Signals)

Standard cameras use one light source. But the SEM has a special detector with four quadrants (like four separate eyes looking from different angles).

• The Analogy: Imagine shining a flashlight on a bumpy rock from the North, South, East, and West.
  • If the North side is bright and the South side is dark, you know the rock is tilted toward the North.
  • If a part of the rock is in shadow, you know something is blocking the light.
• The Problem: Old methods got confused by the shadows, thinking a shadow was a deep hole.
• The NFH-SEM Solution: It doesn't just look at the shadows; it learns the rules of the shadows. It has a built-in "physics engine" that understands exactly how electrons bounce off surfaces. It can tell the difference between a "real shadow" caused by a bump and a "fake shadow" caused by the angle of the light.

2. The "Digital Clay" (Neural Field)

Instead of building the 3D model out of a grid of tiny cubes (which is rigid and blocky), NFH-SEM uses a Neural Field.

• The Analogy: Think of a traditional 3D model as a Lego castle. It's made of distinct blocks. If you want a smooth curve, you need thousands of tiny blocks.
• The NFH-SEM Approach: Think of the Neural Field as liquid clay or a smooth, continuous fog. It doesn't have "blocks." It can define a surface that is infinitely smooth and detailed. The AI "sculpts" this digital clay by constantly asking, "If I move this point of clay, does it match the photos better?"

3. The "Self-Teaching" Loop

This is the magic part. Usually, to calibrate a microscope, you need a perfect reference object (like a known sphere) to tell the machine how it's seeing things.

• NFH-SEM's Superpower: It doesn't need a reference object. It calibrates itself.
• How? As it sculpts the digital clay, it simultaneously learns the "rules" of the microscope's lighting. It's like a sculptor who is also learning how the studio lights work at the same time. If the light seems weird, it adjusts its understanding of the light and the shape of the sculpture together until everything fits perfectly.

4. The "Shadow Hunter"

Shadows are the enemy of 3D reconstruction.

• The Analogy: Imagine trying to draw a map of a cave system, but your flashlight keeps getting blocked by stalactites, leaving dark spots.
• The NFH-SEM Strategy: It uses a "Shadow Hunter" algorithm. It looks at the photos, identifies the dark spots that are just shadows (not real holes), and says, "Ignore this part for a moment, it's just a shadow." It then fills in the missing geometry based on the rest of the data. It does this over and over, getting better at spotting shadows with every pass.

Why Does This Matter?

The paper shows this method working on some amazing tiny things:

• Pollen Grains: It revealed tiny, sticky textures on peach pollen that help them stick to bees. This helps us understand how plants reproduce.
• Fractured Metal: It showed the microscopic "steps" where a piece of silicon carbide broke. This helps engineers understand why materials fail.
• Tiny 3D Prints: It measured layers on a 3D-printed object that were thinner than a human hair (less than 500 nanometers!).

The Bottom Line

NFH-SEM is like giving a blind sculptor a set of four flashlights and a brain that learns physics. It takes flat, confusing 2D photos from a microscope and turns them into incredibly accurate, high-definition 3D models, even when the surface is smooth, the shadows are tricky, or the object is smaller than a grain of sand. It opens the door for scientists to see the hidden 3D world of the very small with unprecedented clarity.

Technical Summary

1. Problem Statement

The 3D characterization of microstructures is critical for materials science, biology, and industrial manufacturing. While Scanning Electron Microscopes (SEMs) provide high-resolution 2D images, they do not inherently capture 3D geometry. Existing methods for 3D reconstruction from SEM data face significant limitations:

• Multi-view Methods (SfM/MVS): Rely on feature matching, which fails on microstructures that are often textureless, repetitive, or smooth.
• Single-view Methods (Photometric Stereo): Use multi-detector signals (Backscattered Electrons - BSE) to estimate surface normals. However, they suffer from severe shadowing artifacts (where geometric occlusion blocks electron signals), require complex calibration using reference samples, and often produce distorted geometries on discontinuous surfaces.
• Learning-based Methods: State-of-the-art neural field and Gaussian Splatting methods trained on macroscopic RGB data fail to generalize to the microscopic SEM domain due to the lack of physical priors, domain-specific data, and the inability to model electron scattering physics.

2. Methodology: NFH-SEM

The authors propose NFH-SEM, a neural field-based hybrid framework that reconstructs high-fidelity 3D surfaces by fusing coarse multi-view geometry with photometric stereo cues from multi-detector SEM signals.

Core Components:

1. Continuous Neural Field Representation:

   • Instead of discrete height maps, the 3D geometry is implicitly modeled as a Signed Distance Function (SDF) parameterized by a Multi-Layer Perceptron (MLP) with multi-resolution hash encoding.
   • This allows for the representation of complex, continuous microstructures.

2. Learnable BSE Forward Model:

   • The method integrates the physics of SEM imaging directly into the optimization. It models the mapping from predicted surface normals ($\hat{n}$) to 4-Quadrant BSE (4Q-BSE) intensities.
   • Physics Embedding: The model replaces the simplified analytical emission terms found in prior work with a learnable fourth-order polynomial for the emission magnification term $R(\theta)$.
   • Self-Calibration: The model includes learnable parameters for detector configuration ($c, d, e$) and polynomial coefficients ($p$) for each of the four quadrants. This allows the system to self-calibrate without needing reference samples, adapting to manufacturing tolerances and specific SEM setups.

3. Iterative Shadow Separation Strategy:

   • A major challenge in BSE imaging is that shadows (caused by self-occlusion) violate the assumption that intensity is solely a function of surface normal.
   • NFH-SEM employs an iterative masking scheme. It identifies regions where the synthesized BSE image significantly deviates from the captured image (indicating shadows) and excludes these regions from the loss calculation during optimization.
   • This creates a feedback loop: better geometry leads to better shadow detection, which in turn leads to cleaner supervision for geometry and the forward model.

4. Three-Stage Optimization Pipeline:

   • Stage I: Supervised only by coarse depth maps (from multi-view SE images) and SDF regularization to initialize the global geometry.
   • Stage II: Introduces 4Q-BSE photometric stereo supervision and jointly optimizes the neural field and the BSE forward model parameters. Shadow masking is not yet active.
   • Stage III: Activates dynamic shadow masking to refine the reconstruction, excluding shadowed pixels from the loss to ensure robustness against severe occlusion.

3. Key Contributions

• A Hybrid Neural Framework: The first neural field-based approach specifically designed for SEM that fuses multi-view geometry with multi-detector photometric cues.
• Learnable Forward Model: A novel physics-informed module that learns the electron scattering and detector response, enabling parameter self-calibration and eliminating the need for reference samples.
• Shadow-Robust Reconstruction: An iterative shadow separation mechanism that effectively handles the "SEM shading effect," allowing accurate reconstruction even in heavily occluded regions.
• New Dataset: The authors released the first multi-view, multi-detector real-world SEM dataset covering diverse samples: Two-Photon Lithography (TPL) microstructures, peach pollen, and Silicon Carbide (SiC) fracture surfaces.

4. Experimental Results

The method was evaluated on real-world samples and a simulated dataset (derived from CAD models of TPL structures).

• Performance vs. Conventional Methods:

  • NFH-SEM significantly outperforms traditional multi-view (Agisoft Metashape) and single-view (gradient integration) methods.
  • It successfully recovers fine details that others miss, such as 478 nm layered features in TPL samples, 782 nm textures on pollen grains, and 1.559 µm fracture steps on SiC particles.
  • Conventional methods often result in over-smoothed surfaces or global distortions due to shadowing.

• Performance vs. Learning-based Methods:

  • Compared to state-of-the-art neural fields (NeuS) and 3D Gaussian Splatting (2DGS, PGSR) trained on RGB data, NFH-SEM achieves superior accuracy.
  • RGB-based methods fail to generalize to SEM data, producing fragmented or distorted geometries because they ignore electron scattering physics.

• Ablation Studies:

  • Removing the learnable forward model or the shadow masking strategy resulted in significant degradation in geometric accuracy (Chamfer distance) and BSE modeling error.
  • The method demonstrated robustness even when initialized with noisy or incomplete coarse geometries.

5. Significance

• Scientific Impact: NFH-SEM enables the accurate 3D characterization of functional microstructures, providing insights into biological mechanisms (e.g., pollen adhesion to pollinators) and material failure analysis (e.g., fracture mechanics).
• Technical Advancement: It bridges the gap between physical imaging models and deep learning, demonstrating that embedding domain-specific physics (electron scattering) into neural fields is crucial for specialized imaging modalities like SEM.
• Practical Utility: By removing the need for reference sample calibration and handling shadows automatically, the method lowers the barrier for high-fidelity 3D SEM reconstruction, making it applicable to a wide range of materials and research fields.

The code and dataset are publicly available at https://github.com/zju3dv/NFH-SEM.