A high-performance end-to-end 3D CLEM processing workflow for facilities

This paper presents a modular, open-source, and scalable end-to-end workflow that integrates existing and new tools to streamline the processing, registration, segmentation, and visualization of 3D Correlative Light and Electron Microscopy (CLEM) datasets, thereby lowering technical barriers and enhancing throughput for research facilities.

The Gist

Imagine you are trying to solve a massive, 3D jigsaw puzzle, but the pieces come from two completely different worlds.

World 1 (Light Microscopy): This is like looking at a city from a helicopter. You can see the bright, colorful lights of specific buildings (like a glowing red skyscraper or a green park) because they have special signs. You know what the buildings are, but the details are a bit blurry. You can't see the bricks or the windows.

World 2 (Electron Microscopy): This is like landing on the street and looking at the buildings through a microscope. You can see every single brick, every crack in the pavement, and the intricate texture of the walls. But everything is black and white, and you have no idea which building is the "red skyscraper" you saw from the helicopter.

The Problem: Scientists want to combine these two views to get the full picture: the colorful identity of the object plus the ultra-detailed texture. This is called CLEM (Correlative Light and Electron Microscopy).

However, putting these two puzzles together is a nightmare.

1. The Drift: As the electron microscope scans the sample slice by slice, the sample can wiggle or shift slightly, like a wobbly stack of pancakes.
2. The Noise: The images are often grainy and fuzzy.
3. The Matching: Figuring out exactly where the "red skyscraper" from the helicopter view sits inside the "brick wall" view is incredibly hard.
4. The Size: These 3D puzzles are huge. They are so big that a normal laptop crashes trying to open them.

The Solution:
The authors of this paper built a "Digital Assembly Line" (a software workflow) to fix all these problems automatically. Think of it as a factory line where raw, messy data goes in one end, and a perfect, colorful, 3D animated movie comes out the other.

Here is how their "factory" works, step-by-step:

1. Straightening the Wobbly Stack (Alignment)

Imagine you have a stack of paper that got slightly crooked.

• The Tool: They use a tool called Taturtle (which looks for special "guide stripes" drawn on the sample) or AMST2 (a smart algorithm that guesses how to straighten the stack even without guide stripes).
• The Analogy: It's like a robot arm that gently nudges every single slice of the pancake stack until they are perfectly aligned, so the image doesn't look like a broken mirror.

2. Cleaning the Grainy Photo (Denoising)

Sometimes the electron microscope photos are like an old, grainy security camera video.

• The Tool: They use Noise2Void, an AI that acts like a photo editor.
• The Analogy: It's like using a "magic eraser" that removes the static and fuzz from the image without blurring the important details. It makes the bricks look sharp again.

3. Teaching the Computer to Find the Objects (Segmentation)

Now that the image is straight and clean, the computer needs to know which part is a "mitochondrion" (a tiny power plant inside a cell) and which part is the background.

• The Tool: They use Empanada-MitoNet, a neural network (a type of AI brain).
• The Analogy: Imagine you want a robot to find all the red cars in a parking lot. If you just tell it "find red things," it might get confused. But if you show the robot 50 examples of red cars first (this is called retraining), it becomes an expert. The authors showed that if you "teach" the AI with a few examples from your specific dataset, it becomes incredibly accurate, finding the objects 94% of the time.

4. Stitching the Two Worlds Together (Registration)

Now we need to merge the "helicopter view" (colorful) with the "street view" (detailed).

• The Tool: BigWarp.
• The Analogy: Imagine you have a transparent map of the city (the colorful view) and a photo of the street (the detailed view). You place a few "pins" on matching landmarks (like a specific tree or a unique building corner) on both images. The software then stretches and warps the transparent map until the pins line up perfectly with the photo. Now, you can see the glowing red building inside the detailed brick wall.

5. The Grand Finale: The 3D Movie (Visualization)

Finally, you have all the data. How do you show it to the world?

• The Tool: Blender with Microscopy Nodes.
• The Analogy: This is the movie studio. They take the 3D model of the cell, the glowing lights, and the detailed textures, and they build a virtual reality scene. They can spin the cell around, zoom in, and even make an animation showing how the parts move. It turns a boring spreadsheet of numbers into a beautiful, understandable movie.

Why is this a big deal?

Before this paper, doing this required being a computer expert, a programmer, and a biologist all at once. You needed expensive software that only rich labs could afford.

This new workflow is:

• Free: It's open-source (like Linux or Wikipedia).
• Modular: You can swap out parts if you need to, like changing tires on a car.
• Scalable: It can run on a regular laptop for small jobs, or on a massive "Supercomputer" (like a fleet of trucks working together) for huge datasets.

In short: The authors built a user-friendly, free, and powerful "assembly line" that takes messy, confusing microscope data and turns it into a clear, beautiful, 3D story that anyone can understand. This helps scientists everywhere study the tiny machinery of life much faster and more accurately.

Technical Summary

1. Problem Statement

Correlative Light and Electron Microscopy (CLEM) integrates the molecular specificity of Light Microscopy (LM) with the ultrastructural detail of Electron Microscopy (EM). While 3D CLEM offers unprecedented spatial understanding of biological samples, its adoption in core facilities is hindered by several critical challenges:

• Fragmentation: Existing workflows rely on disparate, incompatible tools, lacking a unified, modular approach.
• Complexity: The multimodal nature of CLEM data requires precise steps (slice alignment, LM-EM registration, segmentation, visualization) that are technically demanding.
• Accessibility: Many labs lack access to expensive commercial software, and existing open-source solutions often require significant programming expertise.
• Scalability: The massive data volumes generated by 3D FIB/SEM and LM imaging strain standard workstations, necessitating High-Performance Computing (HPC) integration which is often difficult to configure for non-experts.
• Reproducibility: The lack of standardized "recipes" leads to inconsistent results across different facilities.

2. Methodology

The authors developed a modular, end-to-end, open-source pipeline designed to run on standard workstations and scale to HPC environments (specifically utilizing the Flemish Supercomputer Center). The workflow is built around interoperable Python-based tools and graphical user interfaces (GUIs) like Napari, Fiji, and Blender.

The pipeline consists of four main stages:

A. EM Data Pre-processing (Alignment & Denoising)

• Alignment: Addresses spatial drift and Z-axis distortion in 3D FIB/SEM data.
  • With Fiducials: Uses Taturtle (in-house Napari plugin) for template matching on fiducial marks (stripes) combined with Z-compensation, followed by AMST (Alignment to Median Smoothed Template) for non-linear local corrections.
  • Without Fiducials: Uses AMST2, a novel pre-alignment approach that calculates translation transforms based on adjacent slices to correct cumulative bias, followed by AMST for local refinement.
• Denoising: Applies Noise2Void2 (N2V2) via the CAREamics library to enhance data quality while preserving structural details. This step is optional but recommended for segmentation accuracy.

B. LM-EM Registration

• Utilizes BigWarp (embedded in BigDataViewer/MoBIE) for interactive, landmark-based registration.
• Strategy: LM data is treated as the reference (native state), and EM data is transformed to match it, accounting for sample deformation caused by EM preparation (dehydration, embedding).
• Fiducials: Mitochondria (visible in both LM via Mitotracker and EM via morphology) serve as landmarks for precise 3D cross-correlation.

C. Segmentation

• Focuses on the automated segmentation of mitochondria using Empanada and the MitoNet neural network.
• Optimization Strategy:
  • Default Model: Tested on pre-processed data.
  • Retrained Model: A ground truth (GT) dataset is created via semi-manual segmentation (using nnInteractive in Napari). The MitoNet model is retrained on this specific dataset to maximize accuracy.
  • Contrast Handling: Image contrast is inverted to match the training data distribution of MitoNet (originally TEM-based).

D. 3D Visualization and Animation

• Uses Blender with the Microscopy Nodes plugin.
• Capable of rendering large datasets (TIFF and OME-Zarr formats).
• Visualizes LM data as emissive volumes and EM data as light-scattering volumes, with segmentation masks overlaid as 3D meshes.

3. Key Contributions

1. Unified Open-Source Pipeline: A complete, end-to-end workflow that consolidates existing tools (BigWarp, Empanada, Blender) with new developments (Taturtle, AMST2 integration) into a single coherent system.
2. Novel Alignment Tools: Introduction of Taturtle (fiducial-based) and AMST2 (fiducial-free) strategies, providing robust solutions for both scenarios common in facility work.
3. HPC Integration: Demonstrated scalability by running preprocessing (alignment/denoising) and segmentation on HPC clusters using Slurm jobs and containerized environments, making the workflow reproducible and resource-efficient.
4. Model Retraining Protocol: Established a clear methodology for retraining MitoNet on specific datasets, showing that this step is critical for high-precision segmentation, often outperforming simple preprocessing.
5. Accessibility: The entire pipeline is open-source (BSD-3-Clause), includes GUIs (Napari), and provides tutorials and test datasets to lower the barrier to entry for non-programmers.

4. Results

The workflow was validated using two distinct datasets:

• Dataset 1 (MEFs with Fiducials):
  • Alignment: Combined Taturtle-AMST and AMST2 reduced displacement errors from 0.75 nm (raw/Taturtle only) to **0.25 nm**, significantly improving Z-axis consistency.
  • Visualization: Successfully rendered a 2.5 GB 3D SIM dataset and a 17 GB FIB/SEM volume with high isotropic resolution (6 nm).
• Dataset 2 (HeLa cells, No Fiducials):
  • Preprocessing Impact: Applying AMST2 alignment and N2V2 denoising improved mitochondrial segmentation accuracy from 40% to **60%**.
  • Model Retraining: Retraining MitoNet on a small ground-truth subset boosted segmentation accuracy to ~94%, even on data without denoising. This demonstrated that model adaptation is more impactful than preprocessing alone for high-precision tasks.
  • Robustness: The pipeline successfully handled a dataset lacking fiducial markers, proving the efficacy of the AMST2 approach.

5. Significance

• Facility Standardization: Provides a "railroaded" (standardized) approach for core facilities to offer 3D CLEM as a service, reducing technical barriers and ensuring reproducibility across different user projects.
• Democratization of CLEM: By relying entirely on open-source software and offering GUI-based interfaces, the workflow makes advanced 3D CLEM analysis accessible to labs without commercial software budgets or deep coding expertise.
• Scalability for Big Data: The integration with HPC and scalable data formats (OME-Zarr) addresses the growing challenge of managing massive 3D imaging datasets, future-proofing the workflow for increasing data volumes.
• Scientific Impact: The ability to accurately correlate molecular data (LM) with ultrastructure (EM) in 3D enables deeper biological insights, such as visualizing membrane contact sites and organelle interactions with high fidelity.

In conclusion, this paper presents a robust, flexible, and accessible framework that transforms 3D CLEM from a niche, highly technical endeavor into a standardized, high-throughput methodology suitable for widespread adoption in the life sciences.