NucleoNet and DropNet: Generalist deep learning models for instance segmentation of nuclei and lipid droplets from electron microscopy images

This paper introduces NucleoNet and DropNet, crowdsourced deep learning models that enable robust, high-quality instance segmentation of nuclei and lipid droplets in electron microscopy images, facilitating the quantification of cellular differences across various cancer models.

The Gist

Imagine you are trying to count and measure every single grain of sand on a beach, but the beach is made of microscopic cells, and the grains are tiny structures inside them called nuclei (the cell's brain) and lipid droplets (the cell's fat storage).

Doing this by hand with a microscope is like trying to count every grain of sand on a beach with a pair of tweezers. It takes forever, your eyes get tired, and you'll probably make mistakes. This is the problem scientists faced with Electron Microscopy (EM) images: they are incredibly detailed, but there are too many of them to analyze manually.

For years, scientists built "smart robots" (AI models) to help count these grains, but they mostly focused on one specific type of grain: mitochondria (the cell's power plants). They ignored the nuclei and the fat droplets because there wasn't enough "training data" (examples) to teach the robots how to spot them.

This paper introduces two new "smart robots" named NucleoNet and DropNet that finally solve this problem. Here is how they did it, explained simply:

1. The "Crowdsourcing" Strategy: Teaching a Class of Students

To teach the robots what a nucleus or a fat droplet looks like, you need thousands of examples where humans have already drawn a circle around them. Since there weren't enough experts to do this, the scientists turned to crowdsourcing.

• The Analogy: Imagine a teacher who needs to grade 30,000 math tests but only has one hour. Instead of doing it alone, they hire a class of 25 high school students.
• The Process: The scientists took thousands of tricky microscope images and uploaded them to a website called Zooniverse. They recruited high school students to draw outlines around the nuclei.
• The Quality Control: To make sure the students were doing a good job, an expert teacher checked a few tests every week. If a student was doing great, they got points; if they were struggling, they got extra help. The final "answer key" was created by combining the work of five different students for every single image. This ensured the data was accurate enough to teach the AI.

2. The Two New Robots: NucleoNet and DropNet

Once the data was ready, they trained two specialized AI models:

• NucleoNet: This robot is an expert at finding nuclei. It can look at a messy, crowded cell and say, "That's a nucleus! And that's another one right next to it!" It works so well that it can even handle nuclei that look like crumpled paper or have deep folds.
• DropNet: This robot is an expert at finding lipid droplets (fat). These are tricky because they look like little bubbles that can be clear, dark, or have weird holes in them. DropNet learned to spot all these variations, distinguishing real fat droplets from other similar-looking blobs in the cell.

3. The "Magic Glasses" (The Software)

Having a smart robot is useless if you can't talk to it. Many AI tools are like super-computers that require a PhD in computer science just to turn on.

The scientists packaged these robots into a user-friendly tool called empanada (a plugin for a program called napari).

• The Analogy: Think of it like a point-and-click video game. You don't need to write code. You just open your microscope image, click a button that says "Find Nuclei," and the robot instantly draws outlines around every single one. It's as easy as using a filter on Instagram, but for scientific research.

4. Putting Them to the Test: The "Fake" vs. "Real" Tumor

To prove these robots were actually good, the scientists used them to compare lab-grown cancer models (cells grown in a dish) against real human tumors taken from a patient.

• The Experiment: They grew breast cancer cells in four different ways: flat on a plate, floating in liquid, in a ball (spheroid), and in a "blood clot" simulation (emboli).
• The Result: They used NucleoNet and DropNet to instantly count and measure the nuclei and fat droplets in all these groups.
• The Discovery: They found that the "blood clot" model (emboli) looked the most like the real human tumor. The other models looked quite different. This is huge because it tells scientists which lab models are actually good for testing new drugs. Without these robots, this comparison would have taken months of manual work; with the robots, it took a few clicks.

Why This Matters

Before this paper, if you wanted to study cell nuclei or fat droplets in electron microscopy, you were stuck doing it by hand or using tools that didn't work well.

Now, thanks to NucleoNet and DropNet, any scientist (even those without a computer science degree) can:

1. Automate the boring counting work.
2. Get accurate results quickly.
3. Compare different biological models to understand diseases better.

It's like giving every biologist a pair of super-powered glasses that instantly highlight the most important parts of a cell, allowing them to focus on the big discoveries rather than the tedious counting.

Technical Summary

1. Problem Statement

• Context: Electron Microscopy (EM) and Volume EM (vEM) are essential for visualizing cellular ultrastructure at nanometer resolution. However, the resulting datasets are vast and complex, making manual segmentation of organelles time-consuming and labor-intensive.
• Gap: While deep learning (DL) has accelerated segmentation for mitochondria (e.g., MitoNet), there is a significant scarcity of ready-to-use, generalist tools for nuclei and lipid droplets (LDs).
• Challenges:
  • Data Scarcity: Unlike mitochondria, there are few large, heterogenous, annotated datasets for nuclei and LDs. Existing models often rely on single, homogeneous datasets (e.g., specific cell lines or imaging modalities), leading to poor generalization.
  • Usability: Existing tools (e.g., specific U-Net models for nuclear envelopes) often require high computational expertise to set up and lack user-friendly interfaces, limiting adoption by non-experts.
  • Benchmarking: There is a lack of accessible, challenging benchmark datasets for nuclei and LDs to objectively evaluate model performance.

2. Methodology

The authors developed a comprehensive pipeline involving data curation, crowdsourcing, model training, and integration into a user-friendly platform.

A. Data Curation and Crowdsourcing

• NucleoNet (Nuclei):
  • Data Sources: Aggregated ~450,000 image patches (1024x1024) from internal archives (FIB-SEM, AT, TEM) and public repositories (EMPIAR, OpenOrganelle, nanotomy, MICrONS, Platynereis).
  • Crowdsourcing: Utilized the Zooniverse platform to train high-school students to annotate nuclei.
    • Protocol: Each image was annotated by 5 students. An expert provided "ground truth" (GT) for ~50 images per batch to score student accuracy.
    • Consensus: Annotations were pooled and proofread by experts to create high-quality GT.
    • Scale: Generated 6,549 high-quality annotated patches from crowdsourcing, supplemented by public datasets (MICrONS and Platynereis) to reach a total training set of 31,082 labeled images.
• DropNet (Lipid Droplets):
  • Data Sources: Curated 1,792 positive patches (512x512) from FIB-SEM, AT, and TEM datasets (liver, kidney, breast, pancreas, tumor lines).
  • Classification: Defined four visual classes for LDs (e.g., dark outline, no outline, unstained center, artifacts) and distinguished them from true negatives (e.g., insulin granules vs. acinar granules).
  • Strategy: Due to limited data, the model was created by fine-tuning an existing internal variant of MitoNet on the curated LD patches.

B. Model Architecture and Training

• Architecture: Both models utilize the Panoptic DeepLab (PDL) architecture, which performs simultaneous semantic segmentation and instance prediction (via X/Y offsets).
• Training Strategy:
  • Pre-training: Models were initialized with weights from CEM1.5M (a massive mitochondria dataset) to improve generalization.
  • Fine-tuning: Progressive unfreezing of encoder layers.
  • Loss Function: Composite loss combining cross-entropy (semantic), focal loss (center prediction), and L1 loss (offset prediction).
  • Hardware: Trained on 4 NVIDIA A100 GPUs.
• Inference:
  • Models are 2D but handle 3D volumes by running inference in orthogonal planes (xy, xz, yz) and merging results.
  • Integrated into the empanada napari plugin (v1.2) for point-and-click usage, including modules for proofreading (split/merge) and fine-tuning.

C. Benchmarking and Comparison

• Benchmarks: Tested on diverse, unseen datasets: Platynereis dumerilii (SBF-SEM), Rat Islet of Langerhans (TEM), and Human Breast Cancer Spheroids (Array Tomography).
• Comparators:
  • nnU-Net: A self-configuring architecture trained on CellMap EM data.
  • CellSAM: A foundational model tuned for cell segmentation.

3. Key Contributions

1. First Generalist Models: NucleoNet and DropNet are the first freely available, generalist DL models dedicated to instance segmentation of nuclei and lipid droplets in 2D/3D EM.
2. Crowdsourced Dataset: Successfully demonstrated a crowdsourcing pipeline using high-school students to generate high-quality, heterogenous training data for complex EM images.
3. Usability: Integrated these models into empanada, a widely used napari plugin, making advanced segmentation accessible to non-experts via a GUI.
4. Benchmarking: Established new benchmark datasets and evaluation metrics for nuclei and LDs in EM, addressing a gap in the field.

4. Results

• NucleoNet Performance:
  • Achieved F1@0.50 scores ranging from 0.633 to 0.818 across diverse benchmarks.
  • Comparison: Outperformed CellSAM significantly (which failed statistically and visually). Performance was comparable to nnU-Net (trained on specific EM data) but with the advantage of being pre-packaged and easier to use.
  • Error Correction: While the base model had some split/merge errors and false positives, simple post-processing steps in the empanada GUI (filtering small instances, erosion/dilation) improved precision from ~0.57 to 0.93 and F1 from ~0.68 to 0.89.
• DropNet Performance:
  • Achieved F1@0.75 scores of 0.770, 0.840, and 0.911 on three benchmarks.
  • Correctly distinguished LDs from similar-looking secretory granules (e.g., ignored insulin granules, segmented acinar granules as per training definition).
  • Successfully separated tightly clustered LDs with minimal need for manual correction.
• Biological Application:
  • Applied models to compare in vitro cancer models (Adherent, Suspension, Spheroid, Emboli) against in vivo tumors.
  • Finding: The "Emboli" model (recapitulating blood flow) showed the closest ultrastructural resemblance to actual tumors in terms of nuclear and LD morphology.
  • Spatial Insight: Revealed a radial gradient in spheroids where nuclei at the periphery were flatter (high aspect ratio) while central nuclei were rounder, similar to tumor tissue.

5. Significance and Future Outlook

• Accelerated Discovery: These tools enable high-throughput, quantitative analysis of nuclear and lipid droplet morphology, moving beyond qualitative observation to robust statistical analysis.
• Generalizability: The models demonstrate that training on diverse, crowdsourced data yields robust "generalist" models capable of handling varied sample preparations and imaging modalities.
• Limitations & Future Work:
  • Current models are restricted to non-dividing cells (dividing nuclei were excluded due to annotation difficulty).
  • They do not segment sub-nuclear structures (e.g., nucleoli) or extremely large pathological LDs.
  • Future work aims to incorporate Vision Transformers, diffusion models, and multi-class segmentation (all organelles simultaneously) to further improve accuracy and handle pathological variability.

In summary, this work bridges the gap between advanced deep learning and practical biological application by providing accessible, high-performance tools for segmenting two critical but previously under-served organelles in electron microscopy.