Technical Summary: Stitching and Dimensionality Effects on Large Artificially Generated Volume Datasets

Problem Statement

Deep learning-based image generation, particularly for style-transfer tasks, faces inherent computational constraints when processing large-scale scientific datasets (e.g., cryo-electron microscopy volumes). Current hardware limits the size of data processable in a single forward pass, necessitating the division of large volumes into smaller patches, processing them individually, and reassembling them—a process termed "stitching."

This assembly process introduces stitching artifacts, where neighboring patches fail to align perfectly at their borders due to independent processing, normalization, or the use of padded convolutions that introduce non-inherent pixel information. While these artifacts are known to impact segmentation tasks, their specific influence on generative models for style-transfer and downstream scientific analysis remains poorly understood. Furthermore, researchers face a critical decision regarding patch dimensionality: whether to assemble volumes from 2D slices (slice-by-slice) or 3D patches. While 3D approaches offer increased contextual information, they incur significant computational costs and memory limitations, often forcing the use of smaller batch sizes that may destabilize training.

Methodology

The authors investigated these issues using CycleGAN models trained on cryo-electron microscopy (cryoEM) datasets, specifically translating between rat and human mitochondrial volumes (MitoEM dataset) and validating on the openorganelle dataset.

Experimental Design

The study systematically evaluated three primary variables:

1. Stitching Strategies:
   • Tile-and-Stitch: Uses valid convolutions (no padding) during training and inference. Output patches are cropped according to the downsampling factor to remove border artifacts, and neighboring patches are shifted to match the crop size, ensuring no overlap.
   • Padded Convolutions with Averaged Overlap: Uses zero-padding during inference with a "halo" of reflective padding. Patches are assembled with overlap, and the overlapping regions are averaged to mitigate seams.
   • Valid Convolutions without Overlap: Uses valid convolutions but maintains the original input patch size for the step size, resulting in no overlap and no cropping of the output.
2. Patch Dimensionality: Comparison of 2D models (processing slice-by-slice) versus 3D models (processing volumetric patches). A third approach, Orthoslice, was tested, which ensembles predictions from three orthogonal 2D directions.
3. Evaluation Metrics:
   • Perceptual Quality: Measured using the Fréchet Inception Distance (FID) between generated volumes and the target domain.
   • Downstream Performance: Measured via Intersection-over-Union (IoU) on a semantic segmentation task (mitochondria segmentation) using a 3D U-Net trained on the generated volumes.

Model Architecture and Training

• Generators: UNet-based architectures with 5 downsampling steps using strided convolutions.
• Training Adjustments: Instance normalization statistics were frozen during inference to prevent patch-specific normalization artifacts. An additional Structural Similarity (SSIM) loss was incorporated to preserve structural features.
• Data Handling: Patches with low variance (mostly background) were excluded during training. Global normalization statistics were collected during training and frozen for inference to ensure consistency across patches.

Key Results

1. Limitations of Perceptual Metrics (FID)

The study found that FID scores are insufficient for evaluating the quality of generated scientific volumes in the context of downstream tasks.

• FID failed to detect subtle stitching artifacts that significantly impacted downstream segmentation performance.
• Volumes with the best FID scores (closest to the target domain perceptually) did not necessarily yield the best segmentation results. Conversely, volumes with poor FID scores sometimes correlated with lower segmentation performance, but the metric was not a reliable predictor of task-specific utility.

2. Impact of Stitching on Downstream Tasks

• Artifact Propagation: Even subtle stitching artifacts, invisible to the naked eye in probability maps, propagated to the final segmentation masks, causing errors along patch boundaries.
• Optimal Stitching: The Tile-and-Stitch approach (valid convolutions with cropping) consistently produced the most robust downstream segmentation results, outperforming overlap-averaging and no-overlap methods.
• 3D vs. 2D Stitching: The negative impact of stitching artifacts (e.g., in overlap-averaging methods) was more pronounced in 3D models than in 2D models.

3. Dimensionality and Training Stability

• Training Stability: 2D models trained with larger batch sizes were significantly more stable than 3D models, which were often limited to a batch size of 1 due to memory constraints. This stability led to more consistent FID scores and fewer extreme outliers in 2D training.
• Performance Trade-off: While 3D models with artifact-free stitching (Tile-and-Stitch) marginally outperformed 2D models in downstream segmentation tasks, the improvement was small (differences often appearing only in the 3rd or 4th decimal place of IoU). The authors conclude that this marginal gain barely justifies the additional computational cost of training and storing 3D models.
• Ensembling (Orthoslice): Ensembling predictions from three orthogonal directions improved the quality of low-quality volumes (those with initial artifacts) but provided no benefit for already high-quality outputs.

Significance and Claims

The paper claims to be the first to systematically investigate the interplay between stitching strategies, patch dimensionality, and their combined effect on both perceptual metrics and downstream scientific tasks in style-transfer for large 3D biomedical datasets.

The authors emphasize three main contributions:

1. Metric Validity: They demonstrate that perceptual metrics like FID are inadequate for evaluating domain adaptation in biomedical imaging when the ultimate goal is a downstream analytical task. Evaluation must include downstream performance metrics.
2. Best Practices: They establish that artifact-free stitching (via valid convolutions and cropping) is critical for maximizing downstream performance, even if the visual artifacts are imperceptible. They also recommend global normalization statistics to prevent patch-specific artifacts.
3. Practical Guidance: The study suggests that while 3D models offer a theoretical advantage through increased context, the practical benefits are often negated by training instability caused by small batch sizes. For most applications, stable 2D models with rigorous stitching protocols may be the more efficient choice, unless the marginal performance gain of 3D models is strictly required.

Finally, the authors released a revised, memory-efficient CycleGAN codebase capable of handling arbitrarily large datasets via Zarr format, designed to facilitate transparent and artifact-free stitching for the bioimaging community.