Transformation Behavior of Images in Latent Space

Technical Summary: Transformation Behavior of Images in Latent Space

Problem Statement
The training of neural networks for histopathology classification often relies on encoding data into a latent space to reduce complexity and improve performance. While data transformations (e.g., flipping, cropping, color augmentation) are standard practice to artificially enlarge datasets, and unsupervised "embedder" networks are increasingly used to create compact, informative representations, the specific behavior of these latent spaces under transformation remains under-investigated.

Ideally, a perfect embedder network trained on infinite data would render the latent space invariant to label-irrelevant transformations, meaning the latent vector for an image and its transformed version would be identical. However, given finite resources, such perfect networks do not exist. The central research question is: How do unsupervised embedder networks behave under commonly applied data transformations? Specifically, the authors seek to quantify the variance of embeddings resulting from standard transformations and assess whether these networks achieve true invariance or merely robustness.

Methodology
The study conducted a systematic analysis of 392 experiments involving eight distinct datasets, seven different embedder networks, and seven specific image transformations.

• Datasets: Two Whole Slide Image (WSI) datasets were utilized:

  1. TCGA: The open-source Colon Adenocarcinoma collection (1,330 WSIs).
  2. CAD: A non-public dataset of colon adenomas from Heidelberg University Hospital and University Hospital Bonn (520 WSIs).
  • Images were processed into 256x256 pixel tiles at resolutions of 0.25, 0.5, 1.0, and 2.0 microns per pixel (mpp). Background tiles were excluded using Canny Edge Detection.

• Networks: Seven pre-trained unsupervised models were evaluated, representing different architectures and training methodologies:

  • ResNet50-based: Meta SwAV (200 epochs, ImageNet baseline), Lunit Barlow Twin, Lunit MoCoV2, and Lunit SwAV (trained on pathology data).
  • Vision Transformer (ViT)-based: Lunit DINO (08 and 16), and the large-scale H-Optimus-0 (H0) by Bioptimus (1.1 billion parameters, trained on 500k pathology slides using DINOv2).
  • Training Methods: Barlow Twin (BT), Momentum Contrast v2 (MoCoV2), Swapping Assignment between Views (SwAV), and self-Distillation with NO labels (DINO).

• Transformations: The study applied seven transformations to image tiles: horizontal flip, vertical flip, random crop, color jitter, conversion to grayscale, color normalization, and a "random comparison image" (serving as a baseline for unrelated embeddings).

• Metrics and Analysis:

  1. L2-Norm Distance: The Euclidean distance between the original and transformed latent vectors was calculated. These distances were normalized against the average distance between random, unrelated embeddings (Average Random Distance, ARD) to allow cross-network comparison.
  2. Dimensional Distribution: The study analyzed which specific dimensions in the latent space contributed most to the distance changes, sorting dimensions from most to least affected.
  3. Generalized Jaccard Index (GJI): To quantify the overlap of the most affected dimensions across different transformations, the authors developed a GJI metric. This measures the similarity of the sets of top-changing dimensions, assessing the degree of feature disentanglement (i.e., whether different transformations affect the same or distinct dimensions).

Key Results

• Robustness vs. Invariance: Embeddings of original and transformed images were consistently closer to each other than to random embeddings, indicating a degree of robustness. However, the distances were not zero, proving that the networks are not fully invariant to transformations. This supports the utility of transformation-mediated data augmentation, as the networks do not completely neutralize transformation effects.
• Transformation Impact: Color-based transformations (e.g., color jitter, grayscale) induced larger shifts in the latent space compared to spatial transformations (e.g., flipping, cropping). This suggests that color variations, which often stem from scanner differences, significantly impact the latent representation.
• Network Comparisons:
  • General vs. Pathology-Specific: Networks trained specifically on histopathology images (Lunit and Bioptimus models) generally exhibited greater invariance to spatial transformations (like vertical flips) compared to the Meta baseline trained on ImageNet. The Meta network showed larger shifts for vertical flips, likely because vertical orientation carries semantic meaning in natural images but not in histopathology.
  • Disentanglement: The H-Optimus-0 (H0) network demonstrated the best feature disentanglement, evidenced by the lowest Generalized Jaccard Index (0.091). This indicates that different transformations affected distinct sets of dimensions in the H0 network. Conversely, the Lunit SwAV and Meta SwAV networks showed higher overlap (GJI > 0.39), suggesting less effective disentanglement.
• Dimensional Behavior: No single dimension was entirely invariant. While a few dimensions contributed most to the change for any given transformation, the shifts were distributed across the latent space rather than localized to specific features.

Significance and Claims
The paper claims that measuring invariance provides a direct, interpretable method for assessing the quality of latent spaces and comparing different embedder networks without relying solely on downstream task performance.

• Practical Implications: The findings underscore the continued importance of classical image augmentations in preprocessing pipelines, particularly color normalization, to mitigate the significant impact of scanner-induced color variations on latent embeddings.
• Theoretical Insight: The study reveals that current state-of-the-art unsupervised networks do not achieve perfect disentanglement or invariance. The observation that different transformations often affect the same latent dimensions suggests that the networks do not fully separate underlying image features.
• Future Outlook: The authors posit that while these deviations from ideal invariance are expected given finite training resources, it remains an open question whether high sensitivity to transformations negatively impacts downstream classification performance. The proposed methodology offers a tool for systematic comparison of latent space structures, aiding the pursuit of better-structured latent spaces in representation learning.