Self-Supervised Learning with a Multi-Task Latent Space Objective

Technical Summary: Self-Supervised Learning with a Multi-Task Latent Space Objective

Problem Statement
Self-supervised learning (SSL) based on Siamese networks, such as BYOL, SimSiam, and MoCo v3, learns visual representations by aligning different augmented views of the same image. A critical component in modern SSL is the multi-crop strategy, which incorporates small local crops alongside global views to enforce spatial consistency. While this strategy significantly boosts performance in contrastive and clustering-based methods (e.g., SimCLR, SwAV, DINO), it induces severe instability in predictor-based Siamese architectures.

The authors identify that this instability stems from the use of a single shared predictor across all view types (global and local). Because global and local crops differ drastically in scale and content, a unified predictor struggles to align these heterogeneous representations, leading to optimization failure, premature performance plateaus, or accuracy degradation compared to standard two-view baselines.

Methodology
The proposed solution involves two primary architectural and methodological shifts:

1. View-Specific Predictors (Multi-Predictor Multi-Crop):
   Instead of sharing a single predictor head across all views, the authors assign a dedicated predictor to each view type (global, local, and later, cutout) while keeping the encoder (backbone and projection head) shared.

   • Mechanism: Each predictor specializes in aligning representations for its specific view type. This decoupling reduces interference between the alignment objectives of global and local crops.
   • Benefit: This modification stabilizes training without altering the loss function, requiring no new hyperparameters, and only marginally increasing the number of learnable parameters.

2. Multi-Task Formulation with Cutout Views:
   Building on the stabilized multi-crop setup, the authors extend the framework by introducing cutout views.

   • Asymmetric Cutout: A random rectangular region of the image is masked (occluded) in the online view, while the target view remains complete. This creates an asymmetric setup where the online model must predict the full image representation from a partially masked input, effectively acting as a lightweight inpainting task in latent space.
   • Complementary Signal: Unlike local crops which remove external context, cutout removes internal content. This encourages the model to infer missing information from surrounding regions, complementing the scale invariance learned by cropping.
   • Unified Framework: The final formulation treats global, local, and cutout views as distinct latent space tasks. The total alignment loss is a weighted sum of the prediction errors for each view type against the global target representation.

Key Contributions

1. Diagnosis of Instability: The paper identifies the shared predictor as the primary cause of multi-crop failure in asymmetric Siamese SSL and demonstrates that assigning view-specific predictors resolves this issue.
2. Stabilized Multi-Crop Training: The proposed multi-predictor approach enables predictor-based methods (BYOL, SimSiam, MoCo v3) to fully leverage multi-crop augmentations, yielding consistent performance gains.
3. Spatial Augmentation Extension: The introduction of asymmetric cutout views provides a complementary spatial supervision signal. The authors' analysis suggests that spatial augmentations (cropping and cutout) are the most effective source of supervision in SSL, outperforming non-spatial augmentations.
4. Unified Multi-Task Framework: The work presents a simple, backbone-agnostic multi-task formulation that integrates global, local, and masked views into a single training recipe compatible with both CNN and Transformer backbones.

Experimental Results
The approach was validated on ImageNet-1k using ResNet-50 and Vision Transformer (ViT) backbones across BYOL, SimSiam, and MoCo v3.

• ResNet-50 Performance:

  • BYOL: Linear evaluation accuracy improved from 70.7% (standard multi-crop with shared predictor) to 74.7% with the multi-predictor setup. The final multi-task formulation (including cutout) reached 75.6%.
  • SimSiam & MoCo v3: Both frameworks saw similar gains of approximately +4% in linear evaluation when moving from standard multi-crop to the multi-predictor setup.
  • Efficiency: The multi-task SimSiam variant achieved 74.7% accuracy in just 200 epochs, surpassing the original SimSiam baseline (71.3%) which required 800 epochs.

• ViT Performance:

  • ViT-S: Linear accuracy improved from 71.4% to 74.0%.
  • ViT-B: Linear accuracy reached 77.7%, outperforming the standard BYOL baseline by +3.7%.

• Downstream Tasks:

  • COCO Object Detection/Segmentation: The multi-task BYOL model outperformed both supervised pre-training and other SSL methods, achieving the highest AP (41.8 for detection, 38.0 for segmentation) on a ResNet-50 backbone.
  • Comparison: The method consistently outperformed or matched state-of-the-art contrastive, clustering, and non-contrastive methods, and outperformed Masked Image Modeling (MIM) approaches in linear evaluation settings.

Significance and Claims
The paper claims that decoupling predictors across view types is a fundamental requirement for making multi-crop strategies effective in predictor-based Siamese SSL. By treating each spatial transformation as a distinct alignment task, the authors provide a stable, generalizable framework that improves representation quality across different architectures.

The work highlights that spatial augmentations are the dominant source of learning signals in SSL. The integration of cutout views demonstrates that occlusion-based supervision can be effectively incorporated into Siamese frameworks without complex architectural changes or reconstruction objectives. The authors suggest that these principles—specifically the decoupling of predictors and the use of diverse spatial views—could extend to other structured modalities like videos or point clouds. They also note that conditioning predictors on view-specific information (e.g., cutout coordinates) represents a potential future direction toward Joint-Embedding Predictive Architectures.