Technical Summary: Multiview Self-Representation Learning across Heterogeneous Views

Problem Statement

The paper addresses the challenge of learning invariant representations from large-scale unlabeled visual data using a fully unsupervised transfer learning approach. A critical issue arises when features for the same sample are generated by different pretrained models (e.g., DINOv2, CLIP). Due to discrepancies in pretraining objectives and architectures, these features exhibit inherently distinct and heterogeneous distributions. Traditional contrastive learning methods, which typically assume augmented views share a common underlying distribution, are not directly suitable for these heterogeneous views. Furthermore, existing unsupervised transfer methods often struggle to effectively capture and exploit complementary semantic information preserved across multiple distinct pretrained models.

Methodology: Multiview Self-Representation Learning (MSRL)

The proposed MSRL framework learns invariant representations by stacking individual linear models on top of frozen pretrained backbones. The architecture consists of three primary modules:

1. Transfer Learning Module: Utilizes multiple pretrained models ($\phi_l$) to extract features from raw images, resulting in heterogeneous multiview data.
2. Feature Self-Representation Learning (FSRL) Module:
   • Linear Projection: A trainable linear model ($\sigma$) is applied to the frozen backbone outputs to generate low-dimensional features.
   • Information-Passing Mechanism: To aggregate features, the method employs a self-representation property where each feature is represented as a linear combination of its spatially adjacent neighbors. An attention mechanism is used to adaptively select neighborhoods and compute coefficients ($a_{ij}$), emphasizing the most linearly related features within the same category. This allows the model to capture the geometric structure of the features without requiring shared distributions.
3. Assignment Probability Distribution Consistency (APDC) Module:
   • Consensus Building: Since direct feature comparison is difficult due to heterogeneity, the method aligns the assignment probability distributions (softmax outputs) produced by the linear models.
   • Latent Semantic Assignment: A consensus distribution ($\rho_\theta$) is calculated as the average of the assignment probabilities from all $L$ views.
   • Loss Functions: The optimization is driven by three complementary losses:
     • Semantic Pseudolabel Loss ($L_s$): Minimizes the cross-entropy between the predicted cluster assignment (derived from the consensus) and the individual view's assignment.
     • Cluster Diversity Loss ($L_a$): A regularization term to prevent all samples from collapsing into a single cluster.
     • Cross-View Consistency Loss ($L_c$): Minimizes the discrepancy (cross-entropy) between the assignment probability distributions of different views, enforcing them to converge toward the consensus.

The overall objective function is $L = L_s + \alpha L_a + \beta L_c$, optimized via adaptive momentum-based mini-batch gradient descent while keeping the pretrained backbones frozen.

Key Contributions

The paper outlines three primary contributions:

1. MSRL Model: Introduction of a method to learn invariant representations from large-scale visual data across heterogeneous views in a fully unsupervised transfer manner.
2. Information-Passing Mechanism: Development of a mechanism that adaptively aggregates feature information. It selects the most linearly related features within the same category to yield low-dimensional vector representations, effectively handling the geometric structure of heterogeneous features.
3. Assignment Probability Distribution Consistency Scheme: A novel scheme for multiview self-representation learning that exploits complementary information across views. By enforcing consistency in assignment probabilities rather than raw features, it effectively learns invariant representations.
4. Theoretical Analysis: The paper provides theoretical justifications for the information-passing mechanism (via weak neighborhood alignment), the bounded nature of multiview consistency, and the convergence properties of incremental views (demonstrating that adding views stabilizes the consensus distribution and reduces entropy).

Experimental Results

Extensive experiments were conducted on eight benchmark visual datasets (Pets, GTSRB, DTD, Aircraft, Flowers, CIFAR-10, CIFAR-100, and ImageNet-100) using DINOv2 and CLIP ViT-L/14 as backbones.

• Performance: MSRL consistently outperformed state-of-the-art approaches, including SCMVC, CMVC, SparseMVC, HSACC, and TURTLE, across all datasets. For instance, on CIFAR-100 and ImageNet, MSRL achieved clustering accuracies (ACC) of 91.00% and 73.24%, respectively, surpassing the second-best methods by significant margins (at least 1.9% and 3.14%).
• Zero-Shot Comparison: In zero-shot transfer learning tasks, MSRL outperformed CLIP zero-shot transfer and LaFTer on most datasets, demonstrating robustness even without explicit class-level semantic supervision.
• Efficiency: MSRL demonstrated faster training times compared to TURTLE and significantly outperformed SparseMVC, attributed to the efficiency of unsupervised transfer learning on frozen backbones.
• Ablation Studies:
  • Removing the cross-view consistency loss ($L_c$) resulted in performance degradation, confirming its necessity for enforcing invariance.
  • Sensitivity analysis showed stable performance for the hyperparameter $\alpha$ within the range of 1.0 to 10.0.
  • Incremental Views: Experiments adding more pretrained models showed that while adding views generally improves performance, the quality of the added view matters. Including lower-quality models (e.g., ViT-B/32) could negatively impact performance, indicating that not every additional view is beneficial.

Significance and Claims

The paper claims that the MSRL method successfully bridges the gap in learning invariant representations from heterogeneous views without requiring manual annotations or fine-tuning of the entire backbone. Its significance lies in:

• Handling Heterogeneity: Unlike traditional multiview methods that assume homogeneous feature spaces, MSRL explicitly addresses the distributional discrepancies between different pretrained models by operating on assignment probabilities.
• Complementary Information: The framework effectively exploits complementary semantic knowledge across different views, which is crucial for large-scale visual data.
• Theoretical Grounding: The work provides a theoretical foundation for why the consensus distribution acts as a stable attractor and how entropy is reduced as more views are incrementally added, offering a principled approach to multiview self-representation.
• State-of-the-Art Performance: The method establishes new benchmarks for unsupervised transfer learning on standard visual datasets, outperforming existing contrastive and multiview clustering approaches.