Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

1. Problem Definition: Generalized Category Discovery (GCD)

Generalized Category Discovery (GCD) is an open-set recognition task where the goal is to identify both known (partially labeled) and unknown (unlabeled) categories within a dataset.

• Challenge: Existing State-of-the-Art (SOTA) methods often rely on multi-modal representation learning (combining vision and text) but focus heavily on inter-modality alignment (aligning image and text embeddings) while neglecting intra-modality alignment (ensuring consistency within the image modality or within the text modality).
• The Gap: Current approaches often suffer from imbalanced compression. When using contrastive losses, embeddings for known categories are overly compressed toward specific prototypes, while unknown categories remain poorly structured. This imbalance degrades clustering performance, especially for visually similar or fine-grained categories.

2. Methodology: SSR2-GCD

The authors propose SSR2-GCD (Semi-Supervised Rate Reduction for GCD), a framework designed to learn structured, balanced cross-modality representations. The framework consists of three core modules:

A. Retrieval-based Text Aggregation (RTA)

To enhance knowledge transfer, the model generates rich textual prompts for images.

• Mechanism: Instead of using a single prompt, the RTA strategy retrieves the top-$c$ most similar tags and attributes for a query image.
• Aggregation: It aggregates embeddings from these multiple candidates using a weighted sum. The most similar tag/attribute receives a higher weight ($1-\alpha$), while the remaining $c-1$ candidates share the weight $\alpha$.
• Benefit: This overcomes the limitation of CLIP's text encoder (which struggles with long prompts) by creating a semantic-rich textual representation that captures diverse cues for unknown categories.

B. Semi-Supervised Rate Reduction (SSR2) Module

This is the core innovation for representation learning. It replaces standard contrastive losses with a Maximal Coding Rate Reduction objective.

• Objective Function ($\mathcal{L}_{SSR^2}$):
  $$\mathcal{L}_{SSR^2} = -R(\mathbf{Z}) + R_c^s(\mathbf{Z}, \mathbf{Y}^*) + R_c^u(\mathbf{Z}, \mathbf{Y})$$
  • $-R(\mathbf{Z})$: Maximizes the coding rate of the entire embedding space (expands the representation to cover the data manifold).
  • $R_c^s$: Minimizes the coding rate for labeled data (known categories) to compress them into low-dimensional subspaces.
  • $R_c^u$: Minimizes the coding rate for unlabeled data (unknown categories) based on pseudo-labels.
• Key Advantage: Unlike contrastive learning, which pulls positive pairs and pushes negative pairs indiscriminately, SSR2 encourages balanced compression. It ensures that both known and unknown categories are compressed into low-dimensional subspaces with balanced matrix ranks, preventing the "collapse" of known categories and the scattering of unknown ones.
• Dual-Branch Training: The framework employs dual-branch classifiers (one for image, one for text) trained concurrently. The SSR2 loss is applied separately to image and text embeddings to enforce intra-modal consistency.

C. Training Strategy

The training proceeds in two stages:

1. Warm-up Stage: The unsupervised term ($R_c^u$) is excluded to prevent noisy pseudo-labels from corrupting early learning. Only supervised rate reduction and classification losses are used.
2. Alignment Stage: The full SSR2 loss is activated. A Co-teaching strategy is used to align the predictions of the dual-branch classifiers, ensuring that the image and text branches agree on the clustering structure.

3. Key Contributions

1. SSR2 Framework: The first work to incorporate the Maximal Coding Rate Reduction principle into multi-modal GCD. It explicitly addresses the imbalanced compression issue found in contrastive-based methods.
2. Intra-Modality Focus: The paper demonstrates that prioritizing intra-modality alignment (via SSR2) is more critical than pure inter-modality alignment for GCD. It shows that forcing inter-modal alignment without intra-modal constraints leads to misalignment within modalities.
3. RTA Strategy: A novel text aggregation method that leverages multiple prompt candidates to generate robust semantic representations for unknown categories.
4. Dual-Branch Architecture: A concurrent training mechanism that utilizes self-supervised signals from both modalities to guide representation learning.

4. Experimental Results

The authors evaluated SSR2-GCD on 8 benchmark datasets (4 generic: CIFAR-10/100, ImageNet-100/1k; 4 fine-grained: CUB, Stanford Cars, Oxford Pets, Flowers102).

• Performance: SSR2-GCD achieved State-of-the-Art (SOTA) performance across all datasets.
  • On Flowers102, it achieved 93.5% accuracy on "All" categories, outperforming the previous best (GET) by 6.3%.
  • On Stanford Cars, it achieved 89.2% accuracy, outperforming baselines by 3.1%.
• Ablation Studies:
  • Intra vs. Inter: Using only inter-modal loss (CLIP-style) yielded limited gains. Combining inter-modal loss with intra-modal contrastive loss ($L_{con}$) often degraded performance. However, SSR2 alone (intra-modal) consistently outperformed all combinations, proving that intra-modal consistency is the bottleneck.
  • Balanced Compression: Analysis of singular value ranks showed that standard contrastive loss ($L_{con}$) drastically reduced the rank of known categories (over-compression) while leaving unknown categories with high ranks. SSR2 maintained balanced ranks for both known and unknown categories, preventing collapse.
  • Consistency: Metrics on intra-modal consistency ($\rho$) showed SSR2 achieved significantly higher consistency than CLIP-based losses, validating the hypothesis that intra-modal alignment drives clustering success.

5. Significance

This paper fundamentally shifts the paradigm in Multi-Modal GCD:

• Theoretical Insight: It challenges the prevailing assumption that inter-modal alignment is the primary driver of performance. Instead, it proves that intra-modality alignment is the prerequisite for effective clustering in open-set scenarios.
• Practical Impact: By solving the imbalanced compression problem, SSR2-GCD enables more robust discovery of unknown categories, particularly in fine-grained domains where visual similarities are high.
• Generalizability: The use of Rate Reduction provides a mathematically grounded approach to representation learning that is less prone to the collapse issues common in contrastive learning, offering a new direction for semi-supervised and open-set learning tasks.

In summary, SSR2-GCD achieves superior category discovery by balancing the compression of known and unknown classes through a novel rate reduction loss, while leveraging rich textual cues via a retrieval-based aggregation strategy.