Technical Summary: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classification

1. Problem Statement

Multimodal contrastive learning models, such as CLIP, have demonstrated strong capabilities in zero-shot visual classification by aligning image and text embeddings. However, these models typically operate under a flat-label assumption, treating all non-matching text labels as equal negative examples. This assumption breaks down when applied to hierarchically structured label spaces (e.g., biological taxonomies).

In hierarchical settings, standard contrastive learning generates conflicting supervision signals. For instance, while a lion (Panthera leo) and a wolf (Canis lupus) are distinct at the species and genus levels, they share higher-level ancestors (e.g., the order Carnivora). In a standard flat-label contrastive setup, the text label for "wolf" might be treated as a negative target for an image of a "lion," even though they are semantically compatible at the order level. This creates false negatives across taxonomic levels, leading to hierarchical inconsistency where a model's predicted fine-grained label does not belong to its predicted higher-level category.

2. Methodology

The authors propose a Level-Restricted Contrastive Learning framework designed to eliminate cross-level false negatives and ensure balanced optimization across the taxonomy.

2.1. Problem Formulation

The taxonomy is defined as a set $\mathcal{T} = (C_1, \dots, C_L)$ with $L$ hierarchical levels (from coarse to fine). Each image $x_i$ is associated with a taxonomic label vector $t_i = (t_i^1, \dots, t_i^L)$.

• Text Construction: For each level $\ell$, a text label $y_i^\ell$ is constructed containing the taxonomy prefix up to that level: $y_i^\ell = (t_i^1, \dots, t_i^\ell)$.
• Encoding: Images and text labels are encoded using CLIP-based image ($f$) and text ($g$) encoders to produce embeddings $v_i$ and $z_i^\ell$, respectively.

2.2. Level-Restricted Contrastive Loss

Instead of performing a joint contrastive loss over all levels, the objective is decomposed into independent comparisons within each taxonomic level.

• Intra-Level Comparison: Contrastive loss is calculated separately for each level $\ell$. A species-level label is compared only against other species-level labels, avoiding the ambiguity of comparing species against orders or kingdoms.
• Group-Balanced Supervision: To handle mini-batches where multiple images may share the same coarse-level label (e.g., two different species from the same Kingdom), the method aggregates labels. All image-text pairs sharing the same label at a specific level are treated as positive pairs.
  • Image-to-Text Loss ($L_{I \to T}$): Computes similarity between an image embedding and the set of unique text embeddings at that level.
  • Text-to-Image Loss ($L_{T \to I}$): Computes similarity between a text anchor and the set of image embeddings sharing that label.
• Final Objective: The total loss is the average of the losses across all $L$ levels:
  $$\mathcal{L} = \frac{1}{L} \sum_{\ell=1}^{L} \mathcal{L}_\ell$$
  This equal weighting ensures that fine-grained levels receive sufficient supervision and are not dominated by high-confidence coarse-level signals.

2.3. Implementation Details

• Base Model: The framework fine-tunes BioCLIP (a foundation model pre-trained on the Tree of Life).
• Training Data: The model is trained on TreeOfLife-10M.
• Embedding Spaces: The method is instantiated in both Euclidean and Hyperbolic spaces, with the latter offering potential advantages for representing hierarchical structures.

3. Key Contributions

1. Identification of False Negatives: The paper identifies that treating ancestor–descendant labels as negatives in standard contrastive learning creates conflicting signals that violate the flat-label assumption.
2. Level-Restricted Framework: A novel training recipe that restricts contrastive comparisons to labels within the same taxonomic level, ensuring well-defined positive and negative pairs.
3. Group-Balanced Design: A mechanism to handle label collisions in mini-batches by treating all samples with the same label at a specific level as positives, preventing bias toward frequent coarse categories.
4. Empirical Validation: Extensive evaluation across multiple hierarchical benchmarks demonstrating improved consistency and accuracy.

4. Experimental Results

The model was evaluated on three benchmarks: iNaturalist 2021 (iNat21), Rare Species, and CrypticBio.

4.1. Accuracy Improvements

• iNat21: The proposed method improved the average accuracy across all taxonomic levels by 30.47% over the BioCLIP baseline.
• Comparison: The method outperformed OpenCLIP, BioCLIP, and RCME (a previous hierarchy-oriented method) across all datasets.
• Granularity: Significant gains were observed at intermediate levels (Order to Genus), where standard contrastive loss typically suffers most from cross-level ambiguity. Species-level performance also improved despite the optimization not being solely focused on that level.

4.2. Hierarchical Consistency

• Top-Down Constrained Inference: Under a strict protocol where predictions proceed from coarse to fine (restricting candidates to valid descendants), the method maintained superior performance.
• Normalized LCA (nLCA): The hyperbolic variant achieved an nLCA score of 0.71 compared to 0.43 for RCME, indicating that even when the exact species label is incorrect, the prediction remains closer to the correct taxonomic branch.
• Space Comparison: The Euclidean variant performed best on iNat21, while the Hyperbolic variant showed stronger performance on Rare Species and CrypticBio, particularly at deeper taxonomic levels.

4.3. Visualization

t-SNE visualizations of text embeddings revealed that the proposed method produces a clearer hierarchical structure with reduced cross-level mixing compared to BioCLIP. Sibling categories are more distinctly separated, and embeddings from nearby taxonomic levels form coherent groups.

5. Significance and Claims

The paper claims that the proposed Level-Restricted Contrastive Learning framework effectively resolves the issue of hierarchical inconsistency in multimodal models. By restricting comparisons to mutually exclusive label sets within the same taxonomic level and balancing supervision across levels, the method:

• Eliminates conflicting supervision signals caused by false negatives in hierarchical spaces.
• Simultaneously improves both flat classification accuracy and hierarchical consistency.
• Promotes the emergence of structured hierarchical relationships in the embedding space without requiring complex geometry-specific objectives or specialized architectures beyond the contrastive reformulation.

The authors emphasize that this approach is particularly effective for zero-shot classification in domains with complex taxonomies, such as biodiversity, where maintaining logical consistency across levels is critical for real-world applicability.