Free-Grained Hierarchical Visual Recognition

1. Problem Definition

The paper addresses a critical gap in Hierarchical Image Recognition. Traditional methods assume "tidy" supervision where every training image is annotated with a complete path through a semantic taxonomy (e.g., Bird → Bird of prey → Bald eagle). However, real-world annotations are often "messy" and mixed-granularity:

• Intrinsic Ambiguity: Distant or low-quality images may only support a coarse label (e.g., Bird), while clear close-ups support fine-grained labels (e.g., Bald eagle).
• Extrinsic Constraints: Annotation costs, varying expert levels, or evolving protocols often result in incomplete labels.

The Challenge: Existing hierarchical models deteriorate sharply when trained on mixed-granularity data because they rely on complete supervision paths. The authors propose Free-Grained Hierarchical Recognition, a setting where:

1. Training: Labels can appear at any level of the taxonomy (coarse, intermediate, or fine) and vary across instances.
2. Inference: The model must adaptively choose the deepest reliable prediction depth, returning a coarse label if a fine-grained one is uncertain, while maintaining taxonomy consistency.

2. Methodology

The authors propose a comprehensive framework comprising new benchmarks, two novel training methods, and a specific inference strategy.

A. Benchmark Construction

To evaluate this setting, the authors created new datasets by adapting existing hierarchies (CUB, Aircraft, iNat21) and restructuring ImageNet:

• ImageNet-3L: They reorganized the noisy, irregular WordNet hierarchy into a clean, consistent three-level taxonomy (Basic, Subordinate, Fine-grained) based on cognitive psychology principles (Rosch's basic level).
• Foundation-Based Variants (ImageNet-F, etc.): They used large Vision-Language Models (VLMs like CLIP and BioCLIP) to prune labels. If a VLM could not confidently predict a fine-grained label, the label was removed, simulating realistic visual ambiguity. This creates long-tailed distributions of label availability.
• Randomized Variants (CUB-Rand, Aircraft-Rand): Labels are randomly pruned at specific rates to stress-test model robustness under controlled sparsity.

B. Training Methods

To handle missing supervision, two complementary methods are proposed:

1. Text-Guided Pseudo-Attributes (Text-Attr)

• Concept: Visual attributes (e.g., "short legs," "pointed ears") are often consistent across hierarchical levels even when class labels differ.
• Mechanism: A frozen Vision-Language Model (Llama-3.2) generates a text description for each image. These descriptions are encoded and aligned with image features using a contrastive loss.
• Goal: This forces the image encoder to learn "pseudo-attributes" (semantic cues) that are robust to missing class labels, helping the model distinguish fine-grained classes even when only coarse labels are available.

2. Taxonomy-Guided Semi-Supervised Learning (Taxon-SSL)

• Concept: Treats missing labels at specific levels as unlabeled data rather than ignoring them.
• Mechanism:
  • Multi-level Pseudo-labeling: Generates pseudo-labels for all levels using weakly augmented views.
  • Taxonomy-Aligned Consistency: Enforces that two samples are considered "positive pairs" (similar) only if their pseudo-labels agree across all hierarchy levels. If they disagree at any level, they are treated as negative pairs.
  • Contrastive Loss: Pulls consistent samples together and pushes inconsistent ones apart in the feature space.
• Goal: Leverages the hierarchical structure to propagate information from labeled examples to unlabeled levels, reducing noise in pseudo-labels.

C. Free-Grained Inference

Instead of forcing a prediction at the deepest level (which may be wrong), the model adaptively stops at the appropriate depth.

• Confidence-Based: Stops if softmax probability drops below a threshold (often fails due to probability splitting among siblings).
• Consistency-Based (Proposed): Stops only when a deeper prediction violates the hierarchical constraints (e.g., predicting a specific dog breed that doesn't belong to the predicted "Dog" parent). This yields more reliable and deeper correct predictions.

3. Key Contributions

1. Task Formulation: Introduced Free-Grained Hierarchical Recognition, formalizing the problem of learning from mixed, incomplete supervision and adaptive inference.
2. Benchmarks: Created ImageNet-3L and foundation-based variants (ImageNet-F, iNat21-mini-F, CUB-F) that reflect realistic annotation difficulties and long-tailed label availability.
3. Methods: Proposed Text-Attr (semantic guidance via VLMs) and Taxon-SSL (hierarchical consistency via SSL), both of which significantly outperform existing SOTA hierarchical classifiers in this setting.
4. Inference Strategy: Demonstrated that consistency-based stopping is superior to confidence-based stopping for adaptive depth prediction.

4. Experimental Results

• Baseline Failure: State-of-the-art hierarchical methods (HRN, H-CAST) suffer massive performance drops (up to 40%) in Full-Path Accuracy (FPA) when moving from fully supervised to free-grained settings.
• Performance Gains:
  • On ImageNet-F, Text-Attr (H-CAST) achieved 63.2% FPA, significantly outperforming the baseline H-CAST (57.6%) and HRN (37.8%).
  • On iNat21-mini-F (biological domain), Taxon-SSL achieved the best performance (31.9% FPA), outperforming baselines by ~6-16%, highlighting its effectiveness in domains with high visual similarity and limited data.
  • Combination: Combining both methods (Taxon-SSL + Text-Attr) provided consistent, albeit modest, improvements across datasets.
• Label Sparsity Analysis:
  • Text-Attr excels when labeled data is scarce but semantic diversity is high (e.g., ImageNet).
  • Taxon-SSL excels under extreme label sparsity (e.g., CUB-Rand with only 3 fine labels per class) by leveraging unlabeled data structure.
• Inference: Consistency-based stopping allowed models to reach deeper correct levels more frequently than confidence-based methods, avoiding premature stopping or incorrect fine-grained guesses.

5. Significance

This work moves hierarchical recognition closer to real-world deployment scenarios where perfect annotations are impossible.

• Practicality: It acknowledges that "good enough" coarse predictions are often preferable to incorrect fine-grained ones, a crucial insight for safety-critical or expert-assisted applications.
• Robustness: The proposed methods demonstrate that models can learn robust hierarchical representations even with highly incomplete and uneven supervision.
• Foundation Models: It effectively leverages VLMs not just for zero-shot classification, but as a source of semantic supervision to train image encoders to capture shared attributes across hierarchy levels.
• Future Direction: The paper highlights that class imbalance and level imbalance are distinct challenges that must be addressed jointly, setting a new standard for future research in hierarchical and semi-supervised learning.