Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study

Technical Summary: Vision Transformers for Zero-Shot Clustering of Animal Images

Problem Statement
Biodiversity monitoring has reached a critical bottleneck where the volume of visual data generated by camera traps and citizen science platforms exceeds the capacity for manual analysis. Traditional supervised deep learning approaches, such as Convolutional Neural Networks (CNNs), require extensive manual labeling of thousands of images to train species-specific classifiers. These models are inherently limited to fixed sets of species present in training data, struggle with the long-tailed distribution of biodiversity (where rare species lack sufficient training examples), and often fail to generalize to new sites or imaging conditions. While Vision Transformer (ViT) foundation models have demonstrated zero-shot classification capabilities, a gap remains between their classification accuracy and practical deployment in autonomous systems, where extensive manual validation is still required for a significant portion of images. This study investigates whether state-of-the-art ViT foundation models can reduce thousands of unlabeled animal images directly into coherent, species-level clusters without prior species-specific training or manual annotation.

Methodology
The authors developed a comprehensive benchmarking framework and a modular zero-shot clustering pipeline to evaluate the feasibility of organizing unlabeled animal imagery.

• Dataset Preparation: The study utilized 281,127 raw camera trap images from 60 species (30 mammals and 30 birds) sourced from 23 projects within the LILA BC repository. An automated pipeline using the MegaDetector for detection and SpeciesNet for classification was employed to crop animal regions. These crops underwent expert validation, resulting in a final dataset of 139,111 manually validated species-level images.
• Zero-Shot Clustering Pipeline: The pipeline consists of three modular stages:
  1. Feature Extraction: Input images are passed through pre-trained Vision Transformer models to generate high-dimensional embedding vectors.
  2. Dimensionality Reduction: Embeddings are projected to lower-dimensional spaces (specifically 2D) to facilitate clustering and visualization.
  3. Clustering: The reduced embeddings are partitioned using clustering algorithms.
• Experimental Design: The study systematically evaluated 27,600 configurations combining:
  • 5 ViT Models: CLIP, SigLIP, DINOv2, DINOv3, and BioCLIP 2.
  • 5 Dimensionality Reduction Techniques: t-SNE, UMAP, PCA, Isomap, and Kernel PCA (plus direct clustering on raw embeddings).
  • 4 Clustering Algorithms: Two unsupervised (DBSCAN, HDBSCAN) and two supervised (Hierarchical Clustering, Gaussian Mixture Models) methods.
• Evaluation: Performance was measured using V-measure (harmonic mean of homogeneity and completeness) and Adjusted Mutual Information (AMI). The study also analyzed cluster compositions to distinguish between model failures and biologically meaningful patterns (e.g., sexual dimorphism, age classes).

Key Contributions

1. Comprehensive Benchmarking Framework: A systematic evaluation of 27,600 method combinations across 60 species, providing quantitative metrics to guide method selection.
2. Validated Animal Dataset: The release of an open-source dataset comprising 139,111 manually validated image crops, processed through a pipeline mirroring real-world workflows.
3. Ecological Interpretation Framework: An analysis distinguishing between "failures" that represent genuine ecological subcategories (intra-specific variation) versus pure model limitations.
4. Open-Source Toolkit: Release of code, datasets, experimental logs, and an interactive web interface for visualizing clustering results.
5. Practical Deployment Guidelines: Evidence-based recommendations for parameter selection tailored to dataset characteristics, including handling long-tailed species distributions.

Key Results

• Model Performance: Self-supervised models significantly outperformed vision-language and biology-specific models. DINOv3 achieved the highest average V-measure (0.817 across all configurations), followed by DINOv2 (0.769). BioCLIP 2, CLIP, and SigLIP showed substantially lower performance (0.597–0.652).
• Dimensionality Reduction: Non-linear methods t-SNE and UMAP consistently outperformed linear and other non-linear methods (PCA, Isomap, Kernel PCA) by 26–38 percentage points. t-SNE marginally outperformed UMAP on average.
• Clustering Algorithms:
  • Supervised: When the cluster count matched the ground truth (K=30), Hierarchical clustering and GMM achieved near-perfect species-level clustering (V-measure: 0.958) when combined with DINOv3 and t-SNE.
  • Unsupervised: HDBSCAN dramatically outperformed DBSCAN. HDBSCAN discovered cluster counts close to the ground truth (33–37 clusters vs. 30 species) with a low outlier rate (1.14% of images). In contrast, DBSCAN suffered from severe over-clustering (200+ clusters) and high outlier rejection rates (28–29%).
• Robustness to Imbalance: Optimized HDBSCAN configurations (e.g., min_cluster_size=150) maintained high performance (V-measure > 0.92) even under extreme long-tailed distributions, whereas standard configurations failed.
• Intra-Specific Variation: "Over-clustering" (using higher K values or unsupervised methods with lower density thresholds) reliably extracted ecologically meaningful intra-specific patterns, including age classes, sexual dimorphism, pelage differences, and environmental contexts (e.g., snow backgrounds, IR vs. RGB lighting).
• Failure Analysis: Merged clusters often reflected genuine biological similarity (e.g., Wolf and Jackal, Takahē and Swamphen) rather than model errors. Oversplit clusters typically corresponded to high intra-specific variation (e.g., seasonal coat changes in Least Weasels).

Significance and Claims
The paper claims that zero-shot clustering using foundation models offers a practical pathway to reduce the manual annotation burden in ecological research. The study demonstrates that with the correct combination of models (DINOv3), dimensionality reduction (t-SNE), and clustering algorithms (HDBSCAN), it is possible to achieve near-perfect species-level clustering without any labeled training data.

The authors emphasize that this approach transforms the annotation task from exhaustive image-by-image labeling to a targeted review of cluster compositions. They argue that "failures" in clustering are not merely errors but often reveal biologically significant structures, such as phenotypic variation or environmental context, which can be valuable for demographic analysis. The study concludes that these methods provide essential infrastructure for timely conservation decision-making as camera trap networks expand globally, enabling ecologists to sort specific taxonomic groups and prioritize expert attention on ambiguous edge cases. The authors remain modest regarding the generalization to other taxonomic groups (e.g., insects, fish) and note that 2D reduction, while effective for visualization and clustering, may discard information preserved in higher-dimensional spaces.