Technical Summary: Harnessing Self-Supervised Features for Art Classification

Problem Statement

Classifying artworks, particularly paintings, presents a significant challenge due to the complex interplay of fine-grained details and abstract features that define style and genre. Unlike standard object classification where labels correspond to explicit visual objects, artistic style and genre rely on subjective, subtle, and often long-tailed distributions. These characteristics limit the effectiveness of fully supervised approaches, which are prone to overfitting and struggle with generalization. Furthermore, the subjective nature of art labels makes the creation of large, perfectly annotated datasets difficult and expensive.

Methodology

The authors conduct a systematic investigation into the effectiveness of self-supervised learning (SSL) backbones as feature extractors for artwork classification and retrieval, specifically focusing on the WikiArt dataset (approx. 80,000 paintings, 27 style classes, 11 genre classes).

Feature Extractors

The study compares a supervised baseline against two families of self-supervised models:

1. Supervised Baseline: EfficientNetV2-L, pre-trained on ImageNet and fine-tuned end-to-end for the specific style and genre tasks.
2. Self-Supervised Models:
   • DINO (DINOv3-ViT-L/16): Utilizes a student-teacher self-distillation paradigm.
   • CLIP (CLIP-ViT-L/14): Leverages contrastive learning between visual and textual modalities.
   • Note: For self-supervised models, the visual backbone is kept frozen, and the classification token (CLS) is used as the global feature representation.

Classification Strategies

The authors evaluate three distinct strategies for utilizing the extracted features:

1. Zero-Shot Classification: Uses both the visual encoder and a text encoder. Text prompts (e.g., "A painting in the {class} style") are encoded, and the class with the highest cosine similarity to the image embedding is selected.
2. Zero-Shot KNN Classification: A training-free approach where a query image is classified based on the label of the most similar image in a reference set (the training set), determined by cosine similarity of visual embeddings.
3. Linear Classification: A trainable linear layer is attached to the frozen visual backbone. The backbone remains fixed while the linear head is trained on the target task.

Experimental Setup

Experiments were conducted on the WikiArt dataset split into 80% training, 10% validation, and 10% testing. The supervised baseline and linear classifiers were trained using Adam optimization with specific learning rates and early stopping. Retrieval performance was qualitatively assessed using FAISS for indexing and cosine similarity for ranking.

Key Results

The study reports precision, recall, F1-score, and top-1 accuracy (acc@1) for both style and genre classification:

• Superiority of Self-Supervised Features: Employing self-supervised backbones led to consistent improvements over the supervised baseline in specific configurations.
• Performance of Linear Classification: The CLIP-Linear configuration achieved the best overall performance, outperforming the fully fine-tuned EfficientNetV2 baseline.
  • Style: CLIP-Linear (69.2% acc@1) vs. EfficientNetV2 (68.6% acc@1).
  • Genre: CLIP-Linear (84.9% acc@1) vs. EfficientNetV2 (82.2% acc@1).
• Effectiveness of KNN: The CLIP-KNN approach (zero-shot, no training) was remarkably competitive, outperforming the supervised baseline in style classification (70.6% vs. 68.6%) and achieving comparable results in genre classification (81.7% vs. 82.2%).
• Limitations of Zero-Shot Text Prompts: Pure zero-shot classification (relying on text prompts) performed poorly for both DINO and CLIP, suggesting that style and genre recognition in this context are driven primarily by visual alignment rather than textual alignment.
• Model Comparison: Across all protocols, CLIP features generally outperformed DINO features, indicating that CLIP's multi-modal pre-training captures the nuanced visual features required for art classification more effectively than DINO's purely visual self-distillation.

Significance and Claims

The paper claims that decoupling feature extraction from the classification stage is critical for artistic style and genre recognition. The primary contributions and implications are:

1. Decoupled Features: The work demonstrates that pre-trained, frozen self-supervised features are highly transferable and often superior to features learned via full fine-tuning on the specific task.
2. Training-Free Viability: The success of the KNN strategy highlights that high-quality semantic representations can be leveraged for art classification without any task-specific training, reducing computational costs.
3. Real-World Applicability: The authors posit that these components are directly applicable to real-world cultural heritage scenarios, specifically:
   • Immersive Experiences: Integration into Augmented Reality (AR) and Virtual Reality (VR) museum navigation systems to support context-aware navigation and personalized recommendations.
   • Curatorial Tools: Assisting professionals in categorizing previously unlabeled paintings by style and genre.
   • Scalable Archiving: Enabling the organization of large-scale digital archives where manual labeling is infeasible.
   • Accessibility and Education: Providing visually enriched navigation systems for diverse audiences and allowing students to explore collections by style or genre.

The authors conclude that multi-modal pre-training (as seen in CLIP) captures more abstract artistic aspects than standard visual supervision, making it a robust foundation for future art classification and retrieval systems.