Style or Signature? Artist-Disjoint Evaluation of Style Classification in Frozen Vision Embeddings

Technical Summary: Artist-Disjoint Evaluation of Style Classification in Frozen Vision Embeddings

Problem Statement
Frozen image embeddings from large-scale pretrained vision models, particularly CLIP, are increasingly utilized to classify paintings by art-historical style with high reported accuracy. Standard evaluation protocols typically employ random splits of the dataset, where works by the same artist may appear in both the training/reference set and the test/query set. The paper argues that this methodology conflates the recognition of artistic style with the recognition of individual artist identity. Since frozen embeddings encode strong "neural signatures" of specific artists, a classifier may achieve high accuracy by identifying the painter rather than the movement. Consequently, standard aggregate accuracy metrics may overstate a model's genuine understanding of stylistic features, masking the extent to which performance relies on artist memorization.

Methodology
The author proposes and implements an artist-disjoint evaluation protocol (leave-one-artist-out) to decouple style classification from artist identity.

• Dataset: A balanced dataset of 320 paintings spanning four 20th-century movements: Impressionism, Cubism, Abstract Expressionism, and Surrealism. The dataset includes eight artists per movement, with ten works per artist.
• Protocol: Instead of random splitting, the evaluation holds out all works by a single artist as the query set while using the remaining 310 works (from the other 31 artists) as the reference pool. This ensures no work is classified using neighbors created by its own painter.
• Models: The primary experiments use CLIP ViT-B/32. To ensure generality, the pipeline is replicated across three additional encoders: CLIP ViT-L/14, the self-supervised vision-only DINOv2 ViT-B/14, and a supervised ImageNet ResNet-50.
• Metrics: The study compares 5-Nearest Neighbor (5-NN) style accuracy under full-pool (random split) versus artist-disjoint conditions. Additionally, an "artist-recognisability control" is introduced, where the model predicts the artist rather than the style (allowing same-artist neighbors) to quantify how strongly individual artists are encoded.

Key Results

1. Aggregate Performance Drop: Under the artist-disjoint protocol, overall 5-NN style accuracy drops from 0.87 to 0.77. While the aggregate drop appears modest, it conceals a sharply uneven effect across movements.
2. Movement-Specific Variance:
   • Impressionism and Cubism: These movements show high robustness, with accuracy drops of only 3.8 and 5.0 points, respectively. Their artists are difficult for the model to distinguish individually (low artist-recognisability), suggesting the embeddings capture a genuine, shared visual form that subsumes individual authorship.
   • Surrealism: This movement exhibits a dramatic collapse, with accuracy falling 20 points (from 0.713 to 0.513). The model misclassifies nearly half of the held-out works. The fragility is driven by specific artists (e.g., Miró and Masson) whose style scores were heavily dependent on their individual recognisability.
   • Abstract Expressionism: Shows an intermediate drop (12.5 points) but remains relatively robust (0.850), despite having the highest individual artist recognisability among the groups.
3. Encoder Generalization: The pattern holds across all four encoders, including DINOv2 (vision-only, no language). This confirms that the observed fragility in Surrealism and robustness in Impressionism/Cubism are properties of the visual structure captured by the embeddings, not artifacts of language-text alignment.

Key Contributions

1. Artist-Disjoint Protocol: The paper introduces a leave-one-artist-out evaluation standard for frozen vision embeddings, demonstrating that it exposes fragility in style classification that standard random-split evaluations conceal.
2. Quantification of the Gap: The author quantifies the discrepancy between standard and artist-disjoint accuracy, showing it is movement-dependent rather than uniform. They identify the mechanism via an artist-recognisability control, distinguishing between genuine shared form and artist memorization.
3. Visual vs. Linguistic Origin: By including a vision-only model, the paper rules out language-based explanations for the observed patterns, placing the effect firmly in the visual structure of the movements.

Significance and Claims
The paper argues that the question "Does the embedding capture style?" is ill-posed because style operates differently across movements. For movements like Impressionism and Cubism, style is defined by shared visual forms that persist even when individual artists are removed. For Surrealism, the "style" is often inextricably linked to the idiosyncratic visual languages of specific artists; thus, removing the artist removes the style signal.

The primary significance is methodological: standard random-split evaluations risk certifying "artist recognition" as "style understanding." The author advocates for artist-disjoint evaluation as a necessary default for measuring stylistic understanding in frozen embeddings. This approach transforms a brittle aggregate metric into an interpretable per-movement and per-artist picture, revealing which movements possess a recoverable shared visual form and which rely on the recognisability of specific painters. The paper does not claim that fine-tuning would resolve these issues, nor does it propose new applications, but rather calls for a more rigorous evaluation standard to prevent overestimating model capabilities in computational art history.