Detecting Cross-Medium Stylistic Signals in Sketches and Paintings with Pretrained Visual Encoders

Technical Summary: Detecting Cross-Medium Stylistic Signals in Sketches and Paintings with Pretrained Visual Encoders

Problem Statement
This study addresses the challenge of cross-medium visual media understanding within computational art history. Specifically, it investigates whether aspects of an artist's visual style persist across distinct media types—namely, sketches (drawings) and finished paintings—and whether computational models can detect this persistence. The research moves beyond simple artist attribution (naming a maker) to probe the relationship between artistic identity, medium, and visual form. The core difficulty lies in the significant visual and material gap between sketches (often rapid, exploratory, and linear) and paintings (involving different materials, temporal rhythms, and surface effects). The task is formulated as a relational verification problem: given a sketch and a painting, can a model determine if they share the same author, even when the model has not seen that specific artist during training?

Methodology
The authors constructed a controlled corpus of 666 digitized artworks by ten historical artists spanning five broad historical groupings (Renaissance, Baroque, Neoclassical, Impressionist, and Pre-Raphaelite). The dataset comprises 333 sketches and 333 paintings.

The experimental design employs a leave-one-artist-out (LOAO) protocol. In each fold, a verifier is trained on nine artists and tested on the held-out tenth artist. This design ensures that performance reflects generalization to unseen identities rather than memorization of the training set.

The study evaluates six feature families:

1. Pretrained Visual Encoders (Frozen weights):
   • CLIP (ViT-B/32): A vision-language model embedding.
   • DINOv2 (ViT-S/14): A vision-only self-supervised model (no language supervision).
   • SigLIP (ViT-B/16): A vision-language model using a sigmoid contrastive objective.
2. Handcrafted Descriptor Families:
   • Composition and Form: 1398-dimensional features including edge energy, HOG, contour curvature, and symmetry.
   • Tonal Organisation: 69-dimensional features including luminance quantiles and shadow/highlight occupancy.
   • Surface Texture: 92-dimensional features including LBP, Gabor filters, and wavelet energies.

For each feature family, a pairwise verifier (logistic regression with balanced class weighting) is trained on a pair descriptor combining cosine similarity, Euclidean distance, elementwise absolute difference, and elementwise product.

Evaluation utilizes three negative sampling tiers to test robustness:

• Unmatched Negatives: Random pairings of sketches and paintings by different artists.
• Period-Matched Negatives: Pairings restricted to artists from the same historical period.
• Hard Negatives: Pairings where the negative painting is the visually most similar work by a different artist (nearest-neighbor search).

Key Results

• Pretrained Encoders: CLIP and SigLIP demonstrated the strongest cross-medium transfer. In the LOAO evaluation, they achieved mean Area Under the Curve (AUC) values of 0.766 and 0.759, respectively. DINOv2 performed more weakly but remained above chance with a mean AUC of 0.612.
• Handcrafted Descriptors: None of the handcrafted families (Composition, Tonal, Texture) achieved performance significantly above chance in the LOAO setting (AUCs ranging from 0.508 to 0.537).
• Robustness and Limits: While pretrained models succeeded under unmatched and period-matched negatives, performance collapsed under hard negative conditions. CLIP dropped to an AUC of 0.469, DINOv2 to 0.372, and SigLIP to 0.518. This indicates the detected signal is statistical and population-level, not robust enough to distinguish an artist from visually similar impostors.
• Representational Similarity: Mantel matrix correlation tests revealed that CLIP, DINOv2, and SigLIP organize artists into similar relational geometries (correlations between 0.45 and 0.66), suggesting the signal is reproducible across different architectures and training objectives.
• Closed-Set Upper Bound: A secondary analysis using triplet metric learning on CLIP features achieved 100% retrieval accuracy within the fixed corpus, confirming that the features contain sufficient information for discrimination when all artists are known, but this does not translate to generalization.

Key Contributions

1. Systematic Empirical Study: The paper provides a controlled evaluation of cross-medium artist verification between sketches and paintings, a setting that is more demanding than single-medium attribution.
2. Generalization Evidence: It demonstrates that pretrained visual encoders (specifically CLIP and SigLIP) can detect stylistic signals that persist across media and generalize to artists not seen during training. The agreement between CLIP and SigLIP (despite different architectures and objectives) and the success of the vision-only DINOv2 suggests the signal is not an artifact of a single model or language supervision.
3. Identification of Methodological Limits: The study explicitly identifies a clear boundary where the method fails: under hard negative conditions (visually similar different-artist pairs), the signal becomes fragile. This prevents the method from being interpreted as reliable evidence for authentication.

Significance and Claims
The authors argue that cross-medium verification should not be viewed as a standalone tool for assigning authorship or authenticating artworks. Instead, its value lies in examining how artificial intelligence models visual style.

The findings suggest that:

• Pretrained encoders capture aspects of artistic similarity (potentially involving composition, spatial organization, or handling of form) that are not recoverable by standard handcrafted descriptors of composition, tone, or texture.
• These signals are unstable under difficult visual comparisons, highlighting the gap between current computational representations and the nuanced requirements of art-historical authentication.
• The study contributes to debates on the interpretability, reliability, and evaluation of visual representations for complex cultural media, emphasizing that while computational systems can identify regularities, their outputs must be interpreted with caution regarding their evidential limits.

The paper concludes that while pretrained encoders encode persistent cross-medium signals, these encodings remain opaque and dependent on corpus design. Future work requires more diverse datasets and interpretable modeling before such methods can play a stronger role in provenance research.