Do Transformers Understand Ancient Roman Coin Motifs Better than CNNs?

Technical Summary: Do Transformers Understand Ancient Roman Coin Motifs Better than CNNs?

Problem Statement
The automated analysis of ancient numismatics presents unique challenges distinct from mainstream computer vision tasks. While existing research has largely focused on image-to-image matching for coin identification, the sheer volume of ancient coin issues (e.g., over 43,000 Roman imperial issues) makes exhaustive matching impractical. Consequently, the research focus has shifted toward identifying semantic elements (e.g., specific motifs like eagles, shields, or figures) depicted on coins. However, this task is complicated by high intra-class variability due to die differences, wear, and degradation, as well as a lack of high-quality images for many issues. Previous attempts using Convolutional Neural Networks (CNNs) have shown promise but face limitations in capturing nuanced, long-range dependencies within the stylized visual elements of ancient coins. This paper investigates whether the Vision Transformer (ViT) architecture, which has surpassed CNNs in many visual tasks, offers superior performance in recognizing these semantic elements compared to newly trained CNN models.

Methodology
The study employed a multi-modal dataset comprising 100,000 images and unstructured text descriptions of ancient coins sourced from an online auction aggregator. The methodology proceeded through three primary stages:

1. Data Preprocessing and Labeling:

   • Image Extraction: An automated algorithm isolated the reverse side of individual coins from images that often contained multiple coins or both sides. The algorithm identified bounding boxes based on background color consistency and foreground object detection, rejecting batch lots or overlapping coins.
   • Weak Supervision: Labels were generated via text mining of coin descriptions. Search words (in English, French, Spanish, German, and Latin) and their derivatives were used to assign binary presence/absence labels for specific semantic elements. This resulted in "weak supervision," where labels applied to the whole image rather than specific pixel locations, introducing potential noise (e.g., descriptions referencing the obverse side being applied to the reverse image).
   • Concept Mining: A text-mining module identified additional concepts ('seated', 'standing', 'Hercules') beyond the five previously studied by Cooper and Arandjelovi´c.

2. Model Architecture and Training:

   • Vision Transformer (ViT): A 'large' ViT model, pre-trained on ImageNet-21K, was fine-tuned. The input images were split into 16x16 patches. The model utilized a single multi-label architecture trained simultaneously on all semantic elements. To address class imbalance, positive samples were weighted more heavily in the loss function. The final classification head was replaced and trained using Stochastic Gradient Descent (SGD) with momentum.
   • Convolutional Neural Network (CNN): A CNN architecture based on the topology proposed by Cooper and Arandjelovi´c was trained from scratch. Unlike the ViT, separate single-label models were trained for each semantic element. To handle class imbalance, negative samples were undersampled (and oversampled for the 'Hercules' class). The optimizer used was SGD with momentum, as the Adam optimizer previously used in similar work resulted in "dying ReLU" issues for this specific dataset and topology.

3. Evaluation:

   • Models were evaluated on held-out test sets using accuracy, precision, recall, and F1 scores.
   • Explainability: Saliency maps generated via the black-box tool HiPe were used to visualize which image regions influenced the models' predictions, providing qualitative insights into model behavior.

Key Contributions

• First Application of ViT to Numismatics: This paper represents the first application of the Vision Transformer architecture to the identification of semantic elements on ancient coins.
• Comparative Analysis: It provides a direct performance comparison between ViT and CNN architectures on a standardized, large-scale dataset of ancient coin imagery and descriptions.
• Methodological Refinement: The study details a robust preprocessing pipeline for extracting single coin reverses from complex auction images and demonstrates the necessity of specific optimization strategies (SGD vs. Adam) for CNNs in this domain to avoid convergence failures.
• Qualitative Insight: Through saliency mapping, the paper offers a comparative analysis of how ViT and CNN models attend to visual features, revealing distinct differences in their internal representations.

Results

• Performance: The ViT models generally outperformed the newly trained CNN models across most semantic elements in terms of accuracy, precision, and recall. The ViT achieved test accuracies ranging from approximately 77% to 83%, while the CNN models ranged from roughly 67% to 82%. The most significant performance gap was observed in the 'Hercules' classification, where ViT vastly outperformed the CNN.
• Training Efficiency: CNN models trained significantly faster (minutes) compared to ViT models (hours), largely due to the smaller parameter count of the CNNs (37M vs. 304M for ViT) and the single-label training approach.
• Saliency and Interpretability:
  • CNNs: Tended to focus on specific, invariant regions (e.g., top-right for cornucopias, bottom-right for eagles). While sometimes intuitive, this rigid focus often led to errors when motifs appeared in different locations or orientations.
  • ViTs: Exhibited highly variable and diffuse saliency regions, often focusing on context or correlated features rather than the motif itself (e.g., focusing on a chair back when an eagle was held by a figure). This suggests ViTs learn a broader, more generalized set of visual cues but are harder to interpret intuitively.
• Data Noise: The study highlighted that label noise, caused by descriptions referencing both coin sides, significantly impacted performance, particularly for concepts like 'standing' and 'seated'.

Significance and Claims
The paper claims that the Vision Transformer architecture is highly promising for the automated analysis of ancient numismatics, demonstrating superior accuracy over the CNN baseline in this specific domain. The authors posit that ViTs' ability to model long-range dependencies without strong inductive biases towards local features allows them to better handle the stylized and variable nature of ancient coin motifs.

However, the authors remain modest in their conclusions. They note that the performance difference between ViT and the CNNs proposed by Cooper and Arandjelovi´c (in prior work) is not definitive, as the CNNs in this study were trained with different hyperparameters (specifically the optimizer) and did not match the performance of the earlier CNN benchmarks. Consequently, the paper suggests that while ViTs show great potential, further hyperparameter tuning and exploration of other Transformer variants are necessary to definitively establish superiority. The study also emphasizes that the quality of automated analysis is fundamentally limited by the noise inherent in weakly supervised datasets derived from auction descriptions.