Comparative Evaluation of Vision Transformer, Hybrid CNN–MLP, and Transfer-Learned ResNet-18 for CIFAR-10 Image Classification

Technical Summary: Comparative Evaluation of Vision Transformer, Hybrid CNN–MLP, and Transfer-Learned ResNet-18 for CIFAR-10 Image Classification

Problem Statement
Image classification remains a fundamental challenge in computer vision. While Convolutional Neural Networks (CNNs) have historically dominated due to their inherent inductive biases for spatial locality and weight sharing, Vision Transformers (ViTs) have emerged as a powerful alternative by processing images as sequences of patches using self-attention mechanisms. However, the performance of ViTs on smaller, lower-resolution datasets like CIFAR-10 (32×32 pixels) without large-scale pretraining remains a critical area of investigation. This study addresses the comparative efficacy of three distinct deep learning strategies on the CIFAR-10 benchmark: a patch-based Vision Transformer, a hybrid CNN–Multi-Layer Perceptron (CNN–MLP), and a transfer-learned ResNet-18 initialized with ImageNet weights.

Methodology
The research evaluates three architectures trained on the standard CIFAR-10 dataset (50,000 training images, 10,000 test images) using identical preprocessing and augmentation protocols (random cropping, horizontal flipping, and normalization).

1. Vision Transformer (ViT): The model adopts the patch-based representation where 32×32 images are divided into 64 non-overlapping 4×4 patches. These patches are flattened, projected into an embedding space, and processed through a Transformer encoder consisting of six blocks with multi-head self-attention, layer normalization, and feed-forward networks. A learnable classification token is prepended to the sequence.
2. Hybrid CNN–MLP: This architecture serves as a baseline, utilizing convolutional layers to extract local spatial features (edges, textures) followed by a fully connected MLP classifier. It relies on the inductive biases of convolution rather than self-attention.
3. Transfer-Learned ResNet-18: This model utilizes a ResNet-18 backbone pretrained on ImageNet. The original classification head is replaced with a 10-unit fully connected layer to match CIFAR-10 classes, leveraging representations learned from a significantly larger source dataset.

All models were trained for up to 50 epochs using the Adam optimizer with learning-rate scheduling and cross-entropy loss. Evaluation metrics included accuracy, precision, recall, F1-score, loss trajectory analysis, and confusion matrix examination.

Key Contributions

• Implementation and Evaluation: The study implements and evaluates a specific patch-based ViT configuration for CIFAR-10 classification.
• Comparative Analysis: It provides a direct performance comparison between a self-attention-based model, a hybrid convolutional model, and a transfer-learned residual network.
• Granular Error Analysis: Beyond aggregate metrics, the paper offers a detailed confusion-matrix analysis of the ViT to identify specific class-level confusion patterns.
• Inductive Bias vs. Pretraining: The work discusses the relative influence of convolutional inductive biases versus the benefits of large-scale pretraining in a limited-data regime.

Results
The experimental results indicate a clear hierarchy in performance among the three approaches:

• ResNet-18 (Transfer Learning): Achieved the highest reported accuracy of 88.7%. Its training and validation loss curves demonstrated rapid convergence and stable generalization, attributed to the combination of convolutional inductive bias and ImageNet pretraining.
• Hybrid CNN–MLP: Achieved an accuracy of 84.2%. It converged effectively, though with some fluctuations in validation loss, outperforming the ViT by leveraging local spatial structures.
• Vision Transformer: Achieved an accuracy of 78.2% on a 500-sample evaluation subset (derived from the provided confusion matrix). The model showed a decreasing training loss but a widening gap between training and validation loss in later epochs, suggesting weaker generalization compared to the CNN-based approaches under these specific conditions.

Confusion matrix analysis for the ViT revealed non-uniform class performance, with higher recall for classes 0 (airplane) and 9 (truck), and lower recall for class 6 (frog). Frequent confusions were observed between semantically or visually similar neighboring classes (e.g., 0 vs. 1, 2 vs. 3).

Significance and Claims
The paper concludes that while the Vision Transformer successfully learns meaningful patch-based representations and fits the training data, it does not outperform CNN-based approaches in this specific low-resolution, moderate-data setting without large-scale pretraining. The superior performance of the ResNet-18 is attributed to the dual advantages of convolutional inductive biases (locality and weight sharing) and the prior visual knowledge gained from ImageNet pretraining.

The author explicitly states that these results should not be interpreted as evidence that ViTs are intrinsically inferior to CNNs. Instead, the findings highlight that for small datasets like CIFAR-10, a pretrained convolutional architecture currently provides a more robust solution. The study acknowledges limitations, including the use of a partial test set for ViT evaluation, the lack of a controlled architecture-only comparison for ResNet (due to pretraining), and the absence of computational efficiency metrics. Future work is suggested to include testing on the full CIFAR-10 test set, exploring pretrained ViTs, and conducting rigorous hyperparameter and efficiency analyses.