Technical Summary: Benchmarking Deep Learning Models for Raman Spectroscopy

Problem Statement

Deep learning (DL) classifiers for Raman spectroscopy are frequently reported to outperform classical chemometric approaches. However, the field suffers from a lack of standardized evaluation. Existing studies often evaluate models in isolation, comparing them only against traditional machine learning (ML) baselines or trivially adapted vision architectures not originally designed for spectral data. Furthermore, evaluations are frequently conducted on small, experimental datasets or synthetic data, limiting the assessment of real-world robustness. Consequently, direct comparisons between multiple published, Raman-specific deep learning models on shared, open-source datasets are scarce, hindering the identification of effective architectural strategies.

Methodology

This study establishes a unified benchmark to compare five supervised deep learning models and two conventional machine learning methods across three distinct open-source Raman datasets.

Benchmark Models

The study evaluates five Raman-specific DL architectures representing diverse design philosophies:

1. Deep CNN [31]: A LeNet-5 inspired 1D-CNN with fixed kernel sizes.
2. SANet [5]: A Scale-Adaptive Network utilizing multi-scale blocks with varying kernel sizes to capture peaks of different widths.
3. RamanNet [20]: A model designed to avoid translational equivariance by using overlapping sliding windows with unique MLPs per window.
4. Transformer [30]: A Vision Transformer (ViT) adapted for 1D spectra using patch-based tokenization and self-attention.
5. RamanFormer [23]: A modified transformer architecture with reduced embedding dimensions and strided convolutions for spatial hierarchy.

Two conventional ML baselines were also included: Random Forest and Support Vector Classifier (SVC).

Datasets

Three open-source datasets were selected to cover different domains and distribution shift challenges:

• MLROD (Minerals): Contains spectra of minerals and mixtures. It includes a specific "dusty" test regime (simulating Martian conditions) to evaluate performance under significant distribution shift and background interference.
• Bacteria-ID (Biomedical): Contains spectra of bacterial isolates and antibiotic treatments. It features a reference set for pretraining and a fine-tuning set with optical degradation similar to the test set.
• API (Pharmaceutical): Contains spectra of active pharmaceutical ingredients. The train and test splits are drawn from the same distribution, representing a standard in-distribution classification task.

Experimental Protocol

• Preprocessing: To ensure reproducibility and isolate model performance, only intensity scaling was applied. No baseline correction, denoising, or fluorescence removal was performed.
• Training: All models were trained using a unified protocol (Adam optimizer, Cross-Entropy loss) with independent hyperparameter tuning (grid search over batch size and learning rate) for each model-dataset pair.
• Evaluation: Performance was measured using Accuracy and Macro-averaged F1 Score across five independent runs. The study explicitly avoided AUC-ROC as the primary metric, focusing on final class assignment accuracy.

Key Results

1. Performance on Distribution Shift (MLROD)

On the MLROD dataset, all models exhibited significant performance degradation on "dusty" test samples compared to "clean" samples, highlighting sensitivity to background interference and acquisition shifts.

• Top Performers: SANet and Deep CNN demonstrated the strongest overall robustness. SANet achieved the highest overall accuracy (80.34%), outperforming the worst-performing DL model (Transformer) by over 7 percentage points.
• Baseline Comparison: On this large, diverse dataset, the Raman-specific DL models generally outperformed the conventional ML models (Random Forest and SVC), suggesting DL architectures are better suited for capturing discriminative spectral patterns in complex, shifted environments.

2. Multi-Task Classification (Bacteria-ID)

• Isolate Classification: All DL models achieved accuracies between 83% and 86%. The Transformer performed slightly worse than the CNN-based models.
• Treatment Prediction: All models performed exceptionally well (96–98% accuracy), with DL models slightly outperforming conventional ML.
• Observation: The performance gap between DL and ML was narrower here compared to MLROD, but DL models maintained a slight edge.

3. In-Distribution Classification (API)

• Performance: All models, including conventional ML, achieved near-ceiling performance (99–100% accuracy) on the API dataset.
• Label Ambiguity: The study noted that two classes (4-Methyl-2-pentanone and Methyl Isobutyl Ketone) are chemically identical. When these labels were merged (31-class setting), accuracy reached 100% for several models, indicating that errors in the 32-class setting were primarily due to label aliasing rather than model failure.
• ML vs. DL: On this smaller, in-distribution dataset, conventional ML models (particularly SVC and Random Forest) performed comparably to the deep learning models.

Key Contributions

1. First Multi-Model Benchmark: This work presents one of the first studies to compare three or more published, Raman-specific deep learning classifiers across multiple open-source datasets under a single, reproducible protocol.
2. Unified Evaluation Framework: By enforcing consistent preprocessing (minimal), training protocols, and hyperparameter search spaces, the study provides a fair comparison that isolates architectural differences from experimental artifacts.
3. Distribution Shift Analysis: The study explicitly quantifies the brittleness of current models to domain shifts (e.g., dust contamination, hardware variations), showing that while models excel in-distribution, they struggle significantly when test conditions diverge from training conditions.
4. Baseline Establishment: The paper establishes transparent baselines (including Random Forest and SVC) to contextualize the necessity of complex DL architectures.

Significance and Claims

The authors claim that this benchmark provides a necessary foundation for advancing the state-of-the-art in Raman spectroscopy. The study modestly concludes that:

• Architecture Matters: Raman-specific architectures (like SANet and Deep CNN) offer advantages over generic or purely attention-based models (like the standard Transformer) when dealing with large, diverse datasets and distribution shifts.
• Simplicity vs. Complexity: Simpler machine learning methods remain competitive when the classification task involves limited train-test distribution shift and smaller datasets.
• Data Quality over Architecture: The significant performance drop on the "dusty" MLROD samples suggests that current models are highly sensitive to acquisition variability. The authors argue that future progress relies less on architectural novelty and more on the development of large-scale, diverse, and curated experimental datasets to train robust foundation models capable of handling real-world variability.

The paper does not claim to solve the distribution shift problem but rather highlights it as a critical bottleneck, suggesting that future work should focus on unsupervised domain adaptation, self-supervised learning, and foundation models trained on massive, diverse spectral libraries.