Technical Summary: SpecX – A Large-Scale Benchmark for Multi-Modal Spectroscopy and Cross-Paradigm Evaluation

1. Problem Statement

Existing benchmarks for spectroscopic machine learning suffer from three critical limitations that hinder the development of robust, generalizable chemical AI:

1. Insufficient Scale and Modality Coverage: Current datasets typically contain fewer than one million molecules and omit practically important modalities such as UV-Vis, fluorescence, and Raman spectroscopy. This scale is inadequate for training foundation models with robust generalization capabilities.
2. Lack of Experimentally Grounded Evaluation: Most large-scale datasets rely entirely on simulated spectra. These simulations systematically diverge from experimental data due to factors like solvent effects, instrument response, and temperature broadening, creating a significant "simulation-to-reality" gap. No existing benchmark provides a systematically aligned experimental subset.
3. Fragmented Evaluation Across Paradigms: The field is split between specialized spectral models (e.g., MassSpecGym, NMRNet), which focus on signal-level precision but ignore Multimodal Large Language Models (MLLMs), and MLLM-oriented benchmarks (e.g., SpectrumBench), which evaluate high-level reasoning but lack rigorous signal-level protocols. There is no unified framework to compare these paradigms side-by-side.

2. Methodology and Dataset Construction

The authors introduce SpecX, a large-scale benchmark comprising 1.7 million molecules spanning eight spectral modalities: ¹H-NMR, ¹³C-NMR, HSQC-NMR, IR, MS, UV-Vis, Raman, and Fluorescence (FL).

Data Curation and Filtering

• Source Integration: Molecules were aggregated from five public repositories (QME14S, ViBench, ChEMBL, Multimodal Spectroscopic Dataset, MassSpecGym), resulting in an initial pool of ~2.73 million unique SMILES.
• Filtering Pipeline: A multi-stage filter ensured chemical validity and spectral computability:
  • Retained molecules with 5–35 heavy atoms.
  • Restricted elemental composition to {C, H, O, N, S, P, Si, B, F, Cl, Br, I}.
  • Enforced physical meaningfulness for specific modalities (e.g., requiring chromophores/fluorophores for UV/FL simulations).
  • Result: 1,701,739 molecules retained (62.36% retention rate).

Three-Tier Architecture

SpecX is organized into three distinct tiers to support different stages of research:

1. Large Subset (~1M molecules): A simulated dataset for pretraining and tasks involving structure elucidation, spectrum simulation, and functional group prediction. It covers seven modalities (excluding FL for benchmarking).
2. Small Subset (4,496 molecules): A strictly modality-aligned multispectral subset where all seven modalities are available for every molecule. This is used for multimodal Question Answering (QA) and cross-modal evaluation.
3. Exp Subset (432 molecules): A high-quality subset containing experimentally measured spectra (UV-Vis and MS only) for evaluating real-world generalization and the simulation-to-reality gap.

Data Generation Protocols

Spectra were generated using modality-specific simulation protocols to balance physical realism with computational scalability:

• NMR: Simulated using MestReNova (CDCl₃ solvent).
• IR: Generated via Molecular Dynamics (MD) simulations in LAMMPS using GAFF force fields, followed by Fourier transform of dipole moment trajectories.
• MS/MS: Simulated using CFM-ID with fixed collision energy (20 eV).
• UV-Vis, Raman, Fluorescence: Computed using Time-Dependent Density Functional Theory (TD-DFT) and DFT frequency calculations in the ORCA package.

Data Representation

To enable training on Transformer architectures, spectral data was converted into structured text representations:

• Peak-based (NMR, MS): Encoded as lists of attributes (e.g., chemical shift, multiplicity, integration, m/z, intensity).
• Continuous (IR, UV-Vis, Raman): Discretized into fixed-length vectors of intensity tokens.

3. Evaluation Tasks and Models

SpecX defines four evaluation tasks, assessed under both Random Splits (in-distribution interpolation) and Scaffold Splits (out-of-distribution generalization based on Bemis–Murcko scaffolds).

1. Structure Elucidation (Spectra → SMILES): Predicting molecular structure from spectral inputs.
   • Models: Vanilla Encoder-Decoder Transformers trained on individual and combined modalities.
2. Functional Group Prediction: Multilabel classification of functional groups from spectra.
   • Models: XGBoost (on engineered features) and 1D-CNN (on raw spectral vectors). Molecular formulas were withheld to simulate blind analysis.
3. Spectrum Simulation (SMILES → Spectra): Generating expected spectra from molecular structures.
   • Models: Inverted Transformer architecture.
4. Spectral Question Answering (QA): Evaluating MLLMs on chemical reasoning.
   • Models: DeepSeek-V3, GPT-4.1-mini, and Qwen2.5-3B in a zero-shot setting.
   • Tasks: (a) SMILES inference from spectra; (b) Functional group inference. Inputs were provided as structured text (no images) without molecular formula priors.

4. Key Results

Specialized Models vs. MLLMs

• Signal-Level Modeling: Specialized models (Transformers, XGBoost, 1D-CNN) excel at signal-level tasks. In structure elucidation, multi-modality integration achieved a Top-1 accuracy of 59.04% (Random Split) and 29.66% (Scaffold Split) on the Large subset. NMR modalities (¹H, ¹³C) performed best among single modalities.
• High-Level Reasoning: MLLMs demonstrated strong high-level reasoning capabilities but failed significantly on precise spectral grounding.
  • Structure Elucidation: MLLMs achieved near-zero Top-1 accuracy (e.g., 0.6% for DeepSeek-V3 on ¹H-NMR) on the Small subset and 0.000% on the experimental Exp subset.
  • Functional Group Inference: While MLLMs showed superficial accuracy (0.75–0.87), their macro-averaged F1 scores were substantially lower than specialized baselines (e.g., 0.379 for DeepSeek-V3 vs. 0.824 for XGBoost on ¹H-NMR). Qwen2.5-3B collapsed to zero F1 scores, predicting "no" for all samples.

Generalization and Modality Performance

• Out-of-Distribution: Performance drops were significant under Scaffold splits across all tasks, highlighting the difficulty of generalizing to structurally novel chemical spaces.
• Modality Specificity: Raman and IR spectroscopy yielded the highest F1 scores for functional group prediction (up to 0.97), while UV-Vis remained the weakest modality for identifying diverse functional groups due to broad electronic transitions.
• Sim-to-Real Gap: The complete failure of MLLMs on the experimental Exp subset (0.000 accuracy) underscores the current inability of models to transfer from simulated training data to real-world experimental spectra.

5. Significance and Claims

The paper claims that SpecX establishes a unified benchmark for spectral intelligence that bridges the gap between specialized spectral models and general-purpose MLLMs.

• Unified Evaluation: It is the first benchmark to enable side-by-side comparison of signal-level precision (specialized models) and high-level reasoning (MLLMs) across a massive, multi-modal dataset.
• Highlighting Domain Gaps: The results explicitly demonstrate that while MLLMs possess reasoning capabilities, they lack the precise spectral grounding required for chemical tasks, particularly when moving from simulation to reality.
• Call for Spectrum-Native Models: The authors argue that the observed limitations necessitate the development of "spectrum-native foundation models" that tightly couple signal-level continuous spectral encoders with high-level discrete reasoning modules, rather than relying solely on general multimodal pretraining.
• Foundation for Future Research: By providing a large-scale, aligned, and experimentally grounded dataset, SpecX aims to catalyze progress in automated chemical discovery and the investigation of the simulation-to-reality gap.

The paper concludes with modesty regarding its limitations, acknowledging the reliance on simulated data, the bias toward synthetic compounds, and the current restriction of experimental data to UV-Vis and MS. It posits that future work must expand experimental coverage and explore constrained decoding or tool-use plugins to improve MLLM performance on strict chemical tasks.