XRFormer: Multiscale Tokenization for XRF Representation Learning

Technical Summary: XRFormer for XRF Representation Learning

Problem Statement
X-ray fluorescence (XRF) spectroscopy is a critical non-destructive technique for elemental analysis in Cultural Heritage (CH), particularly for pigment identification and unmixing. However, automated learning from XRF spectra remains challenging due to the complex nature of the signals. XRF spectra are one-dimensional signals characterized by sharp elemental peaks, broader spectral structures, and background variations. Existing learning-based models, such as 1D-CNNs, effectively capture local patterns but struggle to model long-range dependencies across the energy axis, which are crucial for understanding correlated elemental emissions and background effects. Conversely, standard Vision Transformers (ViT) and recent spectral adaptations often fail to explicitly account for the specific physical characteristics of XRF data, such as the coexistence of narrow peaks and broad structures, or they require excessive computational resources to maintain high spectral resolution.

Methodology
The paper introduces XRFormer, a transformer architecture specifically tailored for XRF spectra. The core innovation lies in a multiscale convolutional tokenizer that precedes a standard transformer encoder.

1. Multiscale Tokenization: Unlike linear patch-wise embedding (ViT) or simple group-wise spectral embedding (SpectralFormer), XRFormer employs a tokenizer that progressively reduces spectral resolution while increasing embedding dimensionality. This is achieved through three 1D convolutional blocks with increasing effective receptive fields. Strided convolutions downsample the spectral resolution by a factor of two after the first and second blocks, while an adaptive pooling layer at the end produces a fixed number of tokens. This design injects locality and multi-resolution inductive biases, allowing the model to capture both narrow elemental peaks and broader background structures before global self-attention is applied.
2. Architecture: The resulting token sequence is processed by a standard transformer encoder using Multi-Head Self-Attention (MHSA) to model global spectral dependencies. A learnable [CLS] token is prepended to serve as a global representation for downstream tasks.
3. Self-Supervised Pretraining: To address data limitations common in CH applications, the authors investigate two self-supervised objectives:
   • Masked Spectral Modeling (MSM): A reconstruction task where 15% of tokens are masked, and the model learns to reconstruct original spectral values, encouraging the capture of both local peak shapes and long-range dependencies.
   • Peak Presence Prediction (PPP): A physics-informed pretext task where the model predicts a binary signature of peak presence derived from local maxima detection on the input spectrum. This task does not rely on pigment labels, making it a modality-specific objective that forces the model to focus on characteristic emission peaks.

Key Contributions

• Modality-Aware Tokenization: The proposal of a multiscale convolutional tokenizer that effectively combines locality with multi-resolution analysis, specifically designed to handle the dual nature of XRF signals (sharp peaks and broad structures).
• Physics-Informed Self-Supervision: The introduction of the Peak Presence Prediction (PPP) task, which leverages physical properties of XRF spectra (peak detection) as a pretext task to enhance representation learning without requiring labeled pigment data.
• Comprehensive Evaluation: A rigorous evaluation on the Pigments Checker STANDARD v.5 (PCSv5) dataset for both pigment identification and unmixing, comparing XRFormer against ViT, SpectralFormer (with and without Cross-layer Adaptive Fusion), and 1D-CNN baselines.

Experimental Results
Experiments were conducted on the PCSv5 dataset (22 pigments) and the Infraart dataset for pretraining.

• Pigment Identification: XRFormer consistently outperformed ViT, SpectralFormer (without CAF), and 1D-CNN baselines. Specifically, XRFormer improved Absolute Accuracy (AA) from 68.76% (ViT) to 71.29% in the non-pretrained setting. With self-supervised pretraining (MSM+PPP), the best XRFormer configuration achieved an AA of 76.78%, surpassing all other models.
• Pigment Unmixing: XRFormer demonstrated robust abundance estimation, outperforming ViT in Abundance RMSE (0.0435 vs. 0.0457). While SpectralFormer achieved the best unmixing metrics overall, this was attributed to its significantly higher capacity (3.37M parameters, 512 tokens) compared to XRFormer (1.5M parameters, 128 tokens). XRFormer achieved competitive unmixing performance with less than half the parameters and at a lower token resolution.
• Impact of Pretraining: MSM provided consistent gains across both tasks. The addition of PPP further enhanced performance, particularly when peak prominence was tuned appropriately (higher prominence for identification, lower for unmixing).
• Model Scaling: Increasing model capacity from XRFormer-B to XRFormer-L led to systematic improvements in both identification and unmixing metrics.

Significance and Claims
The paper claims that multiscale, modality-aware tokenization is an effective and parameter-efficient foundation for transformer-based XRF modeling, particularly under data-limited conditions. The results suggest that the primary performance gains in XRFormer stem from the alignment of the tokenization strategy with the physical characteristics of XRF signals, rather than solely from encoder-level refinements or increased model complexity. The authors posit that this approach offers a robust alternative to high-resolution, high-parameter models like SpectralFormer for tasks requiring efficient representation learning.

The authors maintain a modest stance regarding the broader impact, noting that the scarcity of large-scale benchmark datasets and diverse real-world samples in the Cultural Heritage field currently limits the ability to conduct deeper ablations or comparisons across multiple independent datasets. They express hope that this work will motivate the community to develop open, standardized benchmarks for rigorous evaluation of future deep learning models in this domain.