SPEAR: Structure Property Explainability with Attention Regularization

Technical Summary: SPEAR – Structure–Property Explainability with Attention Regularization

Problem Statement
Machine learning (ML) models are increasingly employed to learn structure–property relationships from spectroscopic and diffraction data, yet their utility in materials discovery is often constrained by poor interpretability. While attention mechanisms are frequently assumed to be inherently explainable, the authors argue that unregularized attention in practice often yields unstable, fragmented, or intensity-driven attribution patterns. In spectroscopic and diffraction contexts, such as X-ray diffraction (XRD), standard attention models tend to focus on the most intense peaks rather than the physically relevant features driving a specific property. This behavior is exacerbated in combinatorial libraries where peaks co-vary, making it difficult for standard attribution methods to isolate causal features from correlated signals. Consequently, relying on unregularized attention can lead to misleading scientific hypotheses, wasting experimental resources or resulting in incorrect physical conclusions.

Methodology
The paper introduces SPEAR (Structure–Property Explainability with Attention Regularization), a framework that treats attention not merely as a post-hoc visualization tool but as a learnable explanatory object constrained during training. The methodology employs a transformer-based regressor where input spectra are tokenized and processed through self-attention layers. To aggregate information for regression, the model uses an attention-weighted pooling mechanism.

SPEAR distinguishes itself from "plain" attention by introducing two explicit regularization terms into the loss function:

1. Learnable Temperature Control ($\mathcal{L}_\tau$): A learnable temperature parameter $\tau$ rescales attention logits prior to softmax normalization. This allows the model to adaptively control the concentration of attention (sharpness vs. diffuseness). A quadratic penalty on $\log \tau$ relative to a reference value prevents degenerate solutions where attention becomes either excessively sharp or overly uniform.
2. Smoothness Penalty ($\mathcal{L}_{smooth}$): A discrete smoothness penalty (Total Variation) is applied along the key dimension (spectral positions). This enforces coherence across neighboring spectral points, preventing erratic jumps in attention weights that do not align with the continuous physical nature of diffraction features.

The total optimization objective combines the standard prediction loss (Mean Squared Error) with these regularization terms, jointly optimizing for predictive accuracy and attention interpretability.

Key Contributions

• Framework Development: The proposal of SPEAR, which embeds interpretability constraints directly into the training process rather than applying them post-hoc.
• Regularization Mechanisms: The formalization of learnable temperature and smoothness penalties to shape attention distributions toward spatially coherent and physically meaningful patterns.
• Validation Strategy: A dual-validation approach using synthetic benchmarks with known generative structures (Gaussian peaks) and experimental XRD data from a combinatorial rare-earth zirconate thin-film library.

Results
The study evaluates SPEAR against plain attention and SHAP (SHapley Additive exPlanations) across two tasks:

1. Synthetic Benchmarks: In a controlled setting where the target is the amplitude of a specific Gaussian peak, plain attention distributed weights broadly across all peaks, tracking intensity rather than the target. In contrast, SPEAR produced smooth, contiguous attribution profiles strictly aligned with the target peak, matching the ground-truth generative structure. This was achieved without sacrificing predictive accuracy.
2. Experimental Peak Feature Regression: When predicting the $2\theta$ position of the {200} reflection in a (Gd,Dy,Ho,Er)$_2$Zr$_2$O$_7$ library, plain attention assigned importance broadly across all prominent peaks, correlating with peak intensity. SPEAR selectively concentrated attribution on the specific {200} peak, effectively decoupling feature importance from raw signal magnitude.
3. Experimental Property Regression: In predicting thermal conductivity (a property dependent on distributed structural information), plain attention again distributed weight broadly. SPEAR identified a specific subset of features, notably emphasizing the {220} reflection.
   • Scientific Insight: The SPEAR model's focus on the {220} peak led to a reassessment of prior structural analysis. The authors found a correlation between the {220} peak position, tetragonal distortion accommodating cation size disorder, and local thermal conductivity—a relationship previously missed in their synchrotron analysis.
   • Performance: In all cases, SPEAR maintained or slightly improved predictive accuracy and convergence speed compared to plain attention, demonstrating that interpretability constraints do not hinder performance.

Significance and Claims
The paper claims that attention regularization provides a principled training constraint for explainable structure–property regression. By treating attention as a learnable object constrained by physical priors (smoothness and bounded concentration), SPEAR yields stable and mechanistically meaningful explanations without sacrificing predictive performance. The authors emphasize that this approach transforms attention from a potentially misleading visualization into a reliable guide for hypothesis generation. The work suggests that for spectroscopic and diffraction data, where features are physically continuous and correlated, enforcing spatial coherence in attention is essential for extracting valid physical insights. The framework is presented as broadly applicable to other spectroscopic, imaging, and spatially structured datasets in high-throughput materials discovery.