Residue-Level Attributions in Protein Language Models Do Not Recover Allergen Epitopes

Technical Summary: Residue-Level Attributions in Protein Language Models Do Not Recover Allergen Epitopes

Problem Statement
Deep learning models, particularly protein language models (PLMs), have significantly improved the accuracy of protein-level allergenicity prediction, aiding in the safety screening of novel foods. However, a critical gap remains in understanding whether the explanations generated by these models (residue-level attributions) capture biologically meaningful information. Specifically, it is unclear if high attribution scores correspond to experimentally validated immune-recognition sites (epitopes). While models often produce heatmaps that qualitatively appear to align with known motifs, their quantitative alignment with ground-truth epitopes is unknown. The paper distinguishes between model faithfulness (whether an explanation highlights residues that causally influence the model's output) and immunological faithfulness (whether those residues align with experimentally observed immune-recognition sites).

Methodology
The authors constructed an epitope-grounded benchmark to quantitatively evaluate residue-level attribution signals from allergenicity classifiers.

• Data Curation: A dataset was curated from the Immune Epitope Database (IEDB), focusing on experimentally validated, allergy-associated MHC class II epitopes (both B-cell and T-cell). Positive sequences were mapped to full-length parent proteins from UniProt and NCBI. Negative sequences were drawn from reviewed UniProtKB/Swiss-Prot entries, excluding viral/human proteins and allergens. The dataset was split into training and evaluation sets using homology-aware clustering to prevent data leakage.
• Models Evaluated: Three classifiers were tested:
  1. Frozen ESM-2: A standard PLM with a frozen backbone and a trained classification head.
  2. MTL ESM-2: A multi-task learning variant sharing the frozen ESM-2 backbone but adding an auxiliary residue-level epitope prediction head to test if explicit epitope supervision improves alignment.
  3. DeepPlantAllergy: A published architecture retrained on the curated dataset for controlled comparison.
• Attribution Methods: The study evaluated Integrated Gradients (IG), occlusion, attention weights, Gradient×Input, and the auxiliary MTL residue head.
• Evaluation Metrics: Residue-level performance was assessed using AUROC, AUPRC, and Precision@k, comparing attribution scores against binary masks of annotated epitopes. Random baselines were generated to establish statistical significance.
• Functional Validation:
  • Masking: To test model faithfulness, residues ranked by IG were masked to measure the decrease in predicted allergenicity.
  • Saturation Mutagenesis: Systematic single-residue substitutions were performed to determine if the model's sensitivity was driven by specific epitope identities or broader physicochemical properties.

Key Results

1. Strong Classification, Weak Attribution: All three models achieved high protein-level classification performance (AUROC > 0.96). However, none of the residue-level attribution signals (IG, occlusion, etc.) significantly exceeded random baselines in aligning with annotated epitopes across AUROC, AUPRC, or Precision@k. Several signals performed significantly worse than random.
2. Failure of Multi-Task Learning: The MTL ESM-2 model, which included an auxiliary head trained directly on epitope masks, failed to improve the alignment of the primary classification head's attributions with epitopes. This suggests that protein-level allergenicity classification can be achieved without relying on localized epitope features.
3. Dissociation of Faithfulness:
   • Model Faithfulness: Integrated Gradients were found to be model-faithful. Masking residues identified by IG significantly reduced the model's predicted allergenicity more than random masking, confirming IG identifies residues causally important to the model's decision.
   • Immunological Misalignment: Despite being model-faithful, these residues did not overlap with annotated epitopes.
4. Physicochemical vs. Epitope Specificity: Saturation mutagenesis revealed that the models were sensitive to global physicochemical and compositional features (e.g., charge, hydrophobicity, aromaticity) rather than specific epitope sequences. Mutagenesis effects were often larger outside annotated epitope regions than inside them.

Contributions

1. Epitope-Grounded Benchmark: The authors provide a quantitative benchmark using AUROC, AUPRC, and Precision@k to evaluate the immunological faithfulness of allergenicity model explanations.
2. Evidence of Misalignment: The study demonstrates that strong allergenicity classifiers consistently fail to produce residue-level attribution signals that align with experimental epitopes better than random chance.
3. Dissociation of Faithfulness Criteria: The work establishes a clear dissociation between model faithfulness (IG identifies causal residues) and immunological faithfulness (those residues are not epitopes), suggesting models rely on global physicochemical features rather than epitope-specific mechanisms.

Significance and Claims
The paper concludes that residue-level importance signals from current protein language models should not be interpreted as immunological explanations for safety screening or hypoallergen design without quantitative validation. The findings imply that:

• Modifying residues highlighted by attribution methods may not reduce allergenicity if those signals are not epitope-aligned.
• Attribution visualizations may serve for model auditing but do not constitute evidence of epitope-level understanding.
• Future work should focus on biologically constrained predictors that incorporate structural and immunological priors, and the community requires standardized, epitope-grounded benchmarks to evaluate whether model explanations capture biologically meaningful mechanisms.

The authors acknowledge limitations, including the incompleteness and bias of IEDB annotations, the restriction to linear MHC-II residue masks, and the use of in silico perturbations as proxies for experimental immune recognition.