Developing SCL2205 : A Protein Sequence-based Spatial Modelling Dataset for the Protein Language Model Frontier

This paper introduces SCL2205, a high-quality, rigorously curated protein sequence dataset derived from UniProtKB that addresses data leakage and quality issues in existing benchmarks, demonstrating significant performance improvements over state-of-the-art models for protein sub-cellular localization prediction while being openly available to foster reproducible deep learning research.

The Gist

Imagine you are trying to teach a robot to navigate a giant, bustling city (the cell). Your goal is to teach the robot to recognize where different buildings (proteins) are located: is the library in the nucleus? Is the power plant in the mitochondria? Is the post office secreted outside the cell?

For a long time, scientists have been trying to build this robot using Deep Learning (a type of advanced AI). But they've been running into two major problems:

1. Bad Maps: The data they used to train the robot was messy, inconsistent, or outdated.
2. Cheating: Sometimes, the robot was "cheating" by memorizing the answers to questions it had already seen in a slightly different form, making it look smarter than it actually was.

This paper introduces SCL2205, a brand new, high-quality "training manual" designed to fix these problems and build a truly smart navigation robot.

Here is a breakdown of what the authors did, using some everyday analogies:

1. Cleaning Up the Messy Library (Data Preprocessing)

Imagine you have a library with 470,000 books (protein sequences). But many books are torn, written in different languages, or have missing pages.

• The Old Way: Researchers would grab a handful of books, maybe ignore the torn ones, and start teaching the robot. This led to confusion.
• The SCL2205 Way: The authors acted like strict librarians. They:
  • Threw away books with missing pages (low-quality data).
  • Only kept books written in "Eukaryotic" (a specific type of cell) language.
  • Checked the "quality score" on the spine of every book to ensure it was reliable.
  • Result: They created a pristine, curated collection of 22,000+ high-quality books.

2. The "Grouping" Strategy (Label Mapping)

In the city, some buildings are very specific. You might have a "Chloroplast Thylakoid Membrane." That's a very specific room inside a specific building.

• The Problem: If you only have 6 books about "Chloroplast Thylakoid Membrane," the robot can't learn much. It's like trying to teach a student about "The History of a specific brick in a wall" when you only have one brick to look at.
• The Solution: The authors used human intelligence to group these specific rooms into broader categories. They said, "Okay, instead of just teaching about that one specific room, let's teach about the whole 'Plastid' building."
• The Analogy: It's like grouping "Red 2024 Toyota Camry," "Blue 2023 Honda Civic," and "Green 2022 Ford F-150" all under the category "Cars." Suddenly, instead of having 6 examples of a specific car, you have thousands of examples of "Cars." This helps the robot learn the general rules of driving much faster.
• The Result: By doing this manual grouping, they increased their training data by 71%, giving the robot a much richer education.

3. Catching the Cheaters (Stopping Data Leakage)

This is the most critical part of the paper.

• The Problem: In the past, researchers would use a trick called "Homology Augmentation." Imagine you are testing a student on a math quiz. To help them study, you give them a practice quiz where the numbers are slightly changed (e.g., $2+2$ becomes $2+3$). If the student memorized the pattern of the first quiz, they might guess the answer to the second one without actually understanding math.
• The Discovery: The authors found that this "practice quiz" trick was actually cheating. Because the practice questions were so similar to the test questions (due to biological similarity), the robot was memorizing the answers rather than learning the rules.
• The Proof: They showed that even when they tried to be careful, about 4.8% of the "test" data was actually just a copy of the "training" data in disguise. This made previous robots look 10% better than they really were.
• The Fix: SCL2205 uses a strict "separation wall." They ensure that the training data and the testing data are so different that the robot can't cheat. It forces the robot to actually learn the concept of "location" rather than just memorizing patterns.

4. The Results: A Smarter Robot

The authors tested their new dataset against the old, popular datasets (like DeepLoc).

• The Test: They built two robots: one trained on the old messy data, and one trained on their new SCL2205 data.
• The Outcome: The robot trained on SCL2205 was significantly better.
  • On standard tests, it was up to 10.8% more accurate.
  • It worked especially well with the newest, most powerful AI models (called Protein Language Models), which are like the "GPT-4" of biology.

Why Does This Matter?

Think of AI in biology as a new super-power. But if you give a super-power to a robot with bad instructions, it might crash or make dangerous mistakes.

• Trust: SCL2205 ensures that when scientists say an AI model is "90% accurate," they really mean it. It's not just a fluke caused by cheating.
• Efficiency: Because the data is cleaner, the robots don't need to study as long or use as much electricity to learn.
• Open Access: The authors didn't hide their work. They made the dataset free for everyone to download (like a free app on your phone), so other scientists can build better tools to find cures for diseases.

In short: The authors cleaned up the training data, stopped the AI from cheating, and gave it a better curriculum. The result is a more trustworthy, accurate, and powerful tool for understanding how life works at a microscopic level.

Technical Summary

1. Problem Statement

Deep learning (DL) has advanced protein sub-cellular localisation (SCL) prediction, but its potential is hindered by three critical issues in current data preparation:

• Data Quality and Bias: Existing datasets (e.g., DeepLoc) often rely on inconsistent preprocessing, outdated database versions, and suboptimal filtering, leading to avoidable biases.
• Data Leakage via Homology Augmentation: A common strategy in SCL prediction is "homology augmentation" (searching databases to find similar sequences to expand training data). The authors demonstrate that this practice often inadvertently reintroduces training data into the testing set (data leakage), inflating performance metrics and compromising model generalizability.
• Limited Diversity and Sequence Truncation: Many predictors aggressively truncate protein sequences (e.g., to 1,000 residues) to save computational resources, potentially discarding critical C-terminal signals. Additionally, rare sub-cellular locations are often excluded due to low sample counts, limiting model generalization.

2. Methodology

The authors developed SCL2205, a high-quality dataset derived from the UniProtKB/Swiss-Prot release (2022 05). The construction involved a rigorous, multi-stage pipeline:

• Data Source & Filtering:
  • Retrieved 469,935 records from UniProtKB/Swiss-Prot.
  • Applied strict filters: experimental evidence codes (ECO:0000269), eukaryotic origin, annotation quality scores $\ge$ 3, and sequence lengths between 30 and 5,000 amino acids (preserving full-length proteins).
• Manual Label Mapping (Input Augmentation):
  • To address class imbalance and increase data diversity, the authors manually mapped rare, specific sub-compartment labels to higher-order cellular components (e.g., mapping specific membrane sub-locations to a general "Membrane" class).
  • This increased the total dataset size by 71% (from 12,922 to 22,152 sequences) and specifically boosted rare classes like "Plastid" by over 100-fold.
• Custom Homology Reduction:
  • Instead of standard tools like CD-HIT (which favor longer sequences), the authors implemented a custom BLAST-based algorithm that factors in sequence-pair length without bias.
  • Three-Step Reduction Strategy:
    1. Redundancy Reduction (80% threshold): Applied to the preprocessed dataset to remove near-duplicates.
    2. Overlap Reduction (30% threshold): Ensured no sequences in the training set shared >30% identity with the validation or testing sets.
    3. Evaluation Bias Mitigation: Further reduced overlap between validation and testing sets.
• Dataset Splits:
  • Created two training tracks: Training-Validation-Testing (TVT) and Cross-Validation-Testing (CVT).
  • Provided a held-out independent testing set for final evaluation.
• Experimental Design:
  • Leakage Analysis: Tested the impact of homology augmentation by using only 10% of the training set to query a database (RefSeq) and measuring the resulting overlap with the test set.
  • Benchmarking: Compared SCL2205 against the state-of-the-art (SoTA) DeepLoc2 dataset (reduced at 80% homology).
  • Models: Trained both Convolutional Neural Networks (CNN) and Protein Language Model (PLM) embeddings (Rostlab/prot_t5_xl_uniref50) on both datasets.
  • Evaluation: Used independent test sets: DEEP-SS (in-distribution, UniProt-based) and DEEP-HPA (out-of-distribution, Human Protein Atlas-based). Metrics included Macro PR-AUC, McNemar's test, and stratified bootstrapping.

3. Key Contributions

1. Quantification of Data Leakage: The study is the first to quantify the extent of training-to-testing data leakage caused by homology augmentation. They demonstrated that augmenting just 10% of the training set resulted in a 4.8% overlap with the test set, challenging the validity of many existing SoTA predictors.
2. SCL2205 Dataset: A robust, "leak-proof" dataset with:
   • Full-length sequence preservation (up to 5,000 AA).
   • Manual label mapping to enhance class diversity.
   • Stringent homology reduction (≤30% overlap between splits).
3. Open Accessibility: The dataset is released under CC0 1.0 on DRYAD and as a Python package (p-scldata) on PyPI to ensure reproducibility and ease of integration.
4. Methodological Rigor: Introduced a custom homology reduction algorithm and a comprehensive evaluation framework distinguishing between in-distribution and out-of-distribution performance.

4. Results

• Impact of Label Mapping:
  • Models trained with mapped labels (Model A) showed a 9.0% increase in Macro PR-AUC on the in-distribution DEEP-SS test set compared to native labels (Model B).
  • On the out-of-distribution DEEP-HPA set, results were mixed, suggesting a trade-off between generalization (ranking) and specific classification accuracy.
• Dataset Performance (SCL2205 vs. DeepLoc2):
  • PLM-based Models: SCL2205 significantly outperformed DeepLoc2 on the DEEP-SS test set, achieving a 10.8% improvement in Macro PR-AUC (0.561 vs. 0.453).
  • CNN-based Models: SCL2205 showed superior performance on the out-of-distribution DEEP-HPA set, while DeepLoc2 performed slightly better on DEEP-SS.
  • Statistical Significance: McNemar's tests and bootstrap confidence intervals confirmed that the performance gains of SCL2205 (especially with PLMs) were statistically significant.
• Leakage Findings: The homology augmentation experiment confirmed that even small-scale augmentation can reintroduce significant data overlap (4.8%), validating the need for post-augmentation overlap checks.

5. Significance

• Trustworthiness in AI Biology: By exposing and quantifying data leakage in homology augmentation, the paper calls for stricter evaluation standards in biological AI, preventing inflated performance claims.
• PLM Frontier: SCL2205 is specifically optimized for the emerging era of Protein Language Models (PLMs). Its preservation of full-length sequences allows PLMs to leverage terminal signals (N- and C-termini) that are often lost in truncated datasets.
• Sustainable Development: The dataset demonstrates that smaller, high-quality, and well-curated datasets can outperform larger, noisier ones, reducing the computational and environmental costs of training.
• Future Applications: SCL2205 provides a reliable benchmark for developing tools to identify molecular drivers of rare diseases and accelerate targeted therapy development by improving the accuracy of protein spatial mapping.