From variability to consensus: rescoring harmonizes peptide identification across diverse search engines and datasets

This study demonstrates that advanced peptide-spectrum match rescoring strategies significantly harmonize identification outcomes and reduce variability across diverse search engines and datasets, thereby enhancing the robustness and comparability of proteomics analyses while highlighting the continued importance of careful feature selection and database configuration for reliable false discovery rate control.

The Gist

Imagine you are trying to identify thousands of unique voices in a massive, noisy concert hall. This is essentially what scientists do in proteomics: they try to identify specific protein fragments (peptides) from the complex "noise" of a mass spectrometer.

For years, scientists have used different "listening devices" (called search engines) to figure out which voice belongs to which singer. The problem? Each device had its own way of judging the sound. One might say, "That's definitely the singer!" while another says, "Nah, probably just background noise." This made it hard to compare results between different labs or studies.

This paper is like a massive blind taste test (or in this case, a "blind listening test") designed to see if we can get all these different devices to agree on who is singing.

Here is the breakdown of their experiment using simple analogies:

1. The Setup: Seven Different Judges

The researchers took data from four different "concerts" (datasets) recorded on different types of microphones (mass spectrometers). They ran this data through seven different search engines (like Comet, MaxQuant, MSFragger, etc.).

• The Old Way (TDA): Before this study, each engine used a standard rulebook to decide if a match was real or a fake. It was like a judge saying, "If the voice sounds 90% like the singer, I'll accept it."
• The Problem: Because each engine had its own "ear," they often disagreed. One engine might find 10,000 singers, while another found only 5,000, even though they were listening to the same concert.

2. The Solution: The "Super-Referee" (Rescoring)

The researchers introduced a new step called Rescoring. Think of this as hiring a Super-Referee who listens to the initial guesses from all seven judges and then re-evaluates them using a much smarter, more detailed checklist.

They tested three types of Super-Referees:

• Percolator: The veteran referee who has been around for 20 years. Good, but uses a standard checklist.
• MS2Rescore & Oktoberfest: The "AI-powered" referees. These are new and use Machine Learning. They don't just listen to the voice; they predict what the singer's voice should sound like based on the lyrics (the protein sequence) and compare that prediction to the actual recording.

3. The Results: Harmony in the Chaos

The results were like magic.

• Before Rescoring: The judges were all over the place. If you asked Engine A, it found 10,000 singers. If you asked Engine B, it found 4,000. It was chaos.
• After Rescoring: The "AI-powered" referees (MS2Rescore and Oktoberfest) acted as a harmonizer. Suddenly, all seven engines started agreeing on almost the same list of singers. The gap between the "best" engine and the "worst" engine shrank dramatically.
  • Analogy: Imagine a group of people trying to guess the number of jellybeans in a jar. Without help, guesses range from 50 to 500. After they all use a smart calculator to measure the jar's volume, their guesses all converge around 210.

4. The "Trap" Test (FDR Control)

In science, you have to be careful not to count fake singers as real ones (False Discoveries). The researchers used a clever trick called Entrapment.

• The Analogy: They secretly planted a few "fake singers" (decoys) in the crowd that no one should be able to identify. If a judge starts claiming these fake singers are real, the referee knows the judge is making mistakes.
• The Finding: The new AI referees were generally very good at spotting the fakes. However, in a few specific, tricky situations, they were too confident, occasionally thinking a fake singer was real. This is a warning sign: "Don't trust the AI blindly; check its work!"

5. The Database Size Issue

They also tested if the size of the "songbook" (the protein database) mattered.

• Human Samples: It didn't matter much if the songbook was small or huge; the judges found the same singers.
• Metaproteomics (The "Wild" Samples): This was like trying to identify voices in a crowd of people from 100 different countries. Here, the size of the songbook mattered a lot. A bigger songbook helped the judges find more unique voices that were missed by the smaller book.

The Big Takeaway

The main message of this paper is Consensus.

For a long time, scientists worried that the choice of software (the search engine) would change their entire scientific conclusion. This study shows that if you use modern Rescoring tools (especially the AI-driven ones), it doesn't really matter which engine you start with. They all end up at the same destination.

In short:

• Old days: Different tools = Different answers.
• New days: Different tools + Smart Rescoring = The same, reliable answer.

This means scientists can now trust their data more, regardless of which software they use, as long as they run it through this "Super-Referee" step. It makes the whole field of proteomics more reliable, like tuning all the instruments in an orchestra so they play in perfect harmony.

Technical Summary

1. Problem Statement

Proteomics workflows rely on peptide-spectrum match (PSM) search engines to identify peptides from mass spectrometry data. While numerous search engines exist (e.g., X!Tandem, MS-GF+, MaxQuant), they often yield significantly different identification results when applied to the same dataset. This variability stems from differences in scoring algorithms, search strategies, and parameter handling.

Although rescoring methods (such as Percolator, MS2Rescore, and Oktoberfest) have become standard to improve identification accuracy by distinguishing correct from incorrect PSMs, there is a lack of systematic, large-scale comparisons. Existing studies often limit their scope to a single search engine or a single dataset. Consequently, it remains unclear how effectively modern rescoring strategies harmonize results across a broad spectrum of search engines, diverse mass spectrometry platforms, and varying protein database configurations (e.g., human vs. metaproteomic).

2. Methodology

The authors developed a comprehensive benchmarking framework to evaluate the impact of rescoring on identification consistency and accuracy.

• Pipeline Development (mspepid): A custom Nextflow pipeline was built to standardize the processing of data from Thermo Fisher (Orbitrap) and Bruker (TIMS-TOF) platforms. The pipeline handles data conversion (mzML), decoy generation (target-decoy approach), and the execution of search and rescoring tools in a reproducible, containerized environment.
• Search Engines Evaluated: Seven publicly available engines were tested:
  • X!Tandem, Andromeda (MaxQuant), Comet, MS Amanda, MS-GF+, MSFragger, and Sage.
• Rescoring Strategies: Three primary approaches were compared against standard Target-Decoy Approach (TDA) scoring:
  1. Percolator: Semi-supervised learning using TDA.
  2. MS2Rescore: Incorporates predicted fragment ion spectra (using tools like Prosit) as features.
  3. Oktoberfest: Similar to MS2Rescore but with specific implementation details; LDA scores were removed before passing to Percolator to ensure fair comparison.
• Datasets: Four distinct datasets were reanalyzed:
  • Human Datasets: Orbitrap and TIMS-TOF data from the Van Puyvelde benchmark and a Cancer Array dataset.
  • Metaproteomic Dataset: CAMPI benchmark (S07 run), representing a complex mixture of multiple species.
• Database Configurations:
  • Human: Swiss-Prot, Human Reference Proteome, and ProHap (containing variants).
  • Metaproteomic: Two databases of varying sizes (DB1 and DB2) from the CAMPI study.
• Evaluation Metrics:
  • Identification Consensus: Measured via PSM and peptidoform counts and overlap (UpSet plots).
  • FDR Control: Validated using standard TDA and Entrapment-based evaluation (using FDRBench) to assess the False Discovery Proportion (FDP) against the nominal FDR.
  • Resource Usage: Execution time, CPU, and memory consumption were recorded.

3. Key Contributions

• Systematic Benchmarking: The study provides the first large-scale comparison of seven search engines across four datasets and multiple database configurations, evaluating both standard TDA and three advanced rescoring methods.
• Harmonization of Results: It demonstrates that prediction-based rescoring (MS2Rescore, Oktoberfest) significantly reduces the variability in identification counts between different search engines, creating a "consensus" that was previously unattainable.
• Entrapment Validation: The authors applied entrapment databases to rigorously test FDR control, revealing that while most methods are robust, prediction-based rescoring can occasionally lead to slight FDR underestimation in specific configurations.
• Open Source Workflow: The release of the mspepid Nextflow pipeline (as an nf-core workflow) and associated analysis scripts to facilitate reproducible proteomics benchmarking.

4. Key Results

A. Reduction in Variability and Increased Consensus

• TDA-Only: When relying solely on standard TDA scoring, identification counts varied drastically between engines. For example, in the timsTOF dataset, X!Tandem produced almost zero valid PSMs compared to others, and MS-GF+ consistently outperformed MaxQuant.
• Rescoring Impact: Advanced rescoring methods (especially MS2Rescore and Oktoberfest) drastically reduced these discrepancies.
  • In the Cancer Array dataset, the relative difference in identifications between engines dropped from 67% (TDA) to ~8-9% (MS2Rescore/Oktoberfest).
  • "Rescoring" allowed underperforming engines (like X!Tandem) to recover identifications, bringing them in line with top performers.
• Peptidoform Level: These trends held true when aggregating PSMs into peptidoforms (unique amino acid sequences + modifications).

B. Database Size and Composition

• Human Datasets: Database size (Swiss-Prot vs. Reference Proteome vs. ProHap) had minimal impact on the number of identifications for human samples. Swiss-Prot with isoforms appeared sufficient for general applications.
• Metaproteomic Dataset (CAMPI): Database size was critical. The larger database (DB1) yielded 22–31% more identifications than the smaller one (DB2). This highlights that in complex metaproteomics, smaller databases may miss significant portions of the sample's proteome.

C. FDR Control and Entrapment Analysis

• General Validity: Most combinations satisfied the assumption that decoys model incorrect target PSMs.
• Feature Sensitivity: In specific cases (e.g., MS-GF+ on timsTOF), including all features caused "over-separation" of targets and decoys, violating FDR assumptions. Removing specific features (e.g., negative log E-value) resolved this.
• Underestimation Risk: Prediction-based methods (MS2Rescore, Oktoberfest) showed a slightly higher tendency to underestimate the true FDR compared to TDA-only or Percolator, particularly at low FDR thresholds. This suggests a need for careful feature selection.
• Entrapment Limits: For the CAMPI dataset, entrapment analysis was difficult due to high-scoring entrapment hits, though overlap analysis suggested the results remained consistent with other engines.

D. Computational Resources

• Speed: Sage and MSFragger were the fastest engines. MS-GF+ was the slowest, often requiring hours even with parallelization strategies (splitting databases/files).
• Memory: MaxQuant and MS Amanda required significant memory (up to ~745 GB for MaxQuant in some configurations). MS-GF+ and Sage were more memory-efficient but still substantial for large databases.
• Scalability: Comet and Sage scaled efficiently with CPU cores; MaxQuant showed limited per-run scalability.

5. Significance and Conclusion

This study establishes that prediction-based rescoring is a critical step for harmonizing proteomics workflows. While search engines differ in their initial scoring power, the incorporation of predicted spectral features (via MS2Rescore or Oktoberfest) allows them to converge on a consistent set of peptide identifications.

Key Takeaways for Practitioners:

1. Rescoring is Essential: It not only boosts the total number of identifications but also minimizes the "engine bias," making results more comparable across different labs and pipelines.
2. Database Selection Matters: For metaproteomics, database size is a limiting factor for coverage; for human proteomics, standard databases are often sufficient.
3. Validation is Non-Negotiable: Despite the gains in consensus, researchers must still validate FDR control (e.g., via entrapment or checking target-decoy distributions), as advanced rescoring can sometimes lead to optimistic FDR estimates.
4. Algorithm Choice: The choice of search engine may now depend more on computational resources and workflow integration (e.g., speed vs. memory) rather than raw identification performance, as rescoring largely equalizes the output quality.

The authors conclude that the primary limitation in peptide identification is no longer the generation of candidate PSMs, but the discriminatory power of the initial scoring functions, which are effectively overcome by modern, feature-rich rescoring models.