Scoring the downstream chain of small-n label-free proteomics against known ratios: search engine, match-between-runs, imputation, differential testing and enrichment background on a public three-species benchmark

In the microscopic world inside our cells, thousands of proteins work together to keep life running. To understand how these proteins change when a person gets sick or when a treatment works, scientists use a technique called label-free proteomics. Imagine trying to weigh a crowd of people by taking a snapshot of them; the camera captures everyone, but to know how much each person weighs, you have to measure the brightness of their image. In this scientific method, the "brightness" of a protein's signal tells researchers how much of it is present. By comparing these signals between two different conditions—say, healthy tissue versus diseased tissue—scientists can see which proteins have increased or decreased. However, getting an accurate count is difficult. The process involves complex computer programs to identify the proteins, and then a long chain of decisions about how to handle missing data, how to compare the samples, and how to decide what counts as a real change versus a random fluctuation. If these later steps are handled poorly, the final list of important proteins can be filled with errors, leading researchers down the wrong path.

A researcher recently took a public set of data, where the true answers were already known, to test exactly how these different choices affect the final result. The data came from a clever experiment where scientists mixed proteins from three different species: humans, yeast, and a common gut bacterium called E. coli. They created two mixtures with known proportions. In the first mixture, the human proteins made up 65 percent, yeast 30 percent, and bacteria 5 percent. In the second mixture, they kept the human proteins the same but cut the yeast in half and quadrupled the bacteria. Because the starting amounts were known, the researchers knew exactly what the computer should find: the human proteins should show no change, the yeast should appear to be half as much, and the bacteria should appear to be four times as much. This setup acted as a perfect test ground, a controlled environment where the correct answer was written in the stars, waiting to be seen if the tools were sharp enough.

The researcher ran this data through three of the most popular software tools used in the field today: MaxQuant, FragPipe, and DIA-NN. These programs are like different brands of high-end cameras; they all take the raw pictures and try to produce a clear image of the proteins. The study found that while all three tools were generally good at finding the proteins, they differed in how accurately they measured the changes. When looking at the human proteins, which should not have changed at all, one tool called MaxQuant showed a small, consistent shift, making the unchanged proteins look slightly different than they were. Another tool, FragPipe, was very precise with these unchanged proteins, while the third, DIA-NN, was also very accurate but used a different type of machine to capture the data in the first place. The most striking finding was not about which software was best, but how much the final result could be ruined by simple decisions made after the software finished its work.

After the software identified the proteins, the researcher tested a series of common steps that scientists take to clean up the data. One such step is "match-between-runs," a feature that allows the computer to fill in missing numbers for a protein in one sample by borrowing the value from a similar sample. The study showed that while this feature helped find more proteins, it also introduced more noise, especially for the rare bacterial proteins that were hard to detect. Another critical decision was how to handle proteins that were missing from the data entirely. The researcher tested several ways to guess these missing values. One common method, which assumes the missing data is random, completely failed when the data was missing because the protein was simply too scarce to be seen. In that specific case, the method guessed the missing values were average, effectively erasing the massive four-fold increase in bacteria that was actually there.

The most dramatic changes came from how the data was tested for significance. Scientists often use a statistical test to decide if a change is real. The study found that if a researcher simply asked the computer to find any protein that looked different, without requiring a minimum size for that difference, the number of false alarms exploded. On one of the software outputs, a standard test flagged over a hundred human proteins as having changed, even though they were known to be stable. However, when the researcher added a simple rule requiring the change to be at least a certain size before calling it significant, the number of false alarms dropped from over a hundred to just a handful. This single decision reduced the error rate by more than thirty times. Conversely, adding an extra step to "normalize" or re-balance the data, which is common practice, actually made things worse for two of the software tools, creating thousands of new false alarms where none existed before.

The study also looked at how scientists interpret these lists of proteins to understand biological processes, a step called enrichment analysis. When researchers take a list of proteins and ask a database what functions they are involved in, the choice of background matters immensely. If they compare their list against the entire genome of all possible proteins, the computer finds hundreds of "significant" functions, even when the list is just a random collection of proteins. But if they compare the list only against the proteins that were actually detected in the experiment, the false alarms disappear. This suggests that the background used for comparison acts like a lens; looking through the wrong lens makes random noise look like a signal.

The ultimate conclusion of this work is that the choice of software engine is less important than the choices made after the software runs. A single decision, such as adding a fold-change threshold or avoiding an extra normalization step, could change the number of false discoveries by up to thirty-four times. The researcher provided a checklist for others to follow: use strict rules for which data to keep, avoid guessing missing values unless you understand why they are missing, pair statistical tests with a requirement for a minimum change size, and use the detected proteins as the background for enrichment analysis. By following these simple safeguards, scientists can ensure that the stories they tell about their data are based on real biological changes rather than artifacts of their own methods. The study did not claim to have solved every problem in the field, but it clearly showed that with known answers, the path to error is often paved with well-intentioned but unnecessary steps.