Pseudoperplexity Probes Memorization in Protein Language Models

Imagine Protein Language Models (pLMs) like ProtT5 as super-smart students who have read almost every biology textbook in existence. These models are amazing at predicting what comes next in a protein sequence, much like how a language model predicts the next word in a sentence.

But there's a big worry: Did these students actually learn the rules of grammar (how proteins are built), or did they just memorize the specific sentences they read in their textbooks? If they just memorized the books, they might fail when asked to write about a topic they've never seen before.

The Experiment: The "Fake" vs. "Real" Test

To find out if ProtT5 was just reciting its homework or truly understanding the material, the researchers set up a test using a concept called pseudoperplexity. Think of this as a "surprise meter."

• If the model is surprised by a sequence, it means it hasn't seen it before (low memorization).
• If the model isn't surprised at all, it might mean it has seen that exact sequence before (high memorization).

The Setup
The researchers gave ProtT5 two types of protein sequences:

1. The "Seen" List: Sequences that were part of the model's original training data (like a student's old homework).
2. The "Unseen" List: Brand new, genuinely novel sequences that the model had never encountered (like a brand new exam question).

To make sure the test was fair, they matched these lists perfectly. They made sure the "seen" and "unseen" proteins were the same length, came from similar families of organisms, and had similar levels of complexity. It was like ensuring the student was tested on two essays of the same length and difficulty, just written by different people.

The Baseline Check
Before trusting the results, the researchers checked if the "Unseen" list was actually new. They used simple, old-school statistical tools (n-gram models) to analyze the patterns. They confirmed that the "Unseen" sequences were indeed different enough from the training data to be considered new, not just slightly tweaked versions of old ones.

The Results
When they ran the "surprise meter" (pseudoperplexity) on ProtT5:

• The model did react differently to the "Seen" sequences compared to the "Unseen" ones. It was less surprised by the ones it had studied before.
• This proved that ProtT5 does have some memory of its training data.

The Conclusion
However, the difference wasn't huge. It wasn't like the model was reciting the textbook word-for-word. Instead, the "memorization signal" was modest.

In simple terms: ProtT5 isn't just a parrot repeating its training data, but it's not a perfect genius that has never seen a specific sentence before either. It has a detectable but limited memory of what it studied, while still relying mostly on the general rules of protein grammar to handle new information.