Training-seed variance in single-run comparisons of frozen-CLIP adaptation: evidence from cross-dataset facial expression recognition

In the world of artificial intelligence, computers are increasingly taught to recognize the world by studying vast libraries of images and text together. This process creates a powerful, general-purpose brain that can understand pictures it has never seen before. However, when these systems are deployed in the real world, they often stumble. A model trained on one set of photos might fail miserably when shown pictures taken under different lighting, by different cameras, or featuring different people. This is a common problem in facial expression recognition, where a computer trained to spot happiness or anger in one dataset often loses its accuracy when faced with a new collection of faces. To fix this without retraining the entire massive brain, researchers have developed "adaptation" techniques. These methods keep the main brain frozen and unchanged, adding only a small, trainable layer on top to adjust to the new situation. It is a practical solution, but it relies on a dangerous assumption: that if a researcher runs this adjustment process once and gets a result, that result is the truth.

A team of researchers at Hefei University of Technology set out to test whether this single-run assumption holds water. They focused on two competing ways to adapt these frozen AI brains for facial expression recognition. One method, called the "global route," takes a single summary of the entire image and tries to tweak that summary to fit the new task. The other, known as the "token route," works differently; instead of summarizing the image first, it looks at the individual pieces of the image and learns how to weigh them before making a decision. For years, the token route has been assumed to be superior because it preserves more detail. To test this, the researchers took a standard frozen AI model and applied both adaptation methods. But instead of running the experiment once and reporting the winner, they ran it many times, changing only the random starting point of the training process, known as the "seed." They did this for two different sizes of AI models, running the experiment ten times on the smaller model and fifteen times on the larger one, always using the exact same data and rules.

The results of this rigorous re-testing were startling. When the researchers looked at the data from a single run, the token route appeared to be significantly better than the global method on a cross-dataset test, leading by nearly four percentage points. This is the kind of headline result that usually defines a field. However, when they averaged the results across all ten or fifteen different training runs, that lead vanished. In fact, for the smaller AI model, the advantage flipped entirely; the token route actually performed worse than the global method on average. The difference between the best and worst runs was so large that it completely overshadowed the original claim. The researchers found that the random starting point of the training process was enough to reverse the conclusion of the study. Five out of the six major comparisons they made showed this instability, meaning that a single experiment could easily report a "winner" that would lose if the experiment were simply repeated with a different random start.

There was, however, one exception that survived the test of repetition. On a specific test involving images degraded by synthetic distortions like blur or noise, the token route consistently outperformed the global method on the larger AI model. This advantage held true across all fifteen training runs, with the token route leading by about three and a half percentage points. To be absolutely certain this wasn't a fluke, the researchers took this finding to a completely new dataset of facial expressions that they had never touched before. They applied the same strict rules and pre-defined criteria. The token route won again, leading by nearly three points. This confirmed that while the token route is not a universal magic bullet that beats everything in every situation, it does possess a genuine, reproducible strength when dealing with specific types of image corruption on larger models.

The study also uncovered that the way researchers tune their experiments matters just as much as the random starting point. When the researchers allowed the learning rate—a key setting that controls how fast the model learns—to be chosen differently for each random run, the results shifted dramatically. In one case, changing this setting moved the performance difference by more than three points, which was the exact size of the effect the original single run had claimed to find. This suggests that the "winner" of a comparison often depends as much on the specific tuning choices as on the method itself. The researchers concluded that the field has been relying too heavily on single-run comparisons, which can produce misleading directional claims. They demonstrated that for many adaptation methods, the noise introduced by random training variations is large enough to flip the results. The only way to know if a method is truly better is to run it many times and report the spread of results, rather than just the single best outcome.

This work does not declare that one adaptation method is permanently superior to the other. Instead, it reveals that the landscape of AI adaptation is far more unstable than previously thought. The token route is not a guaranteed victory, nor is the global method a failure; their performance is highly sensitive to the random seeds used during training and the specific conditions of the test. The one clear, reproducible finding is that the token route offers a real advantage for larger models when images are corrupted, but this strength does not extend to all tasks or all model sizes. The study serves as a necessary correction to how these technologies are evaluated, urging researchers to move beyond single snapshots of performance and embrace the full picture of variability. By doing so, the scientific community can distinguish between fleeting statistical noise and genuine, reliable improvements in how machines understand human faces.