How Much Regularization Survives Averaging? Update Masking in Federated Learning

In the world of artificial intelligence, there is a persistent challenge known as the "federated" problem. Imagine a group of people, each holding a unique piece of a puzzle, who want to build a single, complete picture together without ever showing their individual pieces to anyone else. This is how modern machine learning often works: a central computer coordinates many different devices, like phones or sensors, to learn a shared skill. The catch is that the data on each device is rarely the same; one phone might have mostly photos of cats, while another has only pictures of cars. When the central computer tries to combine what everyone has learned, the resulting model often struggles to make sense of the whole picture, failing to generalize well to new situations. To fix this, researchers have long sought a way to find "flat" solutions in the mathematical landscape of learning. Think of a sharp peak versus a wide, flat plateau. A model that lands in a sharp peak might work perfectly on the specific data it saw, but it will stumble immediately if the data changes even slightly. A model that settles on a wide, flat plateau is more robust; it can handle small variations in the data without falling apart.

For years, scientists have tried to force these models to find those flat plateaus by adding noise or making small, deliberate mistakes during the learning process. This noise acts like a gentle shaker, preventing the model from getting stuck in a narrow, fragile spot. Recently, a specific technique called "update masking" became popular in centralized training, where a single computer does all the work. This method randomly discards chunks of the learning instructions and rescales the rest, effectively adding a helpful kind of noise that pushes the model toward those stable, flat areas. However, when researchers tried to bring this technique into the federated setting, where many different devices learn separately and then combine their results, it seemed to vanish. The question that a team of researchers from Sophia University and Shendian Energy Co., Ltd. set out to answer was simple: where did that helpful noise go, and could it be recovered?

The researchers discovered that the noise did not disappear; it was simply diluted by the very act of combining the results. In their setup, they had one hundred different devices, or "clients," each learning on its own slice of data. In a standard approach, each device would randomly decide which parts of its learning instructions to keep and which to discard, using its own unique pattern of choices. When the central server collected these updates and averaged them together, the random choices of the individual devices canceled each other out. It was as if ten people were trying to push a heavy object in slightly different random directions; the net result was that the object barely moved. The mathematical penalty that usually forces a model to be robust was weakened by a factor equal to the number of devices in the group. With ten devices, the helpful effect was reduced to one-tenth of its original strength, leaving the model with almost no protection against overfitting.

The team then tested a different strategy: what if every device used the exact same pattern of choices? If all ten people pushed the object in the same random direction, the effect would be preserved. The researchers found that synchronizing these choices did indeed restore the protective noise, but with a major caveat. The strength of the restored noise depended entirely on how much the devices agreed with each other. If the devices were all learning from very different data and their updates pointed in conflicting directions, the synchronized noise became ineffective or even harmful. The measure of this agreement is called "gradient diversity," a concept that essentially counts how much the individual efforts of the devices overlap. When the devices are in harmony, the noise returns with full force. When they are in conflict, the noise is diminished or lost entirely.

To understand why this happened in practice, the researchers ran extensive experiments using a standard image dataset called CIFAR-10, partitioned across one hundred simulated clients. They measured exactly how much of the protective noise survived the averaging process under different conditions. They found that in the most common setup, where devices learn in small batches of data, the survival rate was shockingly low. Out of a possible maximum strength of ten, the noise that actually reached the final model was only about 1.19. This tiny fraction meant that the model was barely any more robust than if no noise had been added at all. The researchers traced this failure to the random sampling of data that happens in every step of standard learning. The noise introduced by picking small batches of images overwhelmed the specific, helpful noise generated by the masking technique, drowning it out.

The team also investigated whether the differences in the data itself—the fact that some devices had more cats and others had more cars—was the culprit. They varied the data distribution to make it extremely uneven, with some devices holding a hundred times more data than others. Surprisingly, this extreme difference had very little effect on the survival rate of the noise. Whether the data was nearly identical or wildly different, the survival rate remained stuck between 1.17 and 1.50. The true barrier was not the diversity of the data, but the method of learning. When the researchers turned off the small-batch sampling and let each device learn from its entire collection of data in one go, the survival rate jumped dramatically to 8.96. This proved that the random noise of small batches was the primary reason the technique failed in federated settings.

However, the study concluded with a sobering reality check. While it is mathematically possible to recover the full strength of the protective noise by using large, full-data batches and synchronized choices, doing so comes at a terrible cost to the model's actual performance. The configurations that allowed the noise to survive were the same ones that produced the worst learning results, with test accuracy dropping significantly. In the experiments, the best-performing models, which used small batches, had almost no protective noise left, while the models with the most noise were too inaccurate to be useful. The researchers found no middle ground where the noise could be preserved without sacrificing the model's ability to learn.

Ultimately, the paper reveals that the failure of update masking in federated learning is not an oversight or a bug, but a fundamental consequence of how the system works. The very mechanism that allows many devices to learn together—averaging their updates—also washes away the specific kind of noise that update masking relies on. The researchers showed that while it is possible to force the noise to survive by synchronizing the devices, the conditions required to do so are incompatible with the practical needs of training a useful model. The protective effect that works so well in a single computer simply does not translate to a network of many, unless one is willing to accept a model that learns very poorly. The study leaves the field with a clear understanding of the limits of this technique, suggesting that future solutions must look elsewhere for ways to help federated models find those stable, flat plateaus.