💻 computer science

FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs

FedLNS is a server-side framework that mitigates adversarial manipulation in federated LLM training by screening malicious client updates through lightweight analysis of normalization-layer parameter changes, achieving superior robustness across diverse model architectures without requiring additional data exchange or labeled attack examples.

Original authors: Kai Li, Jong-Ik Park, Carlee Joe-Wong, Wei Ni, Falko Dressler

Published 2026-08-20

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Kai Li, Jong-Ik Park, Carlee Joe-Wong, Wei Ni, Falko Dressler

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind the smart assistants, search tools, and writing aids we use every day. These systems learn by reading vast amounts of text, but much of the most valuable information is locked away in private places: personal messages, medical records, or proprietary company documents. To teach a model using this private data without ever seeing the data itself, researchers use a method called federated learning. In this setup, the model travels to the data rather than the other way around. The central computer sends a copy of the current model to many different devices, each device trains on its own private information, and then sends back only the changes it learned. The central computer then combines these changes to create a smarter, updated model. This process protects privacy, but it creates a new vulnerability: the central computer cannot see what happened on the local devices. A dishonest device could pretend to learn from private data while actually being fed false instructions, tricking the central computer into adopting harmful lessons that degrade the model's ability to speak or write correctly.

A team of researchers has developed a new way to spot these dishonest updates before they can damage the global model. They call their system FEDLNS, a method that acts like a security checkpoint for the information flowing back to the central server. Instead of trying to inspect every single part of the massive model updates—which can contain billions of numbers and are too complex to check thoroughly—the researchers focus on a tiny, specific set of numbers within the model. These numbers control how the model scales and shifts its internal understanding of language, acting as a kind of signature for how the model was adjusted. By watching how these specific numbers change, the server can build a profile of what normal, honest learning looks like. When a device sends back an update, the server compares its signature against a history of signatures from other devices. If a device's changes look too different from the crowd, the system flags it as suspicious and sets it aside, preventing the bad update from being mixed into the final model.

The researchers tested this approach on three different types of language models, ranging from those that predict the next word in a sentence to those that fill in missing words, and finally to large models similar to the ones powering modern chatbots. They simulated a scenario where up to forty percent of the participating devices were malicious, deliberately trying to corrupt the model by teaching it incorrect associations between words. In these difficult conditions, the new system successfully filtered out the harmful updates. The models trained with this protection performed significantly better than those trained with existing safety methods. Specifically, the protected models made fewer mistakes when predicting the next word, showing a reduction in confusion of up to eighteen percent compared to the next best method. They also showed less uncertainty in their choices, meaning they were less likely to produce confusing or nonsensical sentences.

What makes this discovery particularly useful is that it does not require the central server to have a list of known bad actors, nor does it need a separate, pre-trained detector to catch the liars. The system learns what is normal simply by observing the group of devices over time. It waits until it has seen enough different devices to build a reliable picture of honest behavior, and then it begins screening. The researchers found that the more devices the system observed before starting the screening, the better it performed, though it could still work effectively with a partial view of the group. Crucially, this protection happens entirely on the server side; the devices sending the updates do not need to change their software or send any extra information, making the method easy to add to existing systems without slowing them down.

The study confirms that it is possible to secure federated learning without sacrificing privacy or requiring trusted data. By focusing on a compact, architecture-aware signature rather than the entire model, the researchers created a lightweight shield that stops malicious manipulation before it spreads. In their simulations, this approach consistently outperformed six other common safety techniques across different data distributions and model types. The results suggest that by monitoring the subtle internal shifts of a model, we can maintain the integrity of collaborative learning even when a large portion of the participants are trying to deceive the system. This offers a practical path forward for building more reliable artificial intelligence that can learn from private data without being easily tricked by bad actors.

Technical Summary: FedLNS – Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs

Problem Statement

Federated Learning (FL) enables Large Language Models (LLMs) to learn from distributed private data without centralizing raw text. However, this architecture creates a critical security gap: the server cannot verify the local inputs, targets, or optimization processes that generate client updates. A malicious client can manipulate its local supervision by replacing correct target tokens with incorrect ones (target manipulation), thereby optimizing false context–token associations. When these corrupted updates are aggregated, they degrade the global model's performance and increase the risk of unreliable or hallucinatory generation.

Existing Byzantine-robust FL methods (e.g., Multi-Krum, Coordinate-wise Median, Trimmed Mean) typically operate on full-model updates. This approach ties server-side processing to the high dimensionality of modern LLMs, where benign variations in massive parameter sets can obscure malicious deviations. Furthermore, recent dimension-reduction or supervised screening methods often require labeled attack samples, trusted server data, or separately trained detectors, which are often unavailable in practical federated settings.

Methodology: FedLNS

The authors propose FedLNS (Federated Learning with Normalization Signatures), a server-side framework designed for lightweight, unsupervised malicious-update screening. The core innovation lies in representing client updates not by their full parameter vectors, but by compact "signatures" derived from changes in trainable normalization-layer parameters.

1. Normalization-Layer Signatures:
Instead of analyzing millions of parameters, FedLNS extracts a signature $s_{t,i}$ for each client $i$ at round $t$ . This signature is defined as the difference in trainable normalization parameters (e.g., scaling and bias in LayerNorm or RMSNorm) between the client's local model and the current global model:
$s_{t,i,\ell} = p_\ell(w_{t,i}) - p_\ell(w_t)$
where $p_\ell$ represents the concatenated trainable parameters of normalization layer $\ell$ . These layers control the scaling and shifting of hidden representations, making their parameter changes a compact, architecture-aware indicator of how local training alters the model's internal representations.

2. Server-Side History Bank and Robust Reference:
The server maintains a history bank $B_t$ storing the latest normalization signatures for observed clients. Once a sufficient number of distinct client identities ( $M_{act}$ ) have been observed, the server constructs a robust reference for the current round:

Robust Statistics: For each normalization layer, the server computes the coordinate-wise median ( $m_{t,\ell}$ ) and Median Absolute Deviation ( $d_{t,\ell}$ ) of the signatures in the history bank. This provides a robust estimate of the population center and scale, resistant to outliers.
Standardization: Current client signatures are standardized against this reference to produce feature vectors $z_{t,i}$ .
Deviation Score: A scalar deviation score $\delta_{t,i}$ is computed as the median absolute standardized deviation across the feature vector.

3. Unsupervised Screening via BIC-Guided GMM:
FedLNS employs a Bayesian Information Criterion (BIC) to guide a Gaussian Mixture Model (GMM) screening rule. The server fits both one-component and two-component diagonal GMMs to the standardized client signatures:

If the one-component model is preferred (indicating a homogeneous population), all clients are retained.
If the two-component model is preferred, clients are partitioned into two groups. The group with the lower median deviation score (closer to the historical reference) is retained for aggregation, while the high-deviation group is discarded.
This process requires no prior knowledge of the malicious client fraction, no labeled data, and no trusted server dataset.

4. Aggregation:
After screening, the retained full-model updates are aggregated using standard sample-weighted rules (e.g., FedAvg) or other compatible FL aggregation methods. The screening adds no communication overhead, as signatures are extracted server-side from the returned models.

Key Contributions

Compact Model Update Representation: Introduction of a normalization-layer update signature that reduces the screening representation from full-model dimensionality to a small, architecture-aware subset of parameters.
Unsupervised History-Aware Screening: Development of a server-side retention rule that constructs a robust cross-client reference from observed signatures, combining median/MAD standardization with BIC-guided mixture modeling. This eliminates the need for labeled attacks, trusted data, or auxiliary detectors.
Cross-Architecture Evaluation: Comprehensive evaluation across GPT-style, BERT-style, and LLaMA-style models trained from scratch with 200 clients under both IID and non-IID data partitions.

Experimental Results

The authors evaluated FedLNS against six baselines (FedAvg, Norm-Bound, Median, Trimmed Mean, Multi-Krum, FLAME) under target manipulation affecting up to 40% of the client population.

Perplexity Reduction: Under 40% malicious population, FedLNS achieved the lowest test perplexity across all three architectures and both data partition types.
- IID: Reductions of 11.59% (GPT), 3.88% (BERT), and 4.98% (LLaMA) compared to the strongest baseline.
- Non-IID: Reductions of 18.57% (GPT), 10.05% (BERT), and 5.67% (LLaMA) compared to the strongest baseline.
Semantic Entropy: FedLNS also achieved the lowest token-level semantic entropy, indicating reduced predictive uncertainty. Maximum reductions reached 6.50% in the non-IID BERT setting.
Robustness to Attack Intensity: As the malicious fraction increased from 0% to 40%, FedLNS showed limited degradation in perplexity (2.82% increase) compared to significant increases in baselines (e.g., FedAvg increased by 27.97%).
Bank Coverage: Experiments showed that increasing the activation coverage (the number of clients required before screening begins) generally improved early training performance, with monotonic perplexity improvements observed as coverage increased.

Significance and Claims

The paper claims that FedLNS addresses the scalability and data-dependency limitations of existing FL defenses for LLMs. By leveraging normalization-layer signatures, the method provides a modular, lightweight, and unsupervised screening mechanism that does not rely on:

Raw client data or trusted server datasets.
Labeled malicious updates.
Predefined malicious client fractions.
Separately trained detectors or auxiliary global models.

The authors position FedLNS as a practical solution for improving global-model robustness and reducing predictive uncertainty in federated LLM training, specifically mitigating the downstream risks of semantically inconsistent or hallucinatory generation caused by corrupted training signals. The theoretical analysis supports the method by bounding the screening error and characterizing convergence under screened aggregation, demonstrating that the retained updates approximate the benign global gradient direction.