🤖 machine learning

Output Dilution: Redundant but Fragile Representations in MoE Models

This paper reveals that while Mixture-of-Experts (MoE) models encode moral content as redundantly as dense models, their representations are significantly more fragile due to "output dilution," where the averaging of active experts drastically reduces signal strength, making the encoded information easily overwhelmed by noise despite stable routing.

Original authors: Orion Reblitz-Richardson

Published 2026-08-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Orion Reblitz-Richardson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving field of artificial intelligence, researchers are constantly trying to understand how large language models think. These systems, which can write stories, solve problems, and answer questions, are built from vast networks of mathematical connections. A popular design for these networks is called a "mixture of experts." Imagine a large team of specialists where, for every single sentence the computer processes, only a small group of them is allowed to work. The rest remain silent. This approach is designed to make the computer faster and more efficient by using fewer resources for each task. For scientists studying how to align these models with human values—ensuring they behave ethically and safely—this structure offers a tantalizing possibility. If moral reasoning is handled by a specific, isolated group of these specialists, researchers could potentially tweak or remove just that group to fix bad behavior without disturbing the rest of the system. It is a clean, surgical approach to a messy problem.

However, a new study using a specific open-source model called OLMoE has found that this hope is misplaced. The researchers set out to see if these "expert" modules actually specialize in moral reasoning, or if they all do the same thing. They also wanted to test how strong these moral signals really are. The results were surprising. The study found that the model does not have a dedicated team of moral experts. Instead, every single specialist module, across every layer of the network, contains the same moral information. More importantly, the study discovered that while this moral information is present, it is incredibly fragile. It exists in the system, but it is so weakly expressed that even a tiny amount of digital noise can wipe it out, whereas a standard model of the same size would easily withstand the same disturbance.

The investigation began by examining the internal workings of the OLMoE model, which contains 64 of these specialist modules in each of its 16 layers. The researchers trained simple detectors to see if they could find moral concepts within the output of each individual module. If the "mixture of experts" design worked as hoped, they expected to find that only a few specific modules were good at recognizing moral content, while the others focused on different tasks like grammar or facts. Instead, they found the opposite. Nearly every single module, regardless of its position in the network, was able to identify moral content with high accuracy. The distribution of this ability was perfectly even; no module was a specialist, and no module was a novice. The router that decides which modules get to work also showed no preference for sending moral sentences to specific experts. It treated all inputs the same way. This means that the structural advantage of having separate experts does not help in isolating moral features for targeted intervention.

The researchers then turned their attention to a more subtle question: how robust is this moral information? They compared the OLMoE model to a standard, "dense" model that does not use the expert system but has a similar number of active parameters. When they tested how much noise the models could tolerate before losing their ability to recognize moral content, a stark difference emerged. The standard model was remarkably sturdy, able to withstand significant levels of digital interference while still holding onto its moral understanding. The OLMoE model, by contrast, collapsed almost immediately. It was more than four times more fragile than its standard counterpart. The moral signal in the expert model was so delicate that a tiny amount of static noise was enough to drown it out completely.

To understand why this happened, the team looked at how the information flows through the system. In a standard model, the processing units send a strong, clear signal to the next stage of the network. In the expert model, the system selects a few experts, averages their outputs, and sends that combined result forward. The researchers measured the strength of this combined signal and found it was drastically weaker than in the standard model. The signal was diluted, becoming nearly two orders of magnitude smaller. One way to visualize this is to imagine a conversation in a crowded room. In the standard model, a person speaks with a clear, strong voice that everyone hears. In the expert model, it is as if eight people whisper the same sentence, and the room only hears the average of those whispers. The message is there, but it is so quiet that a slight breeze of noise can make it impossible to hear.

This finding has profound implications for how we understand and control these systems. The study showed that this fragility is not a result of the model failing to learn properly over time. By looking at the model's training history from its very first steps to its final form, the researchers confirmed that the moral information was present from the beginning and remained uniformly distributed and weak throughout the entire process. The model never developed specialized moral experts, nor did it ever strengthen the signal. The fragility is a built-in feature of the architecture itself. Because the system averages the outputs of a few selected experts, the resulting signal is inherently small. This small signal is easily overwhelmed by noise, making the moral encoding in these models much less secure than in traditional designs.

The study concludes that the promise of the mixture-of-experts design for alignment research is an illusion. While the structure offers a way to partition the network into discrete units, it does not create isolated pockets of moral reasoning that can be easily edited or removed. Instead, it spreads the moral information thinly across all units, making it redundant but incredibly vulnerable. For researchers trying to ensure these models behave safely, this means that simply looking at whether a model can identify moral concepts is not enough. They must also measure how securely that information is held. A model might appear to understand ethics perfectly on the surface, yet be so fragile that a minor change in its environment or a slight adjustment during fine-tuning could cause it to lose that understanding entirely. The study suggests that the way these models are built fundamentally changes the nature of their internal signals, creating a permanent structural weakness that training alone cannot fix.

Technical Summary: Output Dilution: Redundant but Fragile Representations in MoE Models

Problem Statement
Mixture-of-Experts (MoE) architectures partition representation space into discrete, inspectable expert modules, theoretically offering unique intervention points for alignment research (e.g., pruning or fine-tuning specific "moral" experts). However, it remains unclear whether moral features actually concentrate within specific experts or if MoE models merely replicate the uniform, redundant encoding patterns observed in dense models. Furthermore, while MoE models often match dense models in performance, their robustness to perturbations—a critical metric for alignment safety—has not been systematically compared. This paper investigates whether MoE architectures induce expert-level moral specialization and how their structural sparsity affects the robustness of encoded moral content.

Methodology
The study conducts a controlled architectural comparison between OLMoE-1B-7B (a 64-expert, top-8 MoE model with 1.3B active parameters) and OLMo-2 1B (a dense transformer with comparable active parameters and hidden dimension). Both models are base (non-instruct) checkpoints from the same research lab, minimizing confounds from training data or philosophy.

The methodology employs three core components:

Per-Expert Probing: Unlike standard layer-wise probing, the author bypasses the router to compute all 64 expert outputs in parallel for each token. They train 1,024 independent binary linear classifiers (64 experts × 16 layers) to decode moral valence based on Moral Foundations Theory.
Fragility Testing: Gaussian noise is injected into model representations at varying magnitudes ( $\sigma$ ). The "critical noise" ( $\sigma^*$ ) is defined as the smallest noise level at which probe accuracy drops below 0.6. This protocol is applied to the full hidden state, as well as specific MoE components (router logits, individual expert outputs, and the aggregated MoE output).
Trajectory Analysis: The author analyzes 11 checkpoints spanning OLMoE's training (from 20B to 5,033B tokens) to determine if expert specialization emerges over time. They track the Gini coefficient of expert accuracy and the stability of the top-5 experts across checkpoints.
Output Scale Measurement: The standard deviation of feedforward block outputs is measured to quantify the "scale" of the signal injected into the residual stream.

Key Findings

No Expert Moral Specialization: Contrary to the hypothesis that MoE models partition moral features into specific experts, moral encoding is uniformly distributed. All 64 experts at every layer decode moral content with high accuracy (mean peak accuracy >90%, Gini coefficient <0.03). The router shows negligible preference for routing moral tokens to specific experts (max preference 1.8%).
Significant Fragility Gap: Despite achieving identical peak probing accuracy (99.0%) to the dense OLMo-2 1B model, the MoE model is 4.2× more fragile. The mean critical noise for OLMoE is $\sigma^* = 0.92$ , compared to $\sigma^* = 3.81$ for the dense model.
Output Dilution as the Mechanism: The fragility gap is caused by output dilution. The MoE block's aggregated output (a weighted average of 8 experts) contributes to the residual stream at a scale 74× smaller than the dense MLP output. While the information content is preserved (allowing high probe accuracy), the signal amplitude is so small that it is trivially overwhelmed by noise that a dense model easily tolerates.
Component Robustness: The router itself is highly robust ( $\sigma^* \approx 9.56$ ), and individual expert outputs are moderately robust. The fragility originates entirely in the aggregated output, where the signal scale is diluted.
Specialization Never Emerges: Analysis of training trajectories reveals that moral encoding is present from the earliest checkpoint (Step 5K) and remains uniformly distributed throughout training. The Gini coefficient remains flat near zero, and the identity of the "best" experts fluctuates randomly. Specialization is not a late-training convergence; it never exists.

Key Contributions

First Expert-Level Moral Probing: This work provides the first analysis of moral feature distribution at the individual expert level in an MoE language model, demonstrating that structural partitioning does not lead to functional specialization for moral content.
Quantification of Output Dilution: The paper identifies and quantifies "output dilution" as a structural consequence of sparse aggregation, showing that the MoE block injects a signal two orders of magnitude smaller than a dense MLP.
Architectural vs. Learned Fragility: By analyzing training trajectories, the author establishes that the lack of specialization and the presence of fragility are architectural properties, not learned behaviors that evolve during training.
Methodological Refinement: The study reinforces the necessity of combining probing accuracy with fragility testing. It demonstrates that high accuracy can coexist with extreme fragility when signal scales differ, challenging the sufficiency of accuracy alone as an alignment metric.

Significance and Claims
The paper claims that for alignment purposes, MoE architectures do not offer the anticipated advantage of discrete, prunable "moral experts." Instead, they introduce a permanent structural fragility due to output dilution. The author argues that this finding has direct implications for intervention strategies:

Intervention Limits: Pruning or fine-tuning specific experts will not isolate moral features, as they are redundantly encoded across all experts.
Fine-Tuning Risks: Because the moral signal in MoE models exists at a very small scale, standard fine-tuning procedures (e.g., LoRA) that modify weights by small absolute amounts could represent large relative perturbations to the moral signal, potentially destroying alignment more easily than in dense models.
Metric Validity: The gap between probing accuracy and fragility is an architectural feature of MoE models. Therefore, relying solely on probing accuracy to assess alignment depth is insufficient, as it fails to capture the security of the encoded information.

The author concludes that the "redundant but fragile" nature of MoE representations is a direct result of the sparse aggregation mechanism, suggesting that future MoE designs must address signal scale if robustness is a priority.

More like this