🤖 machine learning

Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

This paper proposes a scalable, closed-form knowledge editing framework for Mixture-of-Experts LLMs that leverages tensor structures and the Woodbury matrix identity to achieve per-expert updates with up to 6x acceleration while maintaining editing quality comparable to dense-layer baselines.

Original authors: Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin, Daniil Medyakov, Vladimir Solodkin, Aleksandr Beznosikov

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roman Maksimov, Vladimir Aletov, Dmitry Bylinkin, Daniil Medyakov, Vladimir Solodkin, Aleksandr Beznosikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Updating a Giant Library

Imagine a Large Language Model (LLM) as a massive, super-smart librarian who knows almost everything. However, once the librarian is trained, their knowledge is "frozen." If the world changes (e.g., a new president is elected, or a scientific fact is corrected), the librarian keeps giving old, wrong answers.

Usually, to fix this, you have to send the librarian back to school for a long, expensive retraining session. This is slow, costs a fortune in computer power, and risks making the librarian forget other things they already knew.

Knowledge Editing is a shortcut. Instead of retraining the whole librarian, you just go in and swap out a specific file in their memory. Previous methods worked great for "dense" models (where every part of the brain is active), but they struggled with modern models.

The New Challenge: The "Expert" Team

Modern AI models (like the ones used by OpenAI or DeepSeek) don't use a single giant brain. They use a Mixture-of-Experts (MoE) architecture.

The Analogy: Imagine instead of one giant librarian, you have a team of 100 specialized experts (a historian, a coder, a chef, a doctor, etc.).
How it works: When you ask a question, a "router" (a manager) looks at the question and only wakes up the 4 or 8 experts who are relevant. The others stay asleep.
The Problem: If you want to update a fact (e.g., "The capital of France is Paris"), the old editing tools didn't know how to talk to this specific team. They tried to treat the whole team as one big blob, which is inefficient and messy. The existing method to fix this (called MoE-Edit) was like trying to update the experts one by one, in a long line, which took forever.

The Solution: MoTE (The Smart Team Manager)

The authors propose a new method called MoTE (Mixture-of-Experts Tucker Editor). They solved the problem using two main tricks:

1. The "Shortcut" Math (Woodbury Identity)

The old way to update the experts required solving a massive, complex math puzzle that involved inverting a huge matrix (a grid of numbers). It was like trying to solve a Sudoku puzzle with a million squares.

The authors used a mathematical trick called the Woodbury Identity.

The Analogy: Instead of solving the million-square puzzle, they realized they only needed to solve a tiny 50-square puzzle that represents the changes you want to make.
The Result: This turns a task that takes hours into one that takes minutes. They can update the knowledge without ever needing to "wake up" or calculate the full weight of every single expert at once.

2. Respecting the Team Structure (Tensor Decomposition)

The second trick was realizing that the experts aren't random; they are related. They were trained together, so their knowledge is connected in a specific 3D shape (like a block of cheese rather than a flat sheet of paper).

The Analogy: Imagine the experts are a choir. If you want to change the pitch of the song, you don't just shout at one person; you adjust the whole group's harmony.
The Method: The authors used Tucker Decomposition. This is a way of compressing the "block of cheese" (the expert weights) into a smaller, core shape that captures the essence of the group.
The Benefit: By editing this smaller "core" shape, they automatically update all the experts in a way that respects their relationships. This prevents the model from getting confused or "forgetting" other things.

The Results: Fast and Accurate

The paper tested MoTE on several modern AI models (like Qwen and GPT-OSS) and compared it to the previous best method (MoE-Edit) and standard retraining.

Speed: MoTE is up to 6 times faster than the previous best method. It can edit 1,000 facts in a fraction of the time.
Quality: It is just as good at fixing the facts (Efficacy) and not breaking other parts of the model (Specificity) as the slower methods.
Efficiency: It achieves this by only doing the heavy math calculation once at the very end of the process and then spreading that result to the other layers, rather than recalculating for every single layer.

Summary

The paper introduces MoTE, a tool that lets us update modern, "expert-based" AI models quickly and accurately. It does this by:

Using a math shortcut to avoid doing unnecessary heavy calculations.
Respecting the team structure of the experts so the updates make sense.

This means we can keep AI models up-to-date with the real world without needing to retrain them from scratch, saving massive amounts of time and energy.

Technical Summary: Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

Problem Statement
Knowledge editing (KE) offers a lightweight alternative to full fine-tuning for updating factual knowledge in Large Language Models (LLMs). While established methods like ROME and MEMIT provide efficient, closed-form updates for dense feed-forward networks (FFNs), they are incompatible with modern Mixture-of-Experts (MoE) architectures. In MoE models, factual associations are distributed across a pool of experts gated by a router, rather than stored in a single dense matrix. Existing MoE-specific editing methods, such as MoE-Edit, rely on iterative block-coordinate descent (BCD) or sequential processing of experts. This approach creates a computational bottleneck, requiring repeated activation collection and sequential solver iterations that scale poorly with the number of experts and edits, rendering them impractical for large-scale production models.

Methodology: MoTE (MoE Tucker Editor)
The authors propose MoTE, a framework that adapts the closed-form, batched editing paradigm of MEMIT to MoE layers by exploiting their tensor structure. The method consists of three core components:

Woodbury Reduction for Scalability:
The global optimization objective for MoE editing involves inverting a massive system of size $(E \cdot d_{hidden}) \times (E \cdot d_{hidden})$ , where $E$ is the number of experts. MoTE utilizes the Woodbury matrix identity to reformulate this problem. By recognizing that the Gram matrix driving the inversion is a sum of $T$ rank-one terms (where $T$ is the batch size of edits), the method reduces the inversion complexity from cubic in the number of experts to cubic in the batch size ( $T \times T$ ). This allows for a closed-form solution without materializing or inverting the full stacked expert matrix.
Tensor-Structured Reformulation (Tucker Decomposition):
The authors identify that treating expert updates as independent blocks (a flat matrix approach) ignores the structural correlations between experts learned during pretraining. Instead, MoTE represents the update $\Delta W$ as a third-order tensor ( $E \times d_{model} \times d_{hidden}$ ) and constrains it to a low-rank Tucker manifold.
- Factorization: The update is decomposed into a small core tensor $G$ and three factor matrices ( $U_e, U_{out}, U_{in}$ ) derived via Higher-Order SVD (HOSVD) of the original weights.
- Optimization: The optimization problem is collapsed from the high-dimensional space of all experts to the much smaller core tensor $G$ . This reduces the dimensionality of the optimization variable by orders of magnitude while preserving the geometric structure of the MoE layer.
MEMIT-Style Multi-Layer Spreading:
Similar to MEMIT, MoTE spreads edits across multiple critical layers to ensure generalization. However, unlike previous MoE approaches that require re-solving for each layer, MoTE computes the closed-form solution only once at the last critical layer. The resulting core tensor update is then propagated to preceding layers with a decaying coefficient, avoiding redundant computations and backward passes.

Key Contributions

Tensor-Aware Formulation: The paper formulates the KE objective for MoE layers by respecting the tensor structure of stacked expert weights, avoiding naive concatenation or single-expert approximations.
Efficient Closed-Form Solution: By combining the Woodbury identity with Tucker decomposition, the method achieves a fully closed-form, backward-pass-free update. It requires only the inversion of fixed low-rank matrices (size determined by the batch size or core rank), independent of the total number of experts.
Scalability: The approach enables batched editing similar to MEMIT, making it feasible to edit modern, sparse LLMs without the sequential bottlenecks of prior MoE-specific methods.

Experimental Results
The authors evaluated MoTE on three modern MoE LLMs (Qwen3-30B-A3B, GPT-OSS-20B, and Qwen3.6-35B-A3B) using the COUNTERFACT and ZsRE benchmarks.

Editing Quality: MoTE achieves editing quality (efficacy, generalization, and specificity) comparable to strong baselines like MoE-Edit. In some scenarios (e.g., GPT-OSS-20B on ZsRE), it outperforms MoE-Edit in locality (specificity).
Speed: The method accelerates the editing procedure by up to 6x compared to MoE-Edit. This speedup is attributed to collecting activations only once at the last layer and solving the closed-form system only once, rather than iterating through experts and layers sequentially.
Ablation: Experiments confirm that the naive combination of Woodbury reduction with multi-layer spreading (without Tucker decomposition) fails to match the performance of MoE-Edit, validating the necessity of the tensor-structured prior.

Significance and Claims
The paper claims that closed-form, parameter-modifying knowledge editing is not intrinsically tied to dense architectures. By applying the correct algebraic treatment to the layer's structure, these techniques can be extended to complex sparse architectures. The authors position MoTE as a step toward scalable knowledge editing in modern LLMs, demonstrating that principled, optimization-based editing can keep pace with architectural evolution. They emphasize that their method provides a practical path for updating production MoE models efficiently, though they note limitations regarding high computational costs with extremely large batch sizes ( $>>1000$ ) and the initial overhead of computing HOSVD.