Imagine you have a massive, super-smart robot brain built to solve puzzles, write code, and translate languages. This brain isn't made of one giant, heavy block of metal; instead, it's built like a giant factory with hundreds of specialized workstations. This is the world of Large Language Models (LLMs), the AI engines powering everything from chatbots to coding assistants. To make these brains fast and efficient, engineers use a trick called Mixture-of-Experts (MoE). Think of it like a giant team of 256 different "experts" in every single room of the factory. When a question comes in, a smart manager (called a "router") decides which few experts are needed to solve that specific problem, while the rest of the team takes a coffee break. This keeps the robot fast because it only wakes up the workers it actually needs.

But here's the big question: Are all those workers equally important? If you had to shrink the factory to save space and electricity, could you just fire 30% of the workers in every room at random? Or is there a secret pattern? Scientists have been guessing, but they didn't have a clear map of which parts of the brain are fragile and which parts are sturdy. Understanding this matters because these models are getting huge, and if we can safely "prune" (remove) the unnecessary parts, we can make them run faster on cheaper computers, saving energy and money without losing their smarts.

---

The Great Factory Audit: Who Can We Let Go?

In this paper, a team of researchers from Persistent Systems decided to play a high-stakes game of "cut the cord" on a specific, massive AI model called Qwen3.6-35B-A3B. This model is a beast: it has 40 layers (think of them as 40 floors in a skyscraper factory), and on each floor, there are 256 experts. Every time the model processes a word, it picks the top 8 experts to do the work.

The researchers wanted to find out: If we force the model to ignore the "weakest" experts (the ones with the smallest numbers in their brains) in certain layers, does the model crash, or does it keep working? They tested this on a tricky task: translating code from one programming language to another (like turning C++ into Python), using a benchmark called XLCoST.

The Discovery: It's All About Where You Cut

The team tried cutting experts in different patterns, like a surgeon testing different incision sites. They had three main theories:

1. The "Flat Cut" Theory: Just cut 30% of the weakest experts from every floor, everywhere.
2. The "Early Cut" Theory: Maybe the bottom floors are the most important?
3. The "Late Cut" Theory: Maybe the top floors are just repeating what the bottom floors already did?

The results were a total shock to the "Flat Cut" idea.
When they tried the Flat Cut (removing 30% of experts from all 40 floors), the model basically forgot how to do its job. Out of 300 coding puzzles, it only got 150 right (or "Good/Similar"). It started hallucinating, leaking its internal thinking process into the final code, and making syntax errors. It was like firing 30% of the staff in the foundation, the middle offices, and the top floor all at once—the whole building started to wobble.

But then, they found the "Magic Zone."
The researchers discovered that the model's sensitivity changes drastically depending on which floor you are on:

• Floors 0–9 (The Foundation): These are super fragile. If you cut experts here, the model breaks immediately.
• Floors 10–29 (The Middle): Also very fragile.
• Floors 30–39 (The Penthouse): These top floors are surprisingly tough! The experts here seem to be doing a lot of overlapping work. The "router" (the manager) is already confused here, picking experts with very low confidence, which means the experts are somewhat interchangeable.

The Winning Strategy: The "Top-Down" Trim

The team ran a massive experiment with 300 prompts to find the perfect trimming schedule. They found that if you focus your cuts only on the top floors, the model stays healthy.

• The "Late Ramp" Strategy: They cut 35% of the experts on floors 30–34, and 55% on the very top floors (35–39).
• The Result: Out of 300 prompts, this strategy kept 255 outputs working correctly! That's way better than the flat cut, and it removed over 1,100 experts.

But wait, they didn't stop there. They wanted to be sure this wasn't just a lucky fluke, so they tested their best ideas on a fresh, unseen set of 500 prompts. This is where the story got even more interesting.

On the larger test, the "Late Ramp" was still good, but a narrower strategy won the prize:

• The "Very-Late Only" Strategy: They only touched the top 5 floors (35–39) and cut 50% of the experts there.
• The Result: This tiny, surgical strike kept 419 out of 500 outputs working perfectly! Even better, it only removed 640 experts in total (out of a total of 10,240 experts in the whole model).

This means the researchers found a way to remove a significant chunk of the model's "muscle" from the very top, and the model barely noticed. It's like realizing that the top 5 floors of a skyscraper are mostly just decorative balconies; you can remove half the railing, and the building stands just as tall.

A Side Quest: The "Thinking" vs. "Answering" Mystery

The team also tried to see if the model uses different experts when it's "thinking" (reasoning through a problem) versus when it's "answering" (writing the final code). They looked at a slightly older version of the model (Qwen3.5) to see if they could spot a difference.
They found that during the "thinking" phase, the model uses a wide, messy crowd of experts. But when it's time to "answer," it gets more focused and uses fewer experts. This suggests that if we want to prune the model in the future, we might need to be careful not to cut the "thinking" experts, even if they look weak on paper. However, the authors are careful to say this is just a hint for the future, not a proven rule yet.

What About Speed?

They also tried a different trick: instead of cutting experts, they told the model to only pick 6 experts instead of the usual 8 for every word. On a small test of 100 prompts, this made the model run much faster (less "wall-clock time") without losing any quality. But, when they tried to combine this speed-up with the aggressive expert cutting, the model got confused. It seems you can't just stack these shortcuts on top of each other without testing them carefully.

The Bottom Line

The main takeaway from this paper is a clear "Don't Do It" for uniform pruning and a "Do This" for smart pruning.

• Don't just cut 30% of experts from every layer equally; it destroys the model's ability to think.
• Do focus your cuts on the very top layers (floors 35–39). The model is surprisingly tolerant of losing half its experts there.

The authors are careful to note that they haven't actually deleted the weights from the computer's memory yet (they just told the router to ignore them). So, while the model acts smaller and more efficient, it still takes up the same amount of space on the hard drive. The next step, which they suggest for future work, is to actually perform "physical weight surgery" to delete those unused experts permanently and save real storage space.

In short, the researchers found that the "brain" of this AI model has a very specific weak point: the top floors are much more redundant than the bottom ones. By trimming the top, we might be able to build smaller, faster, and cheaper AI models without breaking them.