When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning

Imagine you are the captain of a massive, high-tech spaceship called a "Large Language Model." This ship is built to understand and generate human language, but it's so heavy and complex that it's too slow to fly through the narrow canyons of a real-world application. To make it fly, you need to perform "pruning": carefully removing parts of the ship's engine to make it lighter without crashing.

The tricky part is that the ship doesn't just carry one type of cargo; it carries many different groups of passengers, like "general travelers," "rare-code experts," and "safety inspectors." If you cut a part of the engine that the "rare-code experts" rely on, the ship might fly fine for everyone else, but the experts will be stranded. The big question in this field of computer science is: How do you know which parts to cut so that every group of passengers stays safe, even if you can't test every single possible cut? Scientists use "compression statistics"—mathematical scores that try to predict which parts are useless. But this paper asks a scary question: What if the score says a part is safe to cut, but it actually destroys the ship for a specific group of passengers?

This paper, titled "When Compression Scores Cannot Decide," investigates the hidden limits of these prediction scores. The author, led by Andrew Zhang, argues that a single, average score is like looking at a blurry map: it might show that the ship is generally fine, but it hides the fact that one specific corner is on fire. They found that these scores often fail to predict the worst-case damage for specific groups. Instead of relying on a single "magic number" to decide what to cut, the paper suggests a new strategy: use local clues to build a short list of candidates, and then actually test those specific candidates to see how they perform for every single group before making the final cut.

The Problem with the "Average" Score

Think of a compression score like a teacher grading a class. If the teacher only looks at the class average, they might think everyone is doing great. But what if one student is failing miserably while everyone else is acing the test? The average hides the failure. In the world of AI, researchers use "pruning scores" to decide which neurons (the tiny processing units inside the AI) to remove. These scores often look at the "average" behavior of the AI across all its data.

The author discovered that this approach is dangerous. They found a specific pruning score that was very reliable (with a "split-half reliability" of 0.906, meaning it consistently gave the same answer when tested twice). This score predicted that a certain cut would improve the AI's performance by 16.1%. However, when they actually made the cut, the result was a disaster: the AI performed 6.0% to 7.7% worse than the control groups. The score was right about the average, but it completely missed the fact that it would ruin the experience for specific groups of users.

The "Information Boundary" and the Hidden Gap

To explain why this happens, the author uses a concept called an "information boundary." Imagine you are trying to guess the shape of a hidden object by looking at its shadow. If the shadow is just a simple average, you might think the object is a perfect sphere. But the object could actually be a cube with a sharp corner poking out in a way the shadow doesn't show.

The paper argues that standard pruning methods only see the "shadow" (the pooled average). They miss the "sharp corners" (the specific damage to individual groups). The author calls the gap between what the score sees and what actually happens the "observation fiber." It's like a foggy window: you can see the general shape, but you can't see the details that matter most.

They proved mathematically that if you only look at the average, you could be off by a factor of the number of groups you have. If you have 4 groups, the worst-case damage could be 4 times worse than the average suggests. This is a "conic law" they derived, which acts like a rule of physics for AI pruning: averaging always hides the worst-case scenario unless you do something special to look at the groups separately.

The Solution: A Two-Step Dance

So, if the average score is a liar, what should we do? The paper proposes a two-step process, which they call "Propose" and "Decide."

Step 1: Propose (The Local Clues)
First, you use local clues to build a short list of candidates. In the dense AI models (the big, heavy ones), they used a "group-resolved diagonal" method. This is like checking the engine parts for each specific group of passengers separately, rather than just looking at the whole engine. This method was very good at spotting the general severity of damage (it had a correlation of 0.9239 with the actual worst damage). It could tell you, "Hey, this group of passengers is in big trouble if we cut this part." However, it couldn't tell you exactly which cut was the best among the ones that looked okay. It was good at finding the danger, but bad at picking the winner.

Step 2: Decide (The Real Test)
Once you have a short list of candidates (a "finite menu"), you must stop guessing and start testing. The author found that you cannot rely on a single score to rank these candidates. Instead, you have to measure the actual performance of each candidate on the specific groups.

They tested this on three different AI models (Llama, SmolLM3, and Qwen). By using a "target-matched" approach—where they measured the actual performance of the candidates on the specific groups they cared about—they found real improvements.

• On the Llama model, they reduced the "worst-group perplexity inflation" (a measure of how confused the AI gets) by 7.96%.
• On Qwen, they reduced it by 2.80%.
• On SmolLM3, they reduced it by 2.68%.

These weren't just guesses; they were measured improvements that held up when tested on new, unseen data.

The MoE Twist: The Router's Secret Map

The paper also looked at a different type of AI called "Mixture of Experts" (MoE). Imagine these models as a team of specialists. Instead of one giant brain, you have many small experts, and a "router" decides which expert to use for each question.

In this setup, the router leaves a "trace" or a map showing which experts are used by which groups. The author found that this map was incredibly useful. It could predict which single expert to remove better than random guessing (getting it right 114 out of 192 times, compared to 81 out of 192 for a standard method).

However, just like with the big models, the map wasn't perfect. It could tell you which single expert was the most dangerous to keep, but it couldn't tell you the best combination of experts to remove. To solve this, they had to test the full combinations. When they did, they found two specific moves that improved the AI's performance by 13.7% and 7.2% on the worst-case groups.

The Big Takeaway

The main lesson of this paper is that you cannot trust a single, average number to make life-or-death decisions for AI groups. If you want to make an AI that is fair and robust for everyone, you have to be more careful.

1. Local clues are good for spotting danger: Use group-specific scores to find the big risks.
2. But you must test the winners: Once you have a short list of options, you must actually measure how they perform on the specific groups you care about.
3. The "One-Size-Fits-All" approach fails: A pruning strategy that works for one model or one group might fail completely for another. The author found that "fine directions" (specific cutting patterns) that worked for one model didn't work for another.

The paper concludes that while we can build better maps and lists of candidates, the final decision always requires a direct measurement of the outcome. You can't just calculate your way to safety; you have to check the engine after every cut. This ensures that when you fly your AI spaceship, no group of passengers gets left behind in the dark.