🤖 machine learning

Compressing LLMs with MoP: Mixture of Pruners

The paper introduces MoP (Mixture of Pruners), an iterative framework that unifies depth and width pruning to outperform existing single-dimension methods in accuracy and latency reduction across LLaMA and LLaVA models.

Original authors: Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Leandro Giusti Mugnaini, Keith Ando Ogawa, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jorda

Published 2026-07-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Victor Zacarias, Leandro Giusti Mugnaini, Keith Ando Ogawa, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to carry a massive, heavy library in a backpack. This library contains the sum of human knowledge, written in a language so complex that only the smartest computers can read it. These "libraries" are called Large Language Models (LLMs). They are amazing at writing stories, solving math problems, and even chatting like a friend. But there's a catch: they are so huge and heavy that they require enormous amounts of electricity and super-fast computers just to open the book and read a single sentence. If you want to use them on a regular laptop or a phone, they are often too slow and too heavy to carry.

To fix this, scientists have been trying to make these libraries smaller without throwing away the important books. One way to do this is called "pruning." Think of pruning like gardening. You have a giant, overgrown bush (the big computer model), and you want to trim it down to a neat, smaller shape. You can cut off entire branches (making the bush shorter), or you can trim the leaves off the branches (making the bush thinner). For a long time, gardeners had to choose: either cut off whole branches or trim the leaves, but not both at the same time. The question was, which way makes the bush smaller while keeping it healthy and able to bear fruit?

This paper introduces a new gardening tool called MoP, which stands for Mixture of Pruners. Instead of choosing just one way to trim, MoP is a smart, iterative gardener that tries both methods at every step. Imagine you are trimming your bush. At each cut, MoP asks two questions: "If I cut off this whole branch, how does the bush look?" and "If I trim the leaves off this branch instead, how does it look?" It then picks the version that keeps the bush looking the most like the original, healthy plant. It repeats this process, switching between cutting branches and trimming leaves, until the bush is small enough to fit in your backpack.

The researchers tested this method on some of the smartest computer brains in the world, like the LLaMA-2 and LLaMA-3 models. They found that MoP is much better at shrinking these models than using just one method alone. When they cut the models down by 40% (making them 40% smaller), MoP kept the models just as smart as the competition, but it also made them run significantly faster. In fact, the models became 39% faster at answering questions. Even more surprisingly, they tried this on a model that can see pictures and read text (called LLaVA-1.5). They found that even though they only "taught" the trimmed model with text (no pictures), the model could still understand images almost as well as before. This suggests that mixing the two trimming strategies creates a smaller, faster, and smarter computer brain that doesn't lose its way, even when it gets very small.

Technical Summary: Compressing LLMs with MoP: Mixture of Pruners

Problem Statement
Large Language Models (LLMs) face significant computational demands, necessitating methods to reduce parameter counts and accelerate inference without sacrificing performance. While model pruning is an effective strategy, current approaches typically focus on a single dimension: either depth pruning (removing entire transformer layers) or width pruning (removing internal components like attention heads or MLP neurons). Depth pruning offers superior inference acceleration by reducing sequential computation, whereas width pruning allows for finer-grained selection of redundant structures, often better preserving downstream task performance. The paper identifies a dichotomy in existing literature where these complementary strengths are rarely unified, leading to suboptimal trade-offs between compression ratio, accuracy, and latency.

Methodology: Mixture of Pruners (MoP)
The authors propose MoP, an iterative framework that unifies depth and width pruning into a cohesive strategy. The core mechanism operates as follows:

Iterative Path Selection: At each iteration, MoP generates two candidate models to advance the pruning path:
- Depth Candidate: Removes one complete transformer layer (specifically the third-to-last layer, preserving the final two layers as per literature findings on redundancy).
- Width Candidate: Removes an equivalent number of parameters by pruning attention heads and MLP neurons across layers.
Parameter Matching: To ensure a fair comparison, the width pruning step is calibrated to remove the exact same number of parameters as the single layer removed in the depth step. This requires fine-grained control, which the authors achieve using the AMP (Activation Magnitude Pruning) criterion.
Evaluation and Selection: Both candidates undergo a short recovery fine-tuning step on a subset of training data. A Path Criterion ( $P$ ) then scores the candidates against the original unpruned model using metrics such as Cosine Similarity, Kullback-Leibler (KL) divergence, Perplexity (PPL), or even a randomized selection. The candidate with the score closest to the original model (lowest deviation) is selected to become the base for the next iteration.
Final Recovery: Once the target compression ratio is reached, the final pruned model undergoes full recovery fine-tuning on the complete training dataset.

The framework is modular, allowing for the integration of different pruning criteria for width and depth, as well as different path selection strategies.

Key Contributions
The paper highlights five primary contributions:

Unified Framework: MoP effectively combines depth and width pruning, capturing the latency benefits of layer removal and the fine-grained selectivity of component removal.
Modularity: The design accommodates various combinations of pruning criteria, allowing for future integration of state-of-the-art methods.
Diverse Pruning Scheme: By distributing removals across both dimensions, MoP achieves higher compression ratios without exhausting a single dimension.
Multimodal Extension: Unlike most pruning literature focused solely on text, the authors test MoP on both language-only (LLaMA) and multimodal (LLaVA) architectures.
Text-Only Recovery for Visual Tasks: The authors observe that recovery fine-tuning using a text-only dataset (Alpaca) can surprisingly restore performance on visual benchmarks in pruned vision-language models.

Results
The authors evaluate MoP on LLaMA-2 7B, LLaMA-3 8B, and LLaVA-1.5 7B:

Accuracy: On LLaMA-2 and LLaMA-3, MoP consistently outperforms competing structured pruning methods (e.g., SliceGPT, AmoebaLLM, PruneNet) across 20%, 30%, and 40% compression ratios. Notably, the "worst-seed" performance of MoP (using random path selection) often exceeds the average performance of the best competing baselines, indicating robustness against path selection variance.
Latency: MoP translates structural pruning into real speedups. At 40% compression, the method reduces end-to-end latency by 39% (a 1.63× speedup) on an NVIDIA RTX 4090. At 30% compression, it achieves a 1.38× speedup, outperforming other methods that require more aggressive compression (e.g., 50%) to achieve similar gains.
Multimodal Performance: On LLaVA-1.5, MoP maintains significant reasoning capabilities. At 40% compression, the model retains approximately 78% of its mean predictive capability. Crucially, the study demonstrates that fine-tuning with text-only data (Alpaca) effectively recovers performance on visual tasks (ScienceQA, VizWiz), a finding the authors note as a novel observation.

Significance and Claims
The paper claims that MoP establishes a new frontier in structured pruning by overcoming the structural rigidity of single-axis approaches. The hybrid strategy provides exceptional resilience, ensuring that even under unfavorable path selection, the method delivers top-tier accuracy. The authors emphasize that this approach not only advances the state-of-the-art for efficient LLMs but also validates the viability of hybrid width-depth pruning in the multimodal domain. Furthermore, the discovery that text-only recovery fine-tuning suffices for visual tasks in pruned vision-language models suggests a more efficient pathway for deploying compressed multimodal systems, reducing the need for expensive multimodal recovery datasets.

Technical Summary: Compressing LLMs with MoP: Mixture of Pruners

More like this