💻 computer science

Composing Lightweight Methods on Deep Neural Network: A Survey

This survey introduces a novel taxonomy for composing lightweight deep learning techniques—such as pruning, quantization, and knowledge distillation—to analyze how their selection, ordering, and interaction impact efficiency and accuracy, thereby offering practical guidance for deploying deep neural networks on resource-constrained edge devices.

Original authors: Kilho Son, Yunho Jeon

Published 2026-11-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Kilho Son, Yunho Jeon

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the most powerful artificial intelligence systems, capable of recognizing faces, translating languages, or diagnosing diseases, are too heavy to carry in a pocket. These systems are built from deep neural networks, vast digital structures that learn by adjusting billions of tiny internal settings. While these massive networks excel at complex tasks, they demand enormous amounts of memory and processing power, resources that are scarce on everyday devices like smartphones, medical sensors, or autonomous drones. To make these smart systems useful outside of giant data centers, engineers have spent years developing ways to shrink them without losing their intelligence. They have invented tools to strip away unnecessary parts, simplify the numbers the computer uses, and teach smaller models to mimic the behavior of larger ones.

For a long time, researchers treated these shrinking tools as separate solutions, testing each one in isolation. They asked whether pruning, which removes unused connections, was better than quantization, which simplifies the numbers, or if knowledge distillation, which transfers wisdom from a large teacher model to a small student, was the most effective path. However, in the real world, engineers rarely rely on just one trick. They often combine several methods to squeeze a model down to the size required for a specific device. A new survey by Kilho Son and Yunho Jeon shifts the focus from individual tools to the art of combining them. The researchers examined how different compression techniques interact when used together, discovering that the order in which they are applied and the way they are optimized together can make the difference between a model that works perfectly and one that fails.

The survey categorizes these combinations into three main strategies: doing them one after another, doing them all at once, or mixing the two approaches. The first strategy, sequential application, is like a factory assembly line where a model is first pruned, then quantized, and finally refined. This method is straightforward and easy to implement, making it a popular choice for many developers. The researchers found that this approach works well when the tools address different types of waste. For instance, removing redundant connections first can leave a cleaner structure for the next step to simplify the numbers. However, the study also highlights a significant limitation: once a step is finished, it cannot be changed. If the first step removes information that the second step needed, the damage is permanent, and the final model may struggle to recover its accuracy.

To overcome the rigidity of the assembly line, some researchers use joint optimization, where multiple compression techniques are applied simultaneously while the model is being trained. Instead of finishing one task before starting the next, the system adjusts the pruning, the number simplification, and the learning process all at the same time. The survey suggests that this method can yield better results because the different techniques can adapt to one another in real time. For example, when a model is being simplified and its numbers are being reduced simultaneously, the system can learn to keep the specific parts of the network that are most important for both tasks. The researchers found that this approach often produces models that are both smaller and more accurate than those made with the step-by-step method, but it requires more computational power and careful tuning to balance the competing goals.

The third category, hybrid composition, blends these two worlds. In these scenarios, a team might use joint optimization to handle the most difficult parts of the compression, followed by a separate step to fix any remaining issues. The survey points out that this flexibility is crucial for complex tasks, such as running large language models on limited hardware. In these cases, a model might be pruned and quantized together to save space, and then a separate recovery step is added to restore lost accuracy. The researchers observed that the best results often come from understanding which tools complement each other and which ones conflict. For instance, while pruning and quantization often work well together because they target different redundancies, applying them in the wrong order or without proper coordination can lead to a loss of performance that neither method could have caused alone.

A major part of the survey focuses on the practical reality of running these compressed models on actual hardware. The researchers emphasize that a model might look efficient on paper, with fewer numbers and smaller size, but still run slowly on a real device. This happens because the way a computer chip handles data matters just as much as the data itself. If a compression method creates a pattern that the processor cannot handle efficiently, the speed gains disappear. The survey reviews studies where models were tested on real phones and servers, revealing that the best theoretical compression does not always translate to the fastest real-world performance. For example, a model that is heavily pruned might require special software to run quickly, and if that software is not available, the device might actually run slower than before.

The authors also look at the specific challenges of compressing the newest generation of large language models, which are used for tasks like writing code or generating stories. These models are so large that they require special techniques to fit on consumer devices. The survey highlights how combining techniques like low-rank adaptation, which adds small, trainable layers to a frozen model, with quantization can make these massive systems usable. However, the researchers caution that simply stacking these methods does not guarantee success. The interaction between the different parts of the model must be carefully managed to ensure that the added components do not consume more memory than they save.

Ultimately, this survey provides a roadmap for navigating the complex landscape of model compression. It moves beyond the question of which single tool is the best and instead asks how to build a team of tools that work together. The researchers conclude that there is no single perfect recipe for every situation. The right combination depends on the specific device, the task at hand, and the available resources. By understanding how these methods interact, engineers can design artificial intelligence that is not only powerful but also portable, bringing the benefits of advanced computing to the devices we use every day. The work suggests that the future of efficient AI lies not in finding a single silver bullet, but in mastering the delicate balance of combining multiple strategies to fit the unique constraints of the real world.

=== SUMMARY ===

Technical Summary: Composing Lightweight Methods on Deep Neural Networks: A Survey

1. Problem Statement

Deep learning models, particularly large-scale Convolutional Neural Networks (CNNs) and Transformers, have achieved remarkable performance but impose substantial computational and memory demands. These requirements hinder their deployment on resource-constrained edge devices (e.g., mobile phones, IoT), often forcing reliance on cloud servers which introduces latency and privacy concerns.

While individual model compression techniques—such as efficient architecture design, decomposition, pruning, quantization, and knowledge distillation (KD)—have been extensively studied, existing surveys typically analyze these methods in isolation. There is a lack of systematic research on how composing multiple lightweight methods affects efficiency and accuracy. Specifically, the field lacks a clear understanding of:

How the ordering of methods (e.g., pruning before quantization vs. vice versa) impacts performance.
The optimization dependencies between methods (e.g., whether they should be applied sequentially, jointly optimized, or in a hybrid manner).
The interactions (complementary or conflicting) that arise when multiple compression techniques are combined in real-world deployment pipelines.

2. Methodology

This survey adopts a taxonomy-based approach to analyze the composition of lightweight methods. The authors systematically review literature from major AI, computer vision, NLP, and hardware-aware conferences and journals. The core methodology involves:

Taxonomy Construction: The authors classify compositions based on optimization dependencies rather than just the combination of techniques. The three primary categories are:
- Sequential Application: Methods are applied in a predetermined order (e.g., Pruning $\to$ Quantization $\to$ KD). Each stage treats the output of the previous stage as fixed.
- Joint Optimization: Multiple methods are integrated into a single training, search, or fine-tuning process where variables and objectives are optimized simultaneously or with mutual influence (e.g., pruning masks and quantization intervals updated together).
- Hybrid Approaches: A mix of sequential and joint strategies (e.g., joint pruning and quantization followed by sequential KD for recovery).
Qualitative and Quantitative Analysis:
- The survey reviews representative studies within each category, focusing on within-study comparisons (ablations) to assess the effectiveness of specific compositions.
- Metrics: The authors use Accuracy Drop (AD) to measure quality degradation relative to a baseline and Compression Rate (CR) to measure efficiency gains.
- Hardware Evaluation: The survey examines real-hardware deployment results (latency, memory, throughput) to identify costs hidden by theoretical metrics (e.g., sparse metadata overhead, kernel fusion requirements).
Scope Expansion: The analysis extends beyond traditional computer vision tasks to include Large Language Models (LLMs), incorporating Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA as a distinct composition component for LLMs.

3. Key Contributions

The paper makes four primary contributions to the field of model compression:

Comprehensive Review: It provides the first survey dedicated specifically to the composition of lightweight methods, rather than treating combinations as secondary to individual techniques.
Novel Taxonomy: It proposes a classification system based on optimization dependencies (Sequential, Joint, Hybrid), distinguishing how methods interact during the optimization process.
Empirical Analysis: It synthesizes quantitative evidence from prior studies to evaluate how different compositions affect the accuracy-efficiency trade-off, highlighting trends such as the complementary nature of pruning and quantization.
Practical Guidance and Future Directions: It identifies promising yet underexplored compositions, highlights potential conflicts (e.g., aggressive quantization destroying information needed for recovery), and provides a guide for selecting, ordering, and deploying composed models on physical hardware.

4. Key Results and Findings

4.1 Sequential Application

Effectiveness: Sequential application is effective for further compressing efficient architectures or recovering accuracy after aggressive compression.
Ordering Matters:
- Pruning $\to$ Quantization: Generally preferred. Pruning removes redundant structures first, allowing quantization to focus on important weights, often yielding higher compression with minimal accuracy loss.
- KD as Recovery: Applying Knowledge Distillation after pruning or quantization consistently recovers accuracy drops.
- KD as Pre-processing: Applying KD before pruning can concentrate information, allowing for higher compression ratios in subsequent pruning steps, though evidence on whether this outperforms KD-after-pruning is mixed.
Limitation: Sequential pipelines cannot revise earlier decisions based on later constraints (e.g., pruning might remove information critical for a subsequent quantization step).

4.2 Joint Optimization

Synergy: Joint optimization often outperforms sequential pipelines when methods interact strongly.
- Quantization-Aware NAS: Searching for architectures specifically designed for quantization yields better accuracy at high compression rates than quantizing a fixed architecture.
- Pruning + Quantization: Jointly optimizing sparsity and precision (e.g., using alternating updates or shared loss functions) can achieve significantly higher compression rates (e.g., 462 $\times$ on ResNet-50) with comparable accuracy drops compared to single-method approaches.
- KD + Pruning/Quantization: Jointly optimizing distillation loss with compression objectives can yield better results than sequential application, though straightforward joint training does not always guarantee superiority; careful loss balancing is required.
Complexity: Joint optimization requires more complex search spaces and loss balancing but is beneficial when sufficient retraining budget is available.

4.3 Hybrid Approaches

Flexibility: Hybrid strategies combine the strengths of both approaches. For example, jointly optimizing pruning and quantization to find a robust compressed structure, followed by a sequential KD stage for final accuracy recovery.
LLM Specifics: In LLMs, hybrid pipelines are common (e.g., Pruning $\to$ LoRA $\to$ Quantization). PEFT (like LoRA) often serves as a lightweight recovery or adaptation stage after structural compression.

4.4 Real-Hardware Deployment

Theoretical vs. Practical: High theoretical compression rates do not always translate to lower latency. Factors like sparse kernel support, memory bandwidth, and metadata overhead can negate arithmetic savings.
Platform Dependency: Performance is highly dependent on the target hardware (CPU, GPU, NPU). For instance, structured pruning and dense decomposition often yield better real-world speedups on standard hardware than unstructured pruning, which requires specialized sparse kernels.

5. Significance and Claims

The authors claim that this survey is significant because it shifts the focus from individual compression techniques to the systematic composition of these methods.

No Universal Winner: The paper explicitly states that no single category (Sequential, Joint, or Hybrid) is universally superior. The optimal choice depends on the specific constraints (memory vs. latency), the target hardware, and the nature of the model redundancy.
Complementarity vs. Conflict: The survey emphasizes that methods are complementary when they address distinct bottlenecks (e.g., pruning for structure, quantization for precision) but can conflict if they destroy information needed by subsequent stages (e.g., ultra-low-bit quantization removing data needed for LoRA recovery).
Guidance for Practitioners: The paper provides actionable guidance, such as:
- Using Sequential pipelines for simple, modular compression.
- Using Joint Optimization when methods perturb the same parameters or share a budget.
- Using Hybrid approaches when a jointly optimized stage needs a separate recovery phase.
Future Directions: The authors identify gaps in current research, including the need for more real-device evaluations (measuring DRAM traffic and energy), better handling of LLM-specific constraints (KV cache, PEFT costs), and automated pipelines that optimize for the full deployment stack rather than just model size.

In conclusion, the paper argues that effective deployment of deep neural networks on edge devices requires a holistic view of compression, where the interaction, ordering, and optimization coupling of multiple lightweight methods are carefully engineered rather than applied in isolation.