🤖 machine learning

APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning

APQF is an automated, LLM-guided framework that integrates profiling-driven structured pruning and mixed-precision quantization with adaptive fine-tuning to significantly reduce computational costs while maintaining high accuracy across diverse vision models on resource-constrained devices.

Original authors: Sadegh Jafari, Mohiuddin Bilwal, Fan Zhou, Brian Gelder, Ali Jannesari

Published 2026-08-07

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Sadegh Jafari, Mohiuddin Bilwal, Fan Zhou, Brian Gelder, Ali Jannesari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain, a deep neural network, that can recognize cats, dogs, and cars with incredible accuracy. But there's a catch: this brain is so huge and heavy that it needs a massive supercomputer to run. If you try to put it on a regular phone or a smartwatch, it would melt the battery or take forever to think. To fix this, scientists use "model compression," which is like packing a giant suitcase into a tiny backpack. They do this by pruning (cutting out useless parts of the brain) and quantization (simplifying the numbers the brain uses to think, like switching from high-definition video to a sketch). The big problem is that figuring out which parts to cut and how to simplify them usually requires a human expert to guess and try again, often making mistakes that ruin the robot's smarts.

Enter a new idea called APQF, a smart, automated system that acts like a team of specialized robot agents to pack your giant brain into a tiny backpack without losing its genius. Instead of guessing, this system first takes a detailed "health check" of the brain to see exactly which parts are heavy and which are fragile. Then, it uses a powerful AI planner (a Large Language Model) to decide exactly how much to cut and how much to simplify, layer by layer. It's like having a master tailor who doesn't just use one pair of scissors for the whole suit, but measures every single stitch to make the perfect fit. The result is a robot brain that is tiny, fast, and still just as smart as the original.

The Problem: The "One-Size-Fits-All" Mistake

For a long time, trying to shrink these giant AI brains has been a bit like trying to fit a whole elephant into a shoebox by just cutting off random chunks. Most methods used a "one-size-fits-all" approach. They would say, "Let's cut 50% of every layer" or "Let's make every number use the same small amount of space." But this is a disaster because different parts of the brain are different. Some layers are like the elephant's trunk—super important and sensitive. If you cut those, the brain forgets everything. Other layers are like the elephant's toenails—lots of them, but you can trim them without the elephant noticing.

When you treat a sensitive layer the same as a useless one, you end up with a tiny brain that is too dumb to do its job. Plus, figuring out the right cuts used to be a manual, tedious job that required a human expert to spend hours tweaking settings for every single new model. It was slow, expensive, and didn't work well when you switched from one type of brain to another.

The Solution: A Team of AI Agents

The paper introduces APQF (Agent Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning). Think of APQF not as a single tool, but as a factory floor run by a team of five specialized robot agents, all working together to shrink the model automatically.

The Profiling Agent (The Detective): Before any cutting happens, this agent runs a deep investigation. It measures exactly how much work each part of the brain does and tests how the brain reacts if you remove a piece. It creates a "map" showing which parts are heavy and which are fragile. This isn't a guess; it's based on real data from the specific model.
The Pruning Agent (The Sculptor): Using the detective's map, this agent decides exactly how much to cut from each layer. Instead of using a uniform rule, it asks a smart AI planner (a Large Language Model) to look at the map and say, "Cut 20% here, but only 5% there." It does this in stages, cutting a little, checking the brain's health, and then cutting a bit more.
The Fine-Tuning Agent (The Healer): Every time the sculptor cuts something, the brain might get a little dizzy (lose accuracy). This agent acts like a physical therapist. It gently exercises the brain to help it recover its strength. If the cut was small, it does a light workout. If the cut was big, it does a full recovery session. It uses smart tricks to fix the brain without needing to retrain it from scratch.
The Quantization Agent (The Simplifier): Once the brain is the right size, this agent simplifies the numbers it uses. It decides that some parts can use very small numbers (like 4 bits) while keeping other critical parts using bigger numbers (like 16 bits). This is called "mixed-precision," and it saves a huge amount of space.
The Evaluation Agent (The Judge): This agent checks the final product against the original to make sure it still works. It measures how much space was saved and how much accuracy was kept.

What They Found: Tiny Brains, Big Smarts

The researchers tested this system on several famous AI models, including ResNet, VGG, ViT, DeiT, and Swin, using two standard test sets: ImageNet-1k (a huge collection of 1,000 types of images) and CIFAR-10 (a smaller set of 10 types of images).

The results were impressive. On ImageNet-1k, APQF managed to shrink the computer work needed to run the models down to between 5.6% and 7.7% of the original amount. That's a reduction of 13 to 18 times! Even with this massive shrinkage, the models kept their accuracy very close to the original. For example, on a model called DeiT-Tiny, the compressed version actually got slightly smarter (going from 66.52% to 67.90% accuracy) while using almost no computing power.

When compared to other methods that try to do pruning and simplification at the same time (like a method called GETA), APQF showed it is much better at using limited data. If you only have 200,000 images to train on, GETA's accuracy would crash, dropping to around 51-59%. But APQF stayed strong, keeping accuracy between 68% and 77%. This suggests that APQF is much more efficient at learning how to shrink a model without needing a massive library of data.

On the smaller CIFAR-10 dataset, the results were even wilder. For three of the models tested (DeiT-Tiny, ResNet-50, and Swin-Tiny), the compressed versions ended up being more accurate than the original, uncompressed versions! On the VGG7 model, APQF reached 93.15% accuracy while using only 0.41% of the original computing power. This was the only method at that level of compression that actually improved on the original full-precision model.

Why the "AI Planner" Matters

One of the coolest parts of the paper is that the system uses a "planner" (a Large Language Model) to make the decisions. The researchers wanted to know: does it matter which AI planner they use? They tested six different planners, ranging from expensive, top-tier models to free, open-source ones.

The result? It didn't matter much. Whether they used a premium model or a free one, the final accuracy stayed in a very tight range between 97.4% and 97.9%. This means the system is robust and doesn't rely on one specific, expensive AI to work. It also suggests that the real magic comes from the process of using the profiler's data to guide the planner, rather than just the planner's own intelligence.

What the Paper Rules Out

The paper is very clear about what doesn't work. It explicitly argues against "uniform compression," where you apply the same cutting and simplifying rules to every single layer. Their tests showed that when they forced the system to use the same ratio for everything, the accuracy dropped significantly. They also showed that if you take away the "profiling" data (the detective's map) and let the AI planner guess based on general knowledge alone, the results get worse. This proves that you really need to measure the specific model before you start cutting.

They also found that methods which try to re-optimize the entire network from scratch (like GETA) struggle when you don't have a lot of training data. APQF, by contrast, starts with a pre-trained model and just tweaks it, making it much more data-efficient.

The Bottom Line

APQF suggests that the future of shrinking AI isn't about finding a single magic formula that works for everyone. Instead, it's about building an automated team that measures a specific model, listens to a smart planner, and carefully tailors the compression to that model's unique needs. It turns a messy, manual guessing game into a precise, scientific process. While the researchers note that this still requires some human setup (like choosing which AI planner to use), the system itself handles the heavy lifting, proving that you can make giant AI brains tiny and fast without losing their smarts.

Technical Summary: APQF – Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization

1. Problem Statement

Modern deep neural networks (DNNs), particularly in computer vision, achieve high performance but impose prohibitive computational and memory costs, hindering deployment on resource-constrained edge devices. While model compression techniques like pruning and quantization exist, they face three primary limitations:

Manual and Expertise-Driven: Current methods rely on hand-designed heuristics and manual tuning of per-layer pruning ratios and bit-widths. These choices are often architecture-specific and do not transfer well between different model families (e.g., CNNs vs. Vision Transformers).
Uniform Compression: Many approaches apply uniform pruning ratios and bit-widths across the entire model. This ignores the varying sensitivity of different layers, leading to significant accuracy loss when sensitive layers are over-compressed or redundant layers are under-compressed.
Data-Hungry Joint Optimization: Methods that jointly optimize pruning and quantization (e.g., GETA) often require re-optimizing the entire network from scratch, demanding large training datasets and compute resources to recover accuracy. Additionally, profiling data is typically used for manual inspection rather than to drive automated compression decisions.

2. Methodology: The APQF Framework

APQF (Agentic Profiling-Guided Framework) is an automated pipeline that integrates structured pruning, mixed-precision quantization-aware training (QAT), and accuracy recovery. It utilizes a multi-agent system where Large Language Models (LLMs) act as planners, grounded by empirical profiling data.

2.1 Core Agents

The framework consists of five cooperating agents:

ProfilingAgent: The foundational component. It performs static cost profiling (using ptflops and torch.profiler) to map parameters and multiply-accumulate operations (MACs). Crucially, it performs an empirical DepGraph sensitivity sweep, pruning each structural group at various ratios to measure the resulting accuracy drop. This generates a sensitivity CSV and a compact architecture brief, which serve as the "ground truth" for subsequent decisions.
PrunerAgent: Executes multi-stage structured pruning toward a user-defined target. Instead of a single-step prune, it iteratively proposes pruning strategies via an LLM. The LLM uses the profiling data to assign per-layer pruning ratios (non-uniform), selecting dependency groups that minimize accuracy degradation.
FineTuningAgent: A shared recovery component invoked after pruning stages and after quantization. It adaptively selects the recovery method based on the magnitude of accuracy loss ( $\delta$ $δ$ ):
- Small loss: No recovery.
- Moderate loss: Parameter-Efficient Fine-Tuning (PEFT) methods (LoRA, DoRA, rsLoRA, PiSSA, LoRA+).
- Large loss (final stage): A two-stage recovery (PEFT warm-up followed by full fine-tuning).
QuantAgent: Applies LLM-guided mixed-precision QAT. Using the same profiling evidence, the LLM assigns specific bit-widths (e.g., 2, 4, 6, 8, 16, 32 bits) to each surviving layer. The quantized model is trained using Knowledge Distillation (KD) from a frozen full-precision teacher, minimizing a combined loss of KL divergence (soft targets) and cross-entropy (hard labels). An optional post-QAT KD stage further recovers accuracy.
EvaluationAgent: A deterministic component that evaluates all checkpoints on a fixed validation manifest to ensure fair comparison of Top-1 accuracy and Relative Bit-Operations (Rel. BOPs).

2.2 Workflow

Profiling: Measure sensitivity and cost distribution.
Iterative Pruning: The PrunerAgent proposes a pruning strategy; the model is pruned; the FineTuningAgent recovers accuracy; the loop repeats until the target reduction is met.
Quantization: The QuantAgent assigns bit-widths and trains the model with QAT and KD.
Final Evaluation: The EvaluationAgent reports the final accuracy and compression metrics.

3. Key Contributions

Profiling-Grounded Decision Making: APQF is the first framework to use measured, empirical sensitivity data (via DepGraph sweeps) to drive LLM-based decisions for both pruning ratios and bit-width assignments, moving away from static heuristics.
LLM-Guided Adaptive Recovery: The framework automates the selection of recovery strategies (PEFT vs. full fine-tuning) and hyperparameters based on the specific accuracy drop observed at each stage.
Architecture-Agnostic Pipeline: The system operates seamlessly across both Convolutional Neural Networks (ResNet, VGG) and Vision Transformers (ViT, DeiT, Swin) without architecture-specific rules.
Data Efficiency: By pruning a pretrained model and recovering via distillation/PEFT rather than re-optimizing the entire network from scratch, APQF achieves high compression with significantly less training data than joint optimization methods.
LLM-Agnostic Design: The framework is not tied to a specific LLM; it functions with a range of planners from free open-weight models to premium commercial ones via a standard API.

4. Experimental Results

The authors evaluated APQF on ImageNet-1k and CIFAR-10 using ResNet, VGG7, ViT, DeiT, and Swin architectures.

ImageNet-1k (Constrained Budget): Under a limited budget of 200k images, APQF reduced compute to 5.6%–7.7% of the original bit-operations (13–18x reduction) while maintaining accuracy close to the baseline. In contrast, the joint optimization method GETA failed to recover accuracy under the same budget, dropping to 51–59% Top-1.
ImageNet-1k (Full Data): Even with full training data, APQF achieved comparable accuracy to GETA but with roughly 3x lower relative BOPs, due to its ability to combine structured pruning with aggressive mixed-precision quantization.
CIFAR-10: APQF compressed four of five architectures more aggressively than GETA while retaining higher accuracy. Notably, DeiT-Tiny, ResNet-50, and Swin-Tiny achieved higher accuracy than their uncompressed originals after compression.
VGG7 (CIFAR-10): APQF achieved 93.15% accuracy using only 0.41% of the baseline bit-operations. This was the only method at this compression level to improve upon its full-precision baseline.
Ablation Studies:
- Uniform vs. Adaptive: Removing the LLM's per-layer decisions (using uniform ratios/bit-widths) resulted in the largest accuracy drops, confirming the value of adaptive compression.
- Profiling Importance: Withholding profiling data from the LLM planner reduced accuracy across all models, proving that measured sensitivity is critical for decision quality.
- LLM Robustness: Testing six different LLM planners (including free models like DeepSeek and Gemma) on Swin-Tiny showed consistent accuracy (97.4%–97.9%), demonstrating the framework does not depend on a single premium model.

5. Significance and Claims

The paper claims that APQF represents a shift from manual, heuristic-driven compression to an automated, evidence-based pipeline. Its primary significance lies in:

Bridging the Gap: It successfully combines LLM reasoning with rigorous, measured profiling to make compression decisions that are both architecture-agnostic and highly effective.
Efficiency: It demonstrates that high-compression models can be generated with limited data, making advanced compression accessible for scenarios where large-scale retraining is infeasible.
Scalability: By decoupling the planning logic from the specific LLM used, the framework is future-proof, capable of leveraging stronger planners as they become available without architectural changes.

The authors acknowledge limitations, including the current focus on image classification, the inherent stochasticity of LLM planning (mitigated by careful prompting), and the computational cost of running the profiling and multi-stage recovery loops on large models. However, they posit that the approach is extensible to segmentation, language models, and vision-language models.