🤖 AI

Quantization with Unified Adaptive Distillation to enable multi-LoRA based one-for-all Generative Vision Models on edge

This paper introduces QUAD, a unified framework that enables efficient multi-task Generative Vision Model inference on edge devices by treating LoRA weights as runtime inputs and employing quantization-aware training to achieve significant reductions in memory footprint and latency while maintaining high visual quality.

Original authors: Sowmya Vajrala, Aakash Parmar, Prasanna R, Sravanth Kodavanti, Manjunath Arveti, Srinivas Soumitri Miriyala, Ashok Senapati

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sowmya Vajrala, Aakash Parmar, Prasanna R, Sravanth Kodavanti, Manjunath Arveti, Srinivas Soumitri Miriyala, Ashok Senapati

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart artist living inside your smartphone. This artist is incredibly talented and can do many things: remove unwanted tourists from your vacation photos, change the background of a selfie to a beach, or turn a sketch into a realistic painting.

However, there's a problem. This artist is huge. To make them work, you usually need to carry a giant backpack full of heavy tools.

The Old Way: The "One Artist, One Backpack" Problem

In the past, if you wanted your phone to do three different tasks (like editing photos, generating art, and removing objects), you had to train three separate versions of this artist.

The Problem: To switch from "Photo Editor" mode to "Art Generator" mode, your phone had to unload the first heavy backpack and load a completely new, equally heavy one.
The Result: Your phone's storage (ROM) got clogged with duplicate copies of the artist's brain. Switching tasks was slow because the phone had to stop, pack up, and unpack new gear every time. It was like trying to change a car's engine every time you wanted to drive to the grocery store instead of the park.

The New Solution: "One Brain, Many Outfits"

The researchers at Samsung have invented a clever new system called QUAD (Quantization with Unified Adaptive Distillation). Here is how it works, using simple analogies:

1. The "Plug-and-Play" Outfit (LoRA as Input)

Instead of building a new artist for every task, they kept one single artist (the Foundation Model) and gave them a special wardrobe.

The Old Way: The artist's clothes were sewn permanently onto their body. To change style, you had to rebuild the whole person.
The New Way: The artist wears a basic, neutral suit. The "LoRA" weights are like magical, detachable outfits (a chef's apron, a painter's smock, a photographer's vest).
How it works: When you want to edit a photo, the phone instantly snaps on the "Editor Outfit." When you want to generate art, it snaps on the "Artist Outfit." The core brain never changes; it just swaps the tools it uses. This means you only need to store one version of the artist, saving massive amounts of space.

2. The "Universal Translator" (Unified Quantization)

Here is the tricky part: Digital phones (especially the chips inside them) speak a simplified language called "Quantization" (using fewer numbers to save memory).

The Problem: Usually, the "Editor Outfit" speaks a slightly different dialect than the "Artist Outfit." If you try to force them to speak the same simplified language, the artist might get confused and start drawing weird pictures.
The Solution (QUAD): The researchers created a "Universal Translator." They taught all the different outfits to speak the exact same simplified dialect before they ever left the factory.
The Analogy: Imagine training a group of actors who speak different accents (French, Spanish, Italian) to all perform a play using only hand gestures. They practice together until they all understand the gestures perfectly. Now, the director (the phone) can switch actors instantly without anyone getting confused or needing to relearn the script.

3. The "Lightweight Runtime" (The Delivery Truck)

Finally, they built a tiny, efficient delivery truck (the software stack) that runs on your phone.

Because the artist and the outfits are now perfectly compatible and speak the same language, the truck doesn't need to stop and reload the whole engine. It just swaps the outfit in a split second.

Why Does This Matter? (The Results)

By using this "One Brain, Many Outfits" system:

Storage: Your phone needs 6 times less space to store all these features. Instead of 15GB of clutter, it fits in 2.6GB.
Speed: Switching tasks is 4 times faster. You don't wait for the phone to "load" a new app; it just snaps on a new tool instantly.
Quality: The pictures still look amazing. The "Universal Translator" ensures the artist doesn't lose their talent just because they are speaking a simpler language.

In a Nutshell

This paper is about making your phone's AI smarter and lighter. Instead of carrying a library of heavy, separate books for every task, they created one master book with magnetic, swappable chapters that all speak the same language. This lets your phone do complex AI tasks instantly, without running out of memory or battery.

1. Problem Statement

The rapid integration of Generative AI (GenAI) features (e.g., image editing, object removal, stylization) into mobile applications faces significant hurdles when deploying Large Vision Models (LVMs) on resource-constrained edge devices (smartphones).

Resource Constraints: LVMs are computationally expensive and memory-intensive, making on-device inference difficult.
Deployment Inefficiency: Current mobile deployment pipelines typically compile separate model binaries for each task-specific Low-Rank Adapter (LoRA) combined with the foundation model. This results in:
- Redundant Storage: Multiple copies of the base model increase ROM usage.
- Runtime Overhead: Switching tasks requires reloading different binaries, causing latency.
- Quantization Incompatibility: When LoRAs are quantized independently, they often require different quantization parameters (scales and zero-points). This prevents them from sharing a single static inference graph on Neural Processing Units (NPUs), which typically require fixed quantization profiles.

2. Methodology

The authors propose a unified framework that enables multi-task GenAI inference using a single shared model on edge devices. The solution consists of three core components:

A. LoRA as Runtime Inputs (One-for-All Architecture)

Instead of merging LoRA weights into the model graph during compilation, the authors restructure the architecture to treat LoRA weights as runtime inputs.

Mechanism: The model graph is frozen and compiled once. During inference, specific LoRA weights (matrices $A$ and $B$ ) are streamed as input tensors and bound to the model via lightweight APIs.
Benefit: This allows dynamic task switching without recompilation or reloading the foundation model, enabling true "one-for-all" deployment.

B. QUAD Framework (Quantization with Unified Adaptive Distillation)

To address the incompatibility of quantization parameters across different LoRAs, the authors introduce QUAD.

Goal: Enforce a single, shared quantization profile (scale and zero-point) across the base model and all LoRA adapters.
Process:
1. Sensitivity Analysis: The system identifies the most "quantization-sensitive" LoRA (the one where accuracy drops most when perturbed). This LoRA serves as the anchor to define the fixed quantization parameters.
2. Unified Quantization: If all LoRAs are equally sensitive, a fallback "Unified-LoRA" approach computes global parameters based on merged weight distributions.
3. Knowledge Distillation: The other LoRAs are fine-tuned using knowledge distillation. A "Teacher" (full-precision model) guides a "Student" (quantized model with the anchor's parameters) to minimize reconstruction loss. This aligns the distributions of all LoRAs to the shared quantization profile without retraining the base model.

C. Compilation and Runtime Stack

Graph Optimization: The model is converted to an intermediate representation (ONNX) and then to a hardware-specific IR. Operations include scale folding, constant folding, and the insertion of placeholder inputs for LoRA weights.
Runtime Execution: A lightweight runtime stack manages the frozen base model and dynamically injects LoRA buffers. This supports features like OTA (Over-the-Air) updates for new LoRAs without changing the base model binary.

3. Key Contributions

Dynamic LoRA Injection: A novel architectural shift treating LoRA weights as runtime inputs rather than static graph components, enabling dynamic task switching on a single compiled binary.
QUAD Framework: A quantization-aware training strategy that uses knowledge distillation to align multiple independent LoRAs under a single quantization profile, making them compatible with fixed-parameter NPU hardware.
Unified Deployment Pipeline: A complete system spanning model restructuring, quantization, graph optimization, and a custom runtime stack compatible with mobile NPUs (Qualcomm, MediaTek, Samsung Exynos).
Scalability: The approach allows for the addition of new tasks via OTA updates without increasing the base model size or requiring recompilation.

4. Experimental Results

The system was evaluated on multiple chipsets (Qualcomm GS25, LSI GS25, MediaTek Tab S11) using Stable Diffusion 1.5 based models.

Memory Efficiency:
- Achieved up to 6× reduction in memory footprint compared to traditional multi-binary approaches.
- Example: A 1.4GB base model + 10 LoRAs (120MB each) would require ~15GB with separate binaries. The proposed method requires only 2.6GB (1 base + 1 LoRA loaded at a time).
Latency Improvements:
- Achieved up to 4× improvement in runtime latency.
- Eliminated the overhead of fetching and swapping full model graphs, saving ~1.5 seconds in end-to-end latency for task switching.
Accuracy Preservation:
- Maintained high visual quality across tasks (Image Transformation, Object Removal).
- Metrics: For Object Removal, the quantized model achieved an FID of 5.52, SSIM of 0.94, and PSNR of 33.04 compared to the full-precision server baseline.
- Ablation: The study confirmed that W8A16 (Weights INT8, Activations INT16) provided the best balance of accuracy and efficiency, though W8A8 showed promise with minor accuracy trade-offs.

5. Significance

This work bridges the critical gap between parameter-efficient adaptation (LoRA) and practical on-device deployment. By solving the quantization incompatibility and storage redundancy issues, it enables mobile devices to run multiple high-quality GenAI tasks simultaneously or switch between them instantly without cloud dependency. This paves the way for privacy-preserving, low-latency, and bandwidth-efficient Generative AI applications directly on consumer smartphones.