A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources

This paper presents a systematic evaluation of on-device Large Language Models across various sizes and quantization methods, revealing that heavily quantized larger models outperform smaller high-precision ones beyond a 3.5 bits-per-weight threshold while identifying a shift from communication to computational constraints as model size decreases.

The Gist

Imagine you have a brilliant, world-class chef (a Large Language Model or LLM) who can write stories, solve math problems, and code software. Usually, this chef lives in a massive, high-tech kitchen in the cloud (a data center), and you send your orders there via the internet. The chef cooks, and sends the meal back to your phone.

But what if you want this chef to cook right in your own kitchen (on your laptop or phone)? This is called "On-Device" AI. It's great for privacy (no one else sees your recipes) and speed (no waiting for the internet), but your kitchen is small, and you don't have a million-dollar stove. You have limited space and limited energy.

This paper is like a comprehensive guidebook for fitting a giant chef into a tiny kitchen. The authors tested many different ways to shrink the chef down so they can fit, without losing their cooking skills.

Here is the breakdown of their findings using simple analogies:

1. The "Compression" Problem: How to Shrink the Chef

To fit a huge chef into a small kitchen, you have to pack them tighter. In AI terms, this is called Quantization.

• The Analogy: Imagine the chef's knowledge is written in a giant, heavy encyclopedia. To fit it in your pocket, you have to photocopy it.
  • High Quality (FP16): You keep the original, crisp photos. It's heavy and takes up a lot of space.
  • Low Quality (2-bit): You turn the photos into tiny, blurry dots. It fits easily, but the chef might forget how to chop an onion.
  • The "Sweet Spot" (4-bit): The authors found that if you compress the chef just enough (about 4 bits of detail per "word" of knowledge), they still cook almost as well as the original, but they fit perfectly in your pocket.

2. The Big vs. Small Chef Discovery

The team tested chefs of all sizes, from a "junior cook" (0.5 Billion parameters) to a "Master Chef" (14 Billion parameters).

• Finding: A small, high-quality chef is often worse than a big, slightly compressed chef.
• The Metaphor: Imagine a tiny, perfectly detailed toy car vs. a real, slightly dented car. The real car (even if it's a bit rough) can still drive you to the store. The toy car is perfect in detail but can't actually drive.
• The Lesson: If you have a big model (like 14B), you can compress it heavily, and it will still outperform a tiny model that hasn't been compressed at all. There is a "tipping point" around 3.5 bits where the quality starts to crash.

3. The Traffic Jam: Speed vs. Size

When the chef cooks, there are two steps:

1. Reading the Order (Prefill): Reading your prompt.
2. Cooking the Meal (Decode): Generating the answer word by word.

The paper discovered a fascinating shift in what slows the process down:

• Small Models (The Tiny Kitchen): The bottleneck is the chef's hands. The chef is so small that they are constantly thinking and calculating. The kitchen is fast, but the chef is slow.
• Large Models (The Big Kitchen): The bottleneck is delivering ingredients. The chef is huge, and the kitchen is full of ingredients (data). The chef is ready to cook, but they are waiting for the ingredients to be brought from the pantry to the counter.
• The Metaphor:
  • Small Model: Like a single person trying to carry a heavy box. They are tired (computation bound).
  • Large Model: Like a team of people waiting for a delivery truck. The truck is slow (communication/memory bound).

4. The "Unpacking" Surprise

The paper looked at different ways to compress the data (called q4_k vs q4_0).

• The Analogy: Imagine packing a suitcase.
  • Method A: You fold clothes neatly but use a complex folding pattern that takes time to undo.
  • Method B: You just roll the clothes. It's slightly less space-efficient, but you can unpack them instantly.
• The Finding: Sometimes, a method that looks "less efficient" on paper actually runs faster because the computer's brain (the CPU) can handle the "unfolding" process more easily. It's not just about how small the suitcase is; it's about how easy it is to open it.

5. Power and Memory: The Battery Drain

• Memory: The more you compress, the less space the model takes up. This is straightforward.
• Power (Battery): This was tricky.
  • If the model is tiny, the computer works hard to calculate, draining the battery.
  • If the model is huge and highly compressed, the computer spends most of its time waiting for data to move around (memory transfer). It's like a runner standing still waiting for a bus. The runner isn't tired, but the bus is slow.
  • Result: Extremely compressed large models actually use less power per task because the computer spends more time "idling" waiting for data than actually crunching numbers.

The Final Takeaway: How to Choose?

The authors give you a simple menu for choosing your AI:

1. If you need high accuracy (e.g., medical advice, complex coding): Get a Large Model (like 14B) and compress it to 4-bit. It's the best balance of skill and size.
2. If you need speed and low battery usage (e.g., quick chat, simple summaries): Get a Small Model (like 1B or 3B) and compress it to 4-bit. It's fast and doesn't drain your battery.
3. Avoid the extremes: Don't go below 3-bit (the chef forgets everything) or stick to 8-bit (the chef is too heavy for your pocket).

In summary: You don't need a supercomputer to run AI anymore. You just need to pick the right-sized chef and pack them efficiently. The paper proves that with the right packing (quantization), your laptop can be a powerful, private AI assistant.

Technical Summary

Here is a detailed technical summary of the paper "A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources."

1. Problem Statement

The deployment of Large Language Models (LLMs) on edge devices (e.g., consumer laptops) offers significant privacy benefits by eliminating remote data transmission. However, this transition faces critical hurdles due to limited computational resources, memory constraints, and power budgets. While various compression techniques like quantization exist, there is a lack of a comprehensive, systematic evaluation framework that holistically assesses the trade-offs between:

• Model Capability: Task accuracy across diverse benchmarks.
• Deployment Efficiency: Inference throughput (tokens per second) and latency.
• System Resource Utilization: CPU power consumption, memory footprint, and contention.

Existing works often focus on isolated metrics (e.g., only accuracy or only power) or specific subsets of models, failing to capture the complex, non-linear relationships between model size, quantization precision, and hardware constraints in real-world edge scenarios.

2. Methodology

The authors propose a tripartite evaluation framework and conduct extensive empirical studies using the following setup:

• Hardware Platforms:
  • Consumer Device: A laptop with a 12-core Intel Core i7-1360P CPU and 16GB DDR5 RAM (representing typical user hardware).
  • Testbed: A Linux server with an AMD EPYC 9654 CPU and 64GB RAM, configured via Docker to simulate heterogeneous resource constraints (varying CPU core counts and execution quotas).
• Models: 11 LLMs across three families (Qwen 2.5, Qwen 3, Llama 3) ranging from 0.5B to 14B parameters.
• Quantization Methods: Seven Post-Training Quantization (PTQ) methods implemented in llama.cpp, covering effective bit-widths from ~2 bits to ~8 bits (specifically: q2_k, q3_k, q4_k, q4_0, q5_k, q5_0, q8_0).
• Evaluation Dimensions:
  1. Model Capability: Evaluated on five open-source benchmarks: GSM8K (math), HellaSwag (commonsense), MMLU (general knowledge), HumanEval (code), and TruthfulQA (factuality).
  2. Deployment Efficiency: Measured via Tokens Per Second (TPS) for both prefilling (prompt processing) and decoding (token generation) stages across varying input lengths (64–512 tokens) and output lengths (1024 tokens).
  3. Resource Utilization: Analyzed CPU power (via Windows Performance Recorder), memory footprint (Peak RSS), and energy consumption.

3. Key Contributions

1. Holistic Evaluation Framework: Introduced a multidimensional approach that simultaneously characterizes capability, efficiency, and resource contention, providing a realistic view of on-device LLM performance.
2. Comprehensive Empirical Study: Conducted a large-scale benchmark covering 11 models, 7 quantization schemes, 5 datasets, and 4 input token lengths, filling a gap in existing literature.
3. Analytical Insights & Guidelines: Identified specific bottlenecks (compute vs. communication) based on model size and quantization, offering actionable guidelines for optimizing LLMs in edge environments.

4. Key Results and Findings

A. Model Capability

• Size Matters: Larger models consistently outperform smaller ones across all benchmarks.
• Quantization Resilience: Large models (e.g., 14B) demonstrate superior resilience to aggressive quantization. For instance, a 14B model at 3-bit (q3_k) maintains high accuracy, whereas a 1.5B model suffers significant degradation at the same precision.
• The 3.5 BPW Threshold: There is a performance threshold at approximately 3.5 effective bits-per-weight (BPW). Below this, performance degrades precipitously, especially for smaller models and complex tasks like code generation (HumanEval) and math (GSM8K).
• Model Family Differences: Qwen 2.5 models generally exhibit better resilience to quantization compared to Llama 3 models of similar sizes.

B. Deployment Efficiency (Throughput)

• Bottleneck Shift: The primary performance bottleneck shifts based on model size:
  • Small Models (<0.5B–1B): Compute-bound. Performance is limited by CPU arithmetic capacity. Increasing CPU cores yields linear throughput gains.
  • Large Models (>1B): Communication-bound. Performance is limited by memory bandwidth (data transfer). Increasing CPU cores yields diminishing returns; memory bandwidth and cache efficiency are critical.
• Quantization Impact:
  • Throughput generally decreases as BPW increases (due to larger data movement).
  • Implementation Matters: For the same BPW, specific quantization algorithms perform differently. For example, q4_0 often outperforms q4_k in prefilling due to streamlined unpacking, while q5_k can outperform q5_0 when hardware supports VNNI (Vector Neural Network Instructions) due to better parallelization.
• Resource Contention: Decoding throughput is highly sensitive to CPU quota restrictions (context switching and cache misses), whereas prefilling is more robust.

C. System Resource Utilization

• Memory Footprint: Scales linearly with BPW and model size. However, specific formats (like q4_0) can exhibit anomalies where memory usage is higher than expected due to alignment or metadata overhead.
• Power Consumption:
  • Correlates with computational intensity (unpacking overhead) rather than just BPW. Methods with complex unpacking logic (e.g., q3_k, q5_k) can consume more power due to longer execution times.
  • Inverse Trend at Low Bits: At extremely low bit-widths (e.g., q2_k), larger models sometimes consume less power than smaller ones because the workload shifts to being memory-bound, and the reduced memory footprint improves cache hit rates, reducing CPU idle time.

5. Significance and Guidelines

The paper provides critical guidelines for deploying LLMs on edge devices:

1. Optimal Quantization Point: 4-bit quantization emerges as the "sweet spot," offering a 30–50% throughput acceleration with near-lossless accuracy compared to higher precision.
2. Model Selection Strategy:
   • For High Accuracy: Use large-scale models (e.g., 7B–14B) with moderate quantization (4-bit).
   • For Low Latency/Resource Constraints: Use compact models (e.g., <1B) with similar quantization schemes, as they are less communication-bound.
3. Hardware Optimization:
   • For large models, prioritize memory bandwidth optimization over raw CPU core count.
   • For small models, prioritize CPU arithmetic capacity and efficient instruction sets (e.g., AVX2/VNNI).
4. Algorithm Selection: The choice of quantization method (_k vs _0) significantly impacts performance depending on the hardware's instruction set support (e.g., VNNI).

Conclusion:
This study establishes that "one size does not fit all" for on-device LLMs. Effective deployment requires a nuanced balance where model size, quantization strategy, and hardware characteristics are co-optimized. The authors conclude that heavily quantized large models often outperform smaller, high-precision models, provided the bit-width remains above the critical ~3.5 BPW threshold.