💻 computer science

Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers

This paper proposes a multi-dimensional benchmarking methodology to evaluate the performance and efficiency of running large language models on hardware-accelerated single-board computers, providing practical guidance for deploying generative AI in privacy-sensitive and connectivity-limited edge environments.

Original authors: Harri Renney, Fouad Trad, Michael Mattarock, Zena Wood

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harri Renney, Fouad Trad, Michael Mattarock, Zena Wood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart robot brain (a Large Language Model, or LLM) that can write stories, solve problems, and answer questions. Usually, this brain lives in a giant, expensive data center far away in the "cloud." To use it, you have to send your questions over the internet and wait for an answer.

But what if you are in a place with no internet, or where sending data out is a security risk (like a secret military base or a remote drone)? You need that brain to live right there with you, on a small, cheap computer. This is called "Edge Computing."

This paper is like a shopping guide for building a local robot brain. The authors tested four different small computers (Single-Board Computers) to see which one is best at running these smart brains without needing a massive power plant or a huge suitcase to carry them.

Here is how they did it and what they found, using simple analogies:

The Problem: The "Cloud" vs. The "Backpack"

Running a smart brain in the cloud is like ordering a pizza delivered from a restaurant across town. It's great if you have a fast road (internet), but if the road is blocked, or if you can't let the delivery driver see your secret address (privacy), you're stuck.

The authors wanted to know: Can we bake the pizza in our own kitchen?
They tested small computers that fit in your hand (or even smaller) to see if they could run these smart brains fast enough to be useful.

The Test Kitchen: Four Different "Ovens"

The researchers set up four different "ovens" (hardware setups) to bake the pizza (run the AI):

The Tiny Brick (M5Stack): A tiny, credit-card-sized computer with a special helper chip (NPU) built-in. It's the size of a small matchbox.
The Standard Board (Raspberry Pi 5): A popular, affordable computer board, but it tries to do everything with its main brain (CPU) alone.
The Upgraded Board (Raspberry Pi 5 + HAT): The same standard board, but with a special "speed booster" card (Hailo NPU) attached to it.
The Powerhouse (Jetson Orin Nano): A more expensive, powerful board with a high-end graphics card (GPU) built-in.

The New Way of Measuring Success

Usually, people just ask: "How fast is it?" (Tokens per second). But the authors realized that for small, portable devices, speed isn't the only thing that matters. They introduced two new ways to measure success:

The "Suitcase Density" Test (Throughput Density):
Imagine you are packing for a hiking trip. You don't just want the most powerful tent; you want the most powerful tent that fits in the smallest backpack. The authors measured how much "smart thinking" a device could do per cubic inch of its size.
- The Winner: The tiny M5Stack was the champion here. Even though it wasn't the fastest overall, it packed the most computing power into the smallest space.
The "Battery Life" Test (Energy per Million Tokens):
Imagine you are running a marathon. You don't just want to run fast; you want to run fast without getting tired out. They measured how much electricity it took to generate a million words.
- The Winner: The M5Stack and the Jetson GPU were the most energy-efficient. The standard Raspberry Pi (without a booster) was like a runner who got tired very quickly, using a lot of energy for very little speed.

The Results: What Worked Best?

The "Speed Demon": The Jetson Orin Nano was the fastest overall. If you need the most answers per second and have a bit of room and power, this is the one to pick.
The "Space Saver": The M5Stack (the tiny brick) was the best for tight spaces. It proved you can fit a smart brain into something as small as a matchbox.
The "Smart Upgrade": Adding the Hailo speed booster to the Raspberry Pi made it much better. It became much faster and used much less energy than the Raspberry Pi alone.
The "Don't Do This": Trying to run these smart brains on the standard Raspberry Pi without any special booster chips was very inefficient. It was slow and ate up a lot of power, like trying to run a marathon in heavy boots.

Why Does This Matter? (The Paper's Specific Claims)

The authors say these findings are crucial for three specific real-world scenarios mentioned in the paper:

Unmanned Vehicles (Drones): Drones have tiny batteries. They need a smart brain that doesn't drain the battery. The paper shows you can put a smart brain on a drone that is small enough to fly and efficient enough to not crash the drone due to low power.
Portable, Rugged Operations: Think of soldiers or workers in tough environments carrying gear. They need computers that fit in a small, tough case. The "Suitcase Density" test shows which computers fit best in these tight, rugged boxes.
Satellite Ground Stations: People are building small, portable stations to talk to satellites in remote places. They need to process data locally without waiting for a connection to the main office. The paper shows these small computers can handle the job, especially if they use the special booster chips to save power.

The Bottom Line

You don't need a giant data center to have a smart AI anymore. You can put it in a small box. However, you can't just grab any small computer; you need the right combination of a small computer and a special "helper" chip (accelerator) to make it fast and energy-efficient.

If you need maximum speed, get the Jetson.
If you need maximum smallness, get the M5Stack.
If you want a good balance, add a booster chip to a standard board.

This paper gives you the map to choose the right tool so you can run smart AI anywhere, even where the internet doesn't reach.

1. Problem Statement

The deployment of Large Language Models (LLMs) remains predominantly cloud-centric, creating critical bottlenecks for Operational Technology (OT) and defense environments. These challenges include:

Data Privacy & Security: Sensitive data in critical infrastructure (energy, defense, manufacturing) often cannot leave controlled network perimeters.
Latency & Connectivity: Reliance on stable internet connections is a liability in remote or contested environments (e.g., unmanned vehicles, satellite ground stations).
Evaluation Gaps: Existing edge benchmarking tools (e.g., DeepEdgeBench, LEAF) suffer from three main limitations:
1. They rely heavily on CPU-only inference or legacy GPUs, ignoring modern dedicated edge accelerators (NPUs).
2. They lack coverage of genuine Single-Board Computers (SBCs) suitable for IoT.
3. They use generic evaluation metrics that fail to account for the multi-dimensional constraints of edge deployment (physical size, power budget, and energy efficiency).

2. Methodology

The authors propose a structured, multi-dimensional benchmarking framework to evaluate LLM inference on four distinct IoT-suitable edge platforms.

Experimental Setup:

Hardware Platforms (5 Configurations):
1. M5Stack LLM (Module M140): Features an AX630C SoC with an integrated NPU (3.2 TOPS). Smallest form factor ( $54 \times 54 \times 13$ mm).
2. Raspberry Pi 5: CPU-only (Broadcom BCM2712).
3. Raspberry Pi 5 + AI HAT+: Adds a Hailo-10H NPU (40 TOPS).
4. NVIDIA Jetson Orin Nano Super: Features an Ampere GPU (67 TOPS).
Models: A selection of compact, distilled LLMs (0.5B to 3B parameters), including DeepSeek-R1-Distill-Qwen, Qwen 2.5 (Instruct/Coder), and Llama 3.2.
Inference Configuration:
- Quantization: All models run at 4-bit quantization (Q4_K_M) to minimize memory footprint.
- Runtimes: Ollama (for CPU/GPU), Hailo Ollama (for NPU on Pi 5), and StackFlow (for M5Stack).
- Workload: Fixed prompts generating 100 tokens to measure sustained performance.

Evaluation Metrics:
The paper introduces a multi-dimensional framework moving beyond simple latency/accuracy:

Raw Performance: Token throughput (tokens/s) and Time-to-First-Token (TTFT).
Energy Efficiency: Energy consumption per million tokens (MJ/Mtok).
Composite Metrics (Novel Contributions):
- Throughput Density ( $Tps/m^3$ ): Token throughput normalized by device volume. Crucial for space-constrained deployments (e.g., drones, wearables).
- Energy per Million Tokens (MJ/Mtok): Total energy cost for generation, critical for battery-operated systems.

3. Key Contributions

Gap Analysis: Identified the absence of multi-dimensional frameworks that jointly evaluate inference performance, hardware efficiency, and physical constraints for modern SBCs with dedicated accelerators.
Novel Metrics: Introduced Throughput Density and Energy per Million Tokens to enable hardware selection based on specific deployment constraints (space vs. power vs. speed).
Comprehensive Benchmarking: Provided the first comparative analysis of modern edge accelerators (Hailo-10H, NVIDIA Ampere, AX630C NPU) against CPU-only baselines across multiple model families.

4. Key Results

Hardware Acceleration Impact:
- NPU (Hailo-10H): Delivered 9.5x to 40x energy efficiency gains over CPU-only inference on the Raspberry Pi 5. Throughput increased from ~0.3 tok/s (CPU) to ~6.3 tok/s (NPU).
- GPU (Jetson Orin): Provided a more modest 1.2x to 1.8x efficiency gain over its own CPU baseline but offered the highest absolute throughput (up to ~10 tok/s).
Throughput Density:
- The M5Stack AX630c dominated this metric, achieving >90,000 Tps/m³ due to its tiny form factor, making it ideal for ultra-compact, ruggedized applications.
- The Jetson Orin, while powerful, had the lowest density due to its larger physical volume.
Energy Efficiency:
- CPU-only inference was an order of magnitude less efficient (33–76 MJ/Mtok) compared to accelerated platforms (0.5–5.5 MJ/Mtok).
- The M5Stack and Jetson GPU were the most energy-efficient, consuming ~0.5–1.7 MJ/Mtok.
- The RPi 5 + Hailo showed variable efficiency depending on the model architecture, suggesting NPU performance is more sensitive to model structure than GPUs.

5. Significance and Implications

The findings offer practical guidance for deploying Generative AI in critical, privacy-sensitive, and connectivity-limited environments:

Resilient Distributed Systems: Low-power edge AI (5–25W range) enables decentralized architectures where individual nodes act as intelligent agents. This reduces reliance on central command, enhancing resilience against network jamming or cyberattacks.
Energy-Constrained Autonomy: For drones and autonomous systems with tight energy budgets, the shift from CPU to NPU/GPU inference significantly extends operational time while enabling local decision-making and real-time data summarization.
Satellite Ground Stations: The results validate that 1.5B parameter models can run on portable SBCs (like RPi 5 + Hailo) with high efficiency. This allows ground stations to perform intelligent tasks (command parsing, telemetry reasoning) locally without needing high-bandwidth links to central facilities.

Conclusion:
The paper demonstrates that hardware-accelerated SBCs are now viable for running LLMs locally. The choice of hardware should not be based solely on raw speed; rather, it must be guided by the specific trade-offs between physical footprint (favoring M5Stack), absolute throughput (favoring Jetson), and power efficiency (favoring M5Stack or Jetson GPU), depending on the specific operational constraints of the IoT deployment.