Benchmarking Deep Learning Models for Object Detection on Edge Computing Devices

This paper benchmarks state-of-the-art object detection models across various edge devices to analyze the trade-offs between accuracy, inference speed, and energy efficiency, ultimately providing practical guidance for selecting optimal model-device combinations for real-time applications.

The Gist

Imagine you are trying to build a security camera system for a small shop, but you can't plug it into a massive, expensive cloud server. Instead, you need the camera to "think" and spot intruders right on the spot, using a tiny, battery-powered computer. This is the world of Edge Computing: doing the heavy lifting locally rather than sending data to the cloud.

This paper is like a car review for tiny computers, but instead of testing how fast they drive, the authors tested how well they can "see" and identify objects (like people, cars, or animals) using different types of AI software.

Here is the breakdown of their experiment in simple terms:

The Contenders: The "Brains" (AI Models)

The researchers tested three different families of AI "brains" designed to spot objects. Think of these as different types of detectives:

1. YOLOv8 (You Only Look Once): These are the high-performance detectives.
   • The "Medium" version: A senior detective who is incredibly accurate but takes a long time to think and gets tired (uses a lot of battery) quickly.
   • The "Nano" and "Small" versions: Junior detectives who are faster and use less energy but might miss a few details.
2. SSD (Single Shot Detector): These are the sprinters.
   • They are very fast and use very little energy, but they aren't as good at spotting tricky or small objects. They are like a security guard who runs a quick patrol but might miss a sneaky thief.
3. EfficientDet Lite: These are the balanced detectives. They try to find a middle ground between speed and accuracy.

The Race Track: The "Muscles" (Edge Devices)

The authors tested these detectives on different types of tiny computers, which act as the bodies for the brains:

• Raspberry Pi (Models 3, 4, and 5): These are like the "Swiss Army Knives" of the computing world. They are cheap, small, and popular. The authors tested them both on their own and with a special USB stick attached (called a TPU) that acts like a turbocharger to help them think faster.
• NVIDIA Jetson Orin Nano: This is the "Sports Car" of the group. It's more expensive and powerful, designed specifically for heavy AI tasks.

The Race Results: Speed, Battery, and Accuracy

The researchers ran a marathon where they asked each computer to identify objects in thousands of photos. They measured three things:

1. How long it took to spot an object (Inference Time).
2. How much battery it used per photo (Energy Consumption).
3. How many objects it actually found correctly (Accuracy/mAP).

Here is what they found:

• The "Fast & Frugal" Winner: The SSD models were the clear winners for speed and battery life. They were like a marathon runner who eats very little and runs fast, but they weren't the best at spotting every single detail.
• The "Accurate but Hungry" Winner: The YOLOv8 Medium model was the most accurate detective, finding the most objects correctly. However, it was slow and ate up a lot of battery, like a luxury car that gets poor gas mileage.
• The "Turbocharger" Effect: When they added the TPU accelerator (the USB stick) to the Raspberry Pis, it was like giving a bicycle a jet engine.
  • For the SSD and EfficientDet models, the TPU made them incredibly fast and efficient without hurting their accuracy.
  • However, for the YOLOv8 models, the TPU forced them to shrink their "brain" (compress the model) to fit. This made them faster, but they became less accurate, like a senior detective forced to wear a blindfold to run faster.
• The "Sports Car" Champion: The Jetson Orin Nano was the overall champion. It was the fastest and most energy-efficient for the heavy-duty YOLOv8 models. It could handle the big, accurate models without slowing down or draining the battery too fast.

The Big Takeaway

There is no single "perfect" choice. It depends on what you need:

• If you need maximum speed and battery life (like a drone flying for hours), you should pick the SSD model on a Raspberry Pi with a TPU.
• If you need maximum accuracy (like a self-driving car that must see every pedestrian) and have a powerful device, the Jetson Orin Nano running YOLOv8 is the best bet.
• If you are on a budget and need a balance, the Raspberry Pi 4 or 5 with EfficientDet is a solid middle ground.

In short, the paper teaches us that building smart, local AI is a balancing act. You have to choose between how fast you want the computer to be, how much battery it can spare, and how smart it needs to be. There is no free lunch, but knowing these trade-offs helps you build the right system for your specific job.

Technical Summary

1. Problem Statement

Modern applications, particularly autonomous vehicles and real-time surveillance, require deploying deep learning object detection algorithms on resource-constrained edge devices. While cloud computing offers high performance, it introduces latency and dependency issues. Edge computing solves this but presents significant challenges:

• Resource Constraints: Edge devices (e.g., Raspberry Pi, Jetson) have limited CPU/GPU power, memory, and battery life.
• The Trade-off Dilemma: There is a lack of comprehensive understanding regarding the trade-offs between accuracy (mAP), inference speed (latency), and energy efficiency across different models and hardware configurations.
• Gap in Literature: Existing studies often focus on specific models or lack simultaneous evaluation of energy consumption, inference time, and accuracy across a wide range of modern models (like YOLOv8) and diverse hardware (including TPUs and newer Jetson modules).

2. Methodology

Experimental Setup

The authors conducted a comprehensive benchmarking study involving:

• Edge Devices:
  • Raspberry Pi Series: Pi 3 Model B+, Pi 4 Model B, and Pi 5.
  • Accelerators: Google Coral USB Accelerators (Edge TPU) attached to the Pi models.
  • High-Performance Edge: NVIDIA Jetson Orin Nano (4 GB RAM).
• Object Detection Models:
  • YOLOv8: Nano, Small, and Medium variants.
  • EfficientDet Lite: Lite0, Lite1, and Lite2.
  • SSD: SSD MobileNet V1 and SSDLite MobileDet.
• Frameworks & Deployment:
  • PyTorch: Used for native YOLOv8 deployment on Pis.
  • TensorFlow Lite (TFLite): Used for models on Pis with TPUs (models were compressed from 640x640 to 320x320 input resolution to fit TPU constraints).
  • TensorRT: Used for optimizing models on the Jetson Orin Nano.
  • Web Service: Models were deployed as Flask-API web services to simulate real-world request handling.

Evaluation Metrics

The study measured three critical metrics:

1. Inference Time: Time taken to process an image (excluding pre/post-processing), measured in milliseconds.
2. Energy Consumption: Calculated as Energy per Request (excluding base energy).
   • Formula: $E_{excR} = \frac{TE - BE}{NR}$
   • Where $TE$ is Total Energy, $BE$ is Base Energy (idle), and $NR$ is Number of Requests.
3. Accuracy: Mean Average Precision (mAP) evaluated on the COCO validation dataset (5,000 images) using the FiftyOne tool.

Automation

• Locust: Used to generate automated, sequential HTTP requests to the API endpoints for 5-minute intervals to measure throughput and energy.
• Power Meter: A UM25C USB power meter with Bluetooth connectivity measured real-time energy consumption.

3. Key Contributions

1. Comprehensive Benchmarking: A unique evaluation of state-of-the-art models (YOLOv8, EfficientDet Lite, SSD) across a diverse set of hardware (Pi 3/4/5, TPUs, Jetson Orin Nano).
2. Holistic Metrics: Simultaneous measurement of Accuracy, Latency, and Energy Efficiency (per request), providing a multi-dimensional view of performance.
3. Framework Optimization: Demonstrated the impact of different deployment frameworks (PyTorch, TFLite, TensorRT) and input resolution compression on TPU hardware.
4. Practical Guidance: Provided actionable insights for practitioners on selecting the optimal model-device pair based on specific application constraints (e.g., battery life vs. real-time speed).

4. Key Results

Energy Consumption

• Most Efficient: SSD MobileNet V1 consistently consumed the least energy per request across all devices.
• Least Efficient: YOLOv8 Medium had the highest energy consumption.
• Device Performance:
  • Jetson Orin Nano was the most energy-efficient device for handling requests, despite having the highest idle power consumption.
  • TPU Impact: Adding a TPU reduced the energy consumption per request for all models but increased the device's base idle energy consumption (by 9% to 46% depending on the Pi model).
  • Generational Gap: Pi 4 and Pi 5 were more energy-efficient than Pi 3.

Inference Time (Speed)

• Fastest Models: SSD MobileNet V1 was the fastest model across all platforms.
• Slowest Models: YOLOv8 Medium was consistently the slowest.
• Hardware Acceleration:
  • TPUs: Significantly reduced inference times (e.g., SSD_v1 on Pi 3+TPU dropped from 427ms to 61ms).
  • Jetson Orin Nano: Achieved the lowest absolute inference times (e.g., YOLOv8 Nano at 16ms), outperforming even TPU-accelerated Pis.

Accuracy (mAP)

• Highest Accuracy: YOLOv8 Medium achieved the highest mAP (44 on standard Pis).
• Lowest Accuracy: SSD MobileNet V1 had the lowest mAP (19).
• TPU Impact on Accuracy:
  • EfficientDet & SSD: Accuracy remained stable when deployed on TPUs.
  • YOLOv8: Accuracy significantly dropped on Pis with TPUs (e.g., YOLOv8 Nano dropped from 31 mAP to 16 mAP) due to the required model compression (reducing input resolution from 640x640 to 320x320).
• Jetson Impact: Jetson Orin Nano maintained high accuracy for YOLOv8 but showed a slight decrease in mAP for SSD and EfficientDet models compared to Pis.

Trade-off Analysis (Pareto Front)

• SSD Models: Show a linear correlation between energy and time. Jetson Orin Nano and Pi 5+TPU formed the "Pareto front" (best balance).
• YOLOv8: Jetson Orin Nano emerged as the superior choice, offering the best balance of speed, energy, and accuracy without the accuracy penalty seen on TPUs.

5. Significance and Conclusion

The paper concludes that there is no single "best" configuration; the choice depends on the specific application requirements:

• For Maximum Accuracy: YOLOv8 Medium on Jetson Orin Nano is the optimal choice.
• For Maximum Speed/Energy Efficiency: SSD MobileNet V1 on Jetson Orin Nano or Pi 5 with TPU is preferred.
• Critical Insight: While Edge TPUs drastically improve speed and reduce energy per request, they can severely degrade the accuracy of complex models like YOLOv8 due to input resolution constraints. Therefore, for high-accuracy YOLOv8 deployments, GPU-based edge devices (Jetson) are superior to TPU-based setups.

This study provides a vital reference for researchers and engineers developing edge AI solutions, highlighting the necessity of balancing hardware capabilities, model architecture, and deployment frameworks to meet real-world constraints.