Benchmarking Edge Inference Strategies for Deep Learning Models in Industrial Machine Vision

Imagine you're trying to get a message from a brain (a deep learning model) to a robot hand (an industrial machine) as fast as possible. In the world of industrial machine vision, speed is everything. If the robot is too slow, it misses a defect on a car part or fails to sort grapes correctly. Usually, sending this data to a giant cloud server is too slow or risky, so engineers try to run the "brain" right on the robot's local computer (the "edge").

But here's the twist: just because you have the brain doesn't mean you have the right delivery service. This paper acts like a massive race track test, comparing four different "delivery trucks" (software frameworks) to see which one gets the message to the robot fastest. The four trucks are PyTorch (the standard, reliable baseline), ONNX Runtime (the flexible all-rounder), OpenVINO (the specialist for Intel engines), and TensorRT (the speed demon for NVIDIA engines).

The researchers didn't just guess; they actually ran the same three types of "brains" on four different "chassis" (computers) and timed them down to the millisecond. Here is what the race revealed:

The CPU Race: It's All About the Engine Brand

When the computers used standard Intel processors (the kind found in many desktops), the results were surprisingly consistent. OpenVINO was the clear winner, acting like a turbocharger specifically designed for that engine.

• On the Intel desktops, OpenVINO was the fastest for every single model tested.
• For the YOLOv8 model (a fast object detector), OpenVINO shaved off a massive chunk of time. On the fastest Intel desktop (the i9-9820X), it finished in just 10.9 ms, while the standard PyTorch took 28.7 ms. That's a huge jump!
• ONNX Runtime usually came in second, and PyTorch was the slowest.

However, the race changed completely when they switched to a Jetson AGX Orin (a tiny, powerful computer used in robots, which uses ARM chips, not Intel). Suddenly, OpenVINO lost its magic.

• On this ARM-based machine, ONNX Runtime took the gold medal.
• The gap was shocking for the Grounding DINO model (a complex model that understands text and images). On the Jetson CPU, OpenVINO took a staggering 102,720 ms (over 100 seconds!), while ONNX Runtime finished in just 13,580 ms. That's more than seven times faster! This suggests that using the "wrong" delivery truck for your specific engine can be a disaster.

The GPU Race: The CNN vs. Transformer Showdown

When the researchers switched to powerful graphics cards (GPUs) from NVIDIA, the story got even more interesting. They tested two types of "brains":

1. CNNs (like YOLOv8 and UNet): These are the classic, reliable workhorses of computer vision.
2. Transformers (Grounding DINO): These are the newer, flashier models that can understand language.

For the classic workhorses (YOLOv8 and UNet):
TensorRT was the undisputed champion. It was like having a Formula 1 car on a track.

• On the desktop GPUs, TensorRT crushed the competition. For YOLOv8 on an RTX 2080 Ti, TensorRT finished in 2.100 ms, while PyTorch took 5.270 ms.
• Even on the tiny Jetson GPU, TensorRT was the fastest, finishing YOLOv8 in 10.57 ms compared to PyTorch's 20.84 ms.
• The paper suggests that for these specific types of models, TensorRT is the best choice for NVIDIA hardware.

For the flashy new model (Grounding DINO):
Here is where the paper throws a curveball. The "speed demon" (TensorRT) did not win.

• On the desktop NVIDIA GPUs, the standard PyTorch was actually the fastest, providing the lowest inference time in all tested cases.
• TensorRT was actually slower than PyTorch, taking between 1.6 and 2.1 times longer to finish the job.
• For example, on the RTX 6000, PyTorch took 7.780 ms, while TensorRT was significantly slower.
• The only exception was the Jetson GPU, where TensorRT did beat PyTorch slightly (1082.7 ms vs 1290.8 ms), but it was still much slower than the desktop results.

The Big Takeaway

The main lesson from this paper is that there is no single "best" software for everyone. The paper suggests that your choice depends entirely on two things: what kind of computer you have and what kind of model you are running.

• If you have an Intel CPU, use OpenVINO.
• If you have an ARM CPU (like Jetson), use ONNX Runtime.
• If you have an NVIDIA GPU and are running a classic model (like YOLO or UNet), use TensorRT.
• BUT, if you have an NVIDIA GPU and are running a newer text-based model (like Grounding DINO), the paper suggests you might be better off sticking with the standard PyTorch on desktop GPUs, because the fancy optimizations in TensorRT didn't help and actually slowed things down.

The researchers measured these times over and over (1,000 runs per test!) to be sure. They didn't just simulate the results; they ran them on real hardware. So, if you are building an industrial robot, don't just grab the "fastest" software you hear about. Check your engine and your brain type first, or you might end up with a robot that moves in slow motion!