🤖 machine learning

Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders

This paper demonstrates that single-thread JPEG decoder microbenchmarks fail to predict real-world ML DataLoader performance across diverse CPU architectures, revealing that multi-threaded worker configurations and specific decoder behaviors often invert throughput rankings and highlighting torchvision and simplejpeg as the most robust choices for PyTorch workloads.

Original authors: Vladimir Iglovikov

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vladimir Iglovikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy restaurant kitchen. Your goal is to get delicious meals (data) from the pantry (storage) to the chefs (the AI model) as fast as possible. The "JPEG decoder" is the sous-chef whose only job is to unwrap the food packaging (decode the JPEG images) so the main chef can cook.

For years, people have tried to find the best sous-chef by running a simple test: How fast can one person unwrap one single package? They would time a single person, pick the fastest one, and hire them for the whole kitchen.

This paper argues that this simple test is misleading. Just because someone is the fastest at unwrapping a single package in a quiet room doesn't mean they are the best choice for a chaotic, busy kitchen with multiple staff members working together.

Here is the breakdown of the paper's findings using everyday analogies:

1. The "Solo Sprint" vs. The "Team Relay"

The paper tested 12 different "unwrapping tools" (software libraries like simplejpeg, torchvision, OpenCV, etc.) on five different types of computer processors (like Intel, AMD, and ARM chips).

The Old Way (Solo Sprint): They timed how fast each tool could decode one image by itself.
- Result: Some tools, like simplejpeg, were the clear winners.
The New Way (Team Relay): They simulated a real training environment where a "DataLoader" (the kitchen manager) sends images to multiple workers (processes) at once.
- Result: The rankings changed completely!
- The Twist: On some computers, a tool that came in 9th place in the solo sprint (like imageio) jumped to the top tier when working in a team. On others, the solo winner fell behind.

The Analogy: Imagine a runner who is the fastest sprinter in the world (Solo Winner). But when you put them in a relay race where they have to hand off a baton to three other people, they might be terrible at the handoff. The paper says: Don't hire the best sprinter if you need a relay team; hire the best team player.

2. The "Hardware Personality" Matters

The paper found that the "best" tool depends entirely on the type of computer processor you are using. It's not a one-size-fits-all answer.

The AMD Generation Gap: They tested two generations of AMD processors (Zen 4 and Zen 5).
- On Zen 4, adding more workers helped up to 4 people, but adding an 8th person actually slowed things down (like adding too many cooks to a small kitchen).
- On Zen 5, adding more workers kept helping all the way up to 8 people.
- Lesson: What works for one generation of hardware might be a disaster for the next.

3. The "Broken Package" Problem (Robustness)

In a real kitchen, sometimes a package is torn or labeled wrong. A good sous-chef should handle it gracefully.

Some tools were "strict": If they saw one weird image, they stopped and skipped it.
Some tools were "robust": They handled the weird image and kept going.
The Finding: Some of the fastest "strict" tools skipped a specific image in the test set. If you are training an AI, skipping data can be bad. The paper found that tools like torchvision and simplejpeg were the "robust" winners—they were fast and didn't skip any images.

4. The "ARM Penalty"

They tested a popular tool called TensorFlow.

On Intel and AMD computers, it was fast.
On ARM computers (often used in mobile devices and newer servers), it was significantly slower—about 40% slower than the best option.
Lesson: If you are using an ARM computer, don't assume TensorFlow is your best bet for image loading.

The Final Verdict: How to Choose?

The paper concludes that you cannot pick a tool just by looking at a "fastest single-thread" leaderboard. Instead, you need to look at the whole picture:

If you want the best average speed: Use torchvision. It was the most consistent top performer across the board.
If you want the safest "worst-case" speed: Use simplejpeg. It was the most reliable, never the slowest, and never skipped an image.
If you need a reliable backup: Use OpenCV. It wasn't always the #1, but it was always in the top 10% (above 90% of the winner) on every single computer tested.

The Big Takeaway:
Stop judging a library based on how fast it is in a vacuum. To build a good AI training pipeline, you must test how the tool performs when it's actually working in a team, on your specific type of computer, and handling messy real-world data. The "fastest" tool in a solo race might be the wrong choice for your team.

Technical Summary: Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders

Problem Statement
In machine learning infrastructure, image loading often becomes the bottleneck that caps training throughput, causing GPU idleness. The standard industry response is to benchmark JPEG decoders using single-process, single-thread microbenchmarks to identify the fastest decoder (images/second). This paper argues that this evaluation design encodes a false claim: that single-thread decoder throughput accurately identifies the optimal decoder for training data loaders. In reality, training pipelines operate within complex environments involving multi-process DataLoader workers, Python/C boundary effects, library thread pools, and specific CPU scheduling behaviors. A benchmark that isolates a single decoder call fails to capture these conditions, potentially leading to suboptimal library choices or overlooking robustness policies critical for operational stability.

Methodology
The authors audit the evaluation assumption by comparing a common single-thread protocol against a DataLoader-aware protocol across a controlled, matched environment.

Workload: The full ImageNet validation split (50,000 JPEG files) is decoded entirely from memory to isolate decode and loader overhead from disk or network latency.
Decoders: Twelve Python-accessible decode paths were evaluated: simplejpeg, turbojpeg, jpeg4py, kornia-rs, opencv, imagecodecs, pyvips, pillow, skimage, imageio, torchvision, and tensorflow.
Hardware: Five matched 16 vCPU Google Cloud instances representing different vendors, ISAs, and generations: Intel Emerald Rapids, AMD Zen 4, AMD Zen 5, ARM Neoverse V2 (Google Axion), and ARM Neoverse N1 (Ampere Altra).
Protocols:
1. Single-Thread: Memory-based decode throughput.
2. PyTorch DataLoader: Throughput measured at worker counts $\{0, 2, 4, 8\}$ .
3. Robustness: Accounting for skipped images (failures) during decoding.
Statistical Policy: The study uses raw per-run throughput samples stored in JSON rather than relying solely on table means. Practical significance thresholds are applied (1% for single-thread, 5% for DataLoader throughput) to distinguish between statistically isolated winners and tied top-tier performers.

Key Contributions

Protocol Comparison: A direct comparison of single-thread decoder rankings against DataLoader throughput rankings on the same workload, libraries, and pinned environments.
Reproducible Artifact: Release of an installable CLI and Google Cloud runner scripts to replicate the benchmark locally or in the cloud, including raw JSON data and generated figures.
Isolation of Variables: Identification of rank changes and worker-scaling effects that are invisible when evaluation is restricted to a single CPU family or a single worker count.
Operational Decision Protocol: A framework converting benchmark results into operational recommendations that account for decoder choice, worker count, microarchitecture, and robustness policies.

Key Results

Rank Inversion: Single-thread leaders frequently fail to be DataLoader leaders. On three of the five platforms, the single-thread leader is not the peak DataLoader leader.
- Neoverse V2: imageio ranks 9th in single-thread but ties for the top tier in DataLoader (alongside torchvision).
- AMD Zen 4: torchvision rises from 7th in single-thread to the top DataLoader tier.
- Neoverse N1: imagecodecs leads single-thread but ranks 4th in peak DataLoader throughput.
Worker-Count Scaling: The optimal worker count is not universal. On AMD Zen 4, most decoders peak at 4 workers, whereas on Zen 5 and the ARM platforms, performance generally continues to improve up to 8 workers.
Robustness Failures: Strict decoders (jpeg4py, kornia-rs, turbojpeg) consistently skip the same rare ImageNet JPEG (index 19876) across all platforms, reducing their effective throughput. TensorFlow exhibits a significant performance penalty on ARM architectures (dropping to ~59% of the local winner) compared to x86.
Top-Tier Recommendations:
- Zero-Skip Tier: Only torchvision, simplejpeg, and OpenCV achieved zero observed skips and remained above 90% of the local platform winner across all tested CPUs.
- Performance Nuance: torchvision offers the highest mean normalized throughput, while simplejpeg provides the highest minimum throughput across platforms. OpenCV serves as a robust general-purpose fallback, consistently achieving 91.1%–97.4% of the local winner.

Significance and Claims
The paper's primary significance is methodological: it demonstrates that the evaluation protocol must match the deployment claim.

Scope of Claim: The authors do not claim a universal "fastest" decoder, nor do they claim that ARM is inherently slow or that AMD scales poorly. Instead, they assert that under a fork-based PyTorch DataLoader harness, decoder rankings are context-dependent.
Operational Impact: For PyTorch DataLoader workloads, relying on single-thread benchmarks can lead to selecting a decoder that is 4.7% to 7.4% slower than the optimal choice for that specific platform.
Conclusion: A valid benchmark for training pipelines must report loader-scale throughput, target microarchitecture, and decode-failure behavior simultaneously. The paper provides a framework to make these boundaries explicit, moving away from single-leaderboard metrics toward tiered, context-aware recommendations.