💻 computer science

GPU Acceleration in Deep Learning Training: A Comparative Study on Model Scale and Performance

This study demonstrates the significant performance advantages of NVIDIA RTX 3050 GPUs over Intel CPUs in deep learning training, revealing a 40.6-fold speedup for large-scale ResNet50 models on the CIFAR-10 dataset.

Original authors: Shiyou Li, Hai Qin, Ning Yuling

Published 2026-08-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shiyou Li, Hai Qin, Ning Yuling

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of modern artificial intelligence, computers are taught to recognize patterns, from identifying a cat in a photograph to translating a sentence from one language to another. This learning process, known as deep learning, relies on complex mathematical structures that mimic the way human brains connect neurons. To train these systems, computers must perform billions of calculations, a task that requires immense processing power. For decades, the standard brain of a computer, called the central processing unit or CPU, has been the go-to tool for general tasks. However, as these digital models have grown larger and more intricate, the CPU has often struggled to keep up, slowing down research and development. To solve this, scientists turned to a different kind of processor originally designed for rendering video games: the graphics processing unit, or GPU. While a CPU is like a few highly skilled experts who can solve difficult problems one by one, a GPU is like a massive army of thousands of simpler workers who can all perform the same simple task at the exact same time. This ability to work in parallel makes the GPU uniquely suited for the repetitive math required to train deep learning models, but the exact point where this advantage becomes critical has remained a subject of detailed investigation.

Researchers at Hunan First Normal University set out to measure exactly how much faster a GPU becomes compared to a CPU as the size of the learning model increases. They did not rely on theory or computer simulations alone; instead, they ran a series of controlled experiments using real hardware and real data. The team used a standard set of 60,000 small color images, a collection known as CIFAR-10, which contains pictures of airplanes, cars, birds, and other common objects. They trained three different versions of a digital brain on this data: a very small, simple network with about half a million internal settings, a medium-sized network with roughly 11.7 million settings, and a large, complex network with about 25.6 million settings. For each of these three models, they performed the exact same training process twice: once on a powerful graphics card and once on a high-end central processor, keeping every other variable identical to ensure a fair comparison.

The results revealed a clear and dramatic shift in performance as the models grew larger. When the researchers trained the smallest, simplest model, the graphics card was only twice as fast as the central processor. In this scenario, the time spent moving data between the computer's memory and the processor took up a significant portion of the total time, limiting the speed advantage of the parallel workers. However, as the model size increased to the medium level, the graphics card became more than seven times faster. The larger volume of calculations meant the processor could spend more time working and less time waiting for data. The difference became most striking with the largest model. Here, the graphics card completed ten rounds of training in just 134 seconds, while the central processor took over 5,400 seconds to finish the same work. This meant the graphics card was more than forty times faster, shrinking a training session that would have taken over an hour and a half down to just over two minutes.

Crucially, the study confirmed that this massive speedup did not come at the cost of quality. The models trained on the graphics card and the central processor achieved nearly identical accuracy on the test images, proving that the faster method did not produce a worse result. The researchers found that the relationship between model size and speed is not a straight line; the benefit of using the graphics card grows super-linearly. As the number of internal settings in the model increases, the time saved becomes exponentially greater. This suggests that for small, simple tasks, a standard computer processor might be sufficient, but for the large-scale models that drive today's most advanced artificial intelligence, the graphics card is not just an option but a necessity. The work provides a concrete, quantitative map for researchers and engineers, showing exactly where the tipping point lies and why the shift to parallel computing becomes essential as the digital brains we build become more complex.

Technical Summary: GPU Acceleration in Deep Learning Training

Problem Statement
As deep learning models evolve from early architectures like AlexNet to modern large-scale models, the number of parameters has grown exponentially, creating a bottleneck in computing resources that restricts research efficiency and application implementation. While Graphics Processing Units (GPUs) are widely recognized as the standard hardware for deep learning due to their parallel computing capabilities, there is a lack of systematic, quantitative studies detailing how the acceleration ratio between GPU and Central Processing Units (CPUs) changes across different model scales. Specifically, the magnitude of performance gains for beginners and interdisciplinary researchers often lacks intuitive, data-driven benchmarks across small, medium, and large-scale models.

Methodology
This study employs a controlled variable experimental design to quantitatively evaluate the performance differences between an NVIDIA GeForce RTX 3050 GPU and an Intel Core i9-14900k CPU. The experiments utilize the CIFAR-10 dataset (60,000 32×32 color images) and the PyTorch 2.11.0 framework with CUDA 12.1.

Three distinct neural network architectures representing different scales were trained under identical hyperparameters (10 epochs, batch size 128, Adam optimizer, StepLR scheduler):

Simple CNN: A custom small-scale model with ~0.5 million parameters (3 convolutional layers, 2 fully connected layers).
ResNet18: A shallow residual network with ~11.7 million parameters.
ResNet50: A medium-scale industrial model with ~25.6 million parameters. Note: The paper contains an internal inconsistency regarding this model's architecture; the text describes it as containing "50 convolution layers," while Table 2 lists it as having "101" convolution layers.

The primary metrics recorded were total training time, average time per epoch, and test accuracy. The acceleration ratio was calculated as the ratio of CPU training time to GPU training time.

Key Results
The study demonstrates a super-linear growth trend in GPU acceleration ratios as model complexity increases:

Simple CNN (~0.5M params): The GPU achieved a 2.0x speedup over the CPU. The total training time was reduced from 114.94 seconds (CPU) to 57.99 seconds (GPU). The authors attribute the limited gain to the low computational complexity of the model, where data transmission overhead between CPU and GPU constitutes a significant portion (30–40%) of the total time.
ResNet18 (~11.7M params): The acceleration ratio increased to 7.3x. CPU training time rose to 463.63 seconds, while GPU time increased only marginally to 63.77 seconds.
ResNet50 (~25.6M params): The GPU achieved a 40.6x speedup. Training time was drastically reduced from 90.7 minutes (5,443.67 seconds) on the CPU to 2.2 minutes (134.08 seconds) on the GPU.

Crucially, the test accuracy remained consistent between GPU and CPU training for each model (e.g., ResNet50 achieved ~84.36% on GPU vs. ~84.12% on CPU), confirming that the speedup did not compromise model convergence or solution quality.

Key Contributions

Quantitative Scaling Law: The paper establishes a clear empirical relationship showing that GPU acceleration is positively correlated with model size. As parameters increased by approximately 50 times (from Simple CNN to ResNet50), the acceleration ratio increased by approximately 20 times (from 2.0x to 40.6x).
Mechanism Analysis: The study validates that the "computation-to-communication" ratio is the governing factor. For small models, data transfer overhead limits GPU utility; for large models, the massive parallelizable computation allows the GPU's thousands of cores to operate near peak efficiency, rendering data transfer overhead negligible.
Theoretical Alignment: The experimental results align with Amdahl's Law, demonstrating that as the proportion of parallelizable code ( $P$ ) approaches 1 in computationally intensive models, the theoretical speedup approaches the number of processing units, which is empirically observed here as a 40.6x gain.

Significance and Limitations
The authors claim this study provides a reference for engineering practice and research, offering intuitive quantitative evidence that GPU acceleration becomes indispensable for large-scale deep learning models, whereas CPUs may remain an economical choice for small-scale tasks.

The paper explicitly acknowledges its limitations:

It tests only a single GPU model (RTX 3050) and a single dataset (CIFAR-10).
It does not cover advanced scenarios such as multi-GPU parallelism or distributed training.
Future work suggested by the authors includes comparing different GPU tiers (e.g., A100, RTX 4090), testing larger architectures (e.g., ViT, ResNet152), and investigating the impact of optimization techniques like mixed-precision training and gradient accumulation.

Technical Summary: GPU Acceleration in Deep Learning Training

More like this