Energy efficiency of a GPU-based computing system for High Energy Physics experiments

This paper introduces energy efficiency as a new metric for evaluating GPU hardware and algorithm optimizations in High Energy Physics, presenting a model applied to the LHCb experiment's HLT1 trigger to relate throughput with hardware specifications and guide the development of sustainable computing ecosystems.

The Gist

Imagine you are running a massive, high-speed sorting factory. Every second, millions of tiny packages (data from particle collisions) arrive on a conveyor belt. Your job is to quickly inspect each package, decide if it's interesting, and sort it. This is what the LHCb experiment at CERN does with data from the Large Hadron Collider.

For a long time, this factory used standard "CPU" workers. But as the factory gets busier, these workers are getting tired and the electricity bill is skyrocketing. So, the team decided to hire a new kind of worker: GPUs (Graphics Processing Units). Think of GPUs as a team of thousands of super-fast, specialized robots that can work in parallel.

This paper is about figuring out which robots are the best to hire, not just by how fast they work, but by how much energy they waste.

The Problem: Speed vs. Energy

Usually, when you buy a new machine, you look at its speed. But in a giant factory, speed isn't everything. If a machine is super fast but guzzles electricity like a thirsty elephant, it costs too much to run and generates so much heat you need expensive air conditioning.

The authors wanted a new way to measure these robots: Energy Efficiency. This is simply: How many packages can this robot sort for every single drop of electricity it uses?

The Experiment: Testing the Robots

The team set up a test using 10 different models of NVIDIA GPUs (ranging from older models to the very newest, cutting-edge ones). They ran the exact same sorting task (called HLT1) on all of them.

They measured two things:

1. Throughput: How many packages per second the robot sorted.
2. Power: How much electricity the robot actually drank while doing the job.

The Surprising Discovery: The "Thirsty" vs. "Efficient" Robots

Here is the twist they found: Just because a robot is powerful doesn't mean it will run at its maximum power limit.

Think of a car. If you drive a Ferrari in heavy traffic, you might never reach its top speed, and you won't use all its fuel.

• The "Power-Limited" Robots: Some older or specific workstation robots hit their "fuel cap" (TDP). They are working as hard as they can, but they are capped by their design. They are like a runner sprinting until they are out of breath.
• The "Non-Power-Limited" Robots: Many of the newer, high-end robots were actually not using their full fuel capacity. Even though they were sorting packages at 100% speed, they weren't drinking as much electricity as their specs said they could. They were like a runner who could sprint faster but was only jogging because the task didn't require a full sprint.

The Magic Formula: Predicting the Future

The team didn't just measure these 10 robots; they built a predictive recipe (a mathematical model).

They realized that a robot's speed depends on two main things:

1. How many hands it has (Number of Cores).
2. How fast it can grab items (Memory Bandwidth).

However, they found that doubling the number of hands doesn't double the speed. Because the robots have to talk to each other and wait for instructions, the speed gains get smaller as you add more hands. It's like adding more chefs to a kitchen; eventually, they just get in each other's way.

Using this recipe, they can now look at the "spec sheet" of a brand-new robot that hasn't even been built yet. By plugging in its number of cores and memory speed, they can predict:

• How fast it will sort packages.
• How much electricity it will drink.
• How energy-efficient it will be.

The Winner

When they ranked the robots by energy efficiency (packages per joule of electricity), the results were surprising:

• The fastest robot (RTX PRO 6000) was not the most efficient. It was fast, but it drank a lot of power.
• The most efficient robot (RTX PRO 4000) was actually slower, but it was so frugal with electricity that it sorted more packages per drop of energy than the giants.

Why This Matters

The LHCb experiment is planning to upgrade its factory soon. They can't afford to buy and test every single new robot model that comes out; it would take too long and cost too much.

Thanks to this paper, they can now look at the brochure of a future robot, run it through their "recipe," and know immediately if it's a good hire. They can choose the robot that gives them the best balance of speed and low energy bills, ensuring their massive data factory stays sustainable and affordable for years to come.

In short: They figured out how to predict exactly how much a new computer chip will cost to run and how fast it will work, just by reading its specifications, saving the scientists time, money, and electricity.

Technical Summary

1. Problem Statement

High Energy Physics (HEP) experiments, particularly the Large Hadron Collider (LHC) at CERN, face significant challenges regarding scalability and energy consumption as they transition to the High-Luminosity LHC (HL-LHC) era.

• Scalability Issues: Current CPU-based architectures struggle to handle the massive data volumes (e.g., 40 Tb/s for LHCb) required for real-time trigger and reconstruction.
• Energy Constraints: Traditional CPU approaches lack energy efficiency, leading to prohibitive electricity costs and cooling infrastructure requirements.
• Hardware Selection Difficulty: While GPUs offer a promising alternative, the market offers a vast array of models with varying specifications (core counts, clock speeds, memory bandwidth, Thermal Design Power). Testing every candidate GPU individually is time-consuming and expensive.
• The Gap: There is a lack of predictive models that can estimate throughput and energy efficiency (events processed per Joule) based solely on hardware specification parameters, without requiring full-scale benchmarking.

2. Methodology

The authors developed a predictive framework to model GPU performance and power consumption for the LHCb High-Level Trigger 1 (HLT1) workload.

• Dataset: The study utilized 10 NVIDIA GPUs spanning four architectures (Ampere, Ada Lovelace, Hopper, Blackwell) and two fabrication processes (Samsung 8nm and TSMC 4nm).
• Workload: Benchmarks were run using the hlt1_pp_default reconstruction sequence (Allen v7r10p1), which includes ~300 algorithms for particle tracking, vertexing, and classification.
• Measurements:
  • Throughput: Measured events per second (kHz).
  • Power: Monitored via nvidia-smi and validated with external power distribution units (PDU).
  • Key Metrics: Streaming Multiprocessor (SM) clock, memory clock, and power draw were recorded during steady-state operation.
• Modeling Approach:
  1. Throughput Model: A power-law function relating throughput ($TP$) to compute capacity ($N_{cores} \times f_{clk}$) and memory bandwidth ($BW$).
  2. Power Model: A distinction between "power-limited" and "non-power-limited" GPUs, followed by an exponential decay model for power demand per core.
  3. Energy Efficiency: Calculated as the ratio of Throughput to Power ($E_{eff} = TP / P$).

3. Key Contributions & Findings

A. Throughput Modeling

The authors fitted a power-law model to the measured data:
$$TP = k \times (N_{cores} \times f_{clk})^a \times BW^b$$

• Results: The fitted exponents were $a = 0.59$ (compute capacity) and $b = 0.28$ (memory bandwidth).
• Insight: The HLT1 workload is compute-bound rather than memory-bound. The sublinear scaling ($a, b < 1$) indicates that doubling hardware resources does not double throughput due to synchronization overhead and complex control flow (branching) in pattern recognition algorithms.
• Accuracy: The model predicts throughput with a root-mean-square residual of ~3% across different architectures.

B. Power Consumption & Limiting Criteria

A critical finding is that 100% GPU utilization does not guarantee reaching the Thermal Design Power (TDP).

• Power-Limited vs. Non-Power-Limited:
  • Power-Limited: The workload demands more power than the GPU's TDP allows (e.g., Ampere GPUs and some workstation models). The GPU hits its TDP ceiling, and clocks may throttle.
  • Non-Power-Limited: The workload demands less power than the TDP (e.g., high-end gaming and datacenter GPUs). These GPUs run below their TDP because the algorithm's branching logic leaves some functional units idle.
• Power Demand Curve: For TSMC 4nm GPUs, the power demand per core ($P_{core}$) follows an exponential decay as core count increases, converging to a floor of ~19.6 mW.
  • If a GPU's TDP per core is below this demand curve, it is power-limited.
  • If above, it is non-power-limited, and actual power is predicted by the demand curve.

C. Energy Efficiency

The study defines energy efficiency as events per Joule.

• Trade-off: The GPU with the highest raw throughput is not necessarily the most energy-efficient.
  • Example: The RTX PRO 6000 has the highest throughput (229 kHz) but ranks 4th in efficiency due to high power consumption (481 W).
  • Example: The RTX PRO 4000 has modest throughput (84 kHz) but is the most energy-efficient (581 events/J) because of its low TDP (145 W).
• Architecture Impact: Newer architectures (Blackwell, Hopper) on TSMC 4nm are significantly more efficient than older Ampere (Samsung 8nm) GPUs.

4. Significance and Impact

• Predictive Hardware Selection: The models allow LHC collaborations (LHCb, ATLAS, CMS) to rank candidate GPUs for future upgrades (e.g., Run 4 and Run 5) using only official datasheet parameters, eliminating the need for costly individual benchmarks.
• Algorithm Optimization Guidance: The results suggest that algorithms optimized for power-limited GPUs (like the current RTX A5000) may not be optimal for newer, non-power-limited hardware. Future optimizations should target reducing branching and improving thread utilization to maximize throughput on modern GPUs without hitting power walls.
• Sustainable Computing: By prioritizing energy efficiency (events/J) alongside throughput, CERN can manage the massive power budgets required for the HL-LHC upgrades, ensuring a sustainable computing ecosystem.
• Generalizability: While tested on LHCb HLT1, the methodology is applicable to any GPU-based application requiring performance projection across hardware generations.

Conclusion

This paper establishes a robust framework for evaluating GPU-based HEP computing systems. It demonstrates that energy efficiency is a distinct and critical metric separate from raw throughput. By combining a throughput power-law model with a nuanced power consumption model that accounts for architectural differences and workload characteristics, the authors provide a vital tool for making sustainable, cost-effective hardware decisions for the future of particle physics.