Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware

Imagine you've just built a super-smart robot friend that lives right inside your computer, ready to answer questions, write stories, or help you code without ever sending your secrets to a giant cloud server. This is the world of "local" artificial intelligence, a trend that's exploding because people want privacy and speed. But there's a catch: these brainy robots are hungry. While we've known for a while that training these massive AI brains on huge supercomputers uses a lot of electricity (like powering a small city for a day), we haven't really known how much energy it takes to just talk to them on a regular home computer. It's like knowing a car gets terrible gas mileage on the highway, but having no idea how much fuel it burns just idling in your driveway while you chat with the driver. As more people start running these AI assistants on their own laptops and gaming PCs, understanding that "active" energy bill becomes a real concern for both your wallet and the planet.

This is exactly where a team of researchers from FH Aachen University of Applied Sciences in Germany stepped in. They wanted to see which of the popular, free-to-use AI models are the most "energy-efficient" when running on a standard consumer graphics card—specifically, an NVIDIA RTX 4060 Ti, the kind of card you might find in a high-end gaming PC. Instead of just asking, "Which model gives the best answers?" they asked, "Which model gives the best answers while sipping the least amount of electricity?" They set up a controlled experiment where they fed nine different open-source AI models (ranging from tiny 1-billion-parameter models to larger 7-billion-parameter ones) a fixed set of 15 questions, from simple facts like "What is the capital of France?" to complex tasks like writing code or explaining deep concepts.

The researchers acted like energy detectives, using a tool called nvidia-smi to check the power consumption of the graphics card twice every second while the models worked. Crucially, they didn't just measure the total power; they carefully subtracted the baseline "idle" power of the card to isolate the exact energy cost of the AI's thinking process. Their main goal was to calculate how much total energy (in Joules) it took to generate each single word (or "token") of the answer. Their findings suggest that bigger isn't always better when it comes to energy. In fact, the tiny models, specifically gemma3:1b and llama3.2:1b, turned out to be the champions of efficiency. These little powerhouses managed to generate a single word of text using as little as 0.56 Joules of energy and could spit out over 170 words per second.

In contrast, the larger 7-billion-parameter model (Mistral) was much less efficient, consuming up to 4.4 times more energy per word than the most efficient tiny model. The study also uncovered a quirky surprise: one of the models, qwen3.5:2b, used a surprisingly high amount of energy per prompt. The researchers suggest this wasn't because the model was slow, but because it was "thinking" internally—using a technique called extended reasoning to plan its answer before speaking—which burns extra energy even if the final output is short. This highlights that simply looking at the total energy for a whole question can be misleading; the true efficiency metric is how much energy it takes to produce each token. Ultimately, the paper suggests that for anyone running AI on their own hardware, the most energy-efficient choice isn't necessarily the most powerful or largest model, but rather the one with the right architecture and size for the job, proving that sometimes, the smallest brain is the greenest.