More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
This paper introduces a controlled evaluation framework for locally deployed compact open-weight language models on mathematical reasoning, demonstrating that accuracy alone is insufficient for model selection by revealing significant trade-offs where Gemma3:4b offers superior energy efficiency despite Qwen3:4b's higher accuracy on specific datasets.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the last few years, computers have learned to read and write with a fluency that feels almost human. These systems, known as large language models, can solve problems, write stories, and answer questions by predicting the next word in a sentence based on patterns they learned from vast amounts of text. For a long time, the only way to use these powerful tools was to send a question over the internet to a massive server farm owned by a technology company. But a new trend is shifting this dynamic. Researchers and developers are now able to download smaller versions of these models and run them directly on their own computers, much like installing a standard software program. This shift offers significant benefits: your data stays private on your machine, you do not have to pay per question, and you can use the technology even without an internet connection. However, running these programs locally comes with a new set of challenges. When a model runs on your own hardware, it consumes electricity, generates heat, and takes time to think. Just because a model is small enough to fit on a laptop does not mean it is efficient enough to be practical for daily use.
A team of researchers at the Florida Institute of Technology set out to understand the real-world trade-offs of running these local models. They wanted to know if the most accurate model was also the most efficient, or if choosing the best tool for the job required balancing speed, energy, and correctness. To find the answer, they focused on a specific type of task: mathematical reasoning. Math problems are ideal for this kind of test because they have clear, objective answers. Unlike a creative writing prompt where the "right" answer is subjective, a math problem either has the correct solution or it does not. The researchers selected three compact models, each containing fewer than five billion parameters—a measure of the model's size and complexity. These models, developed by Google, Microsoft, and Alibaba, were chosen because they are small enough to be run on standard consumer computers without needing specialized, expensive equipment.
The team designed a rigorous experiment to ensure a fair comparison. They gathered three distinct sets of math problems ranging from eighth-grade arithmetic to university-level calculus and advanced statistics. They then fed these exact same questions to each of the three models, running them all on the same computer with identical settings. Crucially, they did not just look at whether the models got the right answer. They also measured exactly how long each question took to solve, how many words the model generated to get there, and how much electrical energy the computer's graphics card consumed during the process. They built a system to automatically check the final answers, stripping away extra text or formatting to see if the core number or solution matched the expected result. This allowed them to separate genuine reasoning errors from simple formatting mistakes.
The results revealed a surprising disconnect between accuracy and efficiency. No single model was the best at everything. One model, developed by Alibaba, achieved the highest accuracy on the eighth-grade math and advanced statistics tests. Another model, from Google, performed best on the calculus problems. The third model, from Microsoft, struggled significantly across all three categories, getting very few answers correct. However, when the researchers looked at energy efficiency, the rankings changed dramatically. The Google model was the clear winner in terms of power usage. For every unit of electricity it consumed, it produced roughly three times as many correct answers as the Alibaba model. The Alibaba model, while often the most accurate, required significantly more time and energy to generate its responses, producing far more text than necessary to reach the solution. The Microsoft model, despite being the least energy-intensive in some cases, was so inaccurate that its efficiency gains were irrelevant.
The study also highlighted the importance of how these models are configured. The Alibaba model has a "thinking" mode where it pauses to reason through a problem before answering. The researchers disabled this feature for their test to ensure a fair comparison with the other models, which do not have the same capability. They noted that this choice might have lowered the Alibaba model's accuracy compared to how it would perform with thinking enabled, but it was necessary to keep the test conditions consistent. Furthermore, the researchers found that the low performance of the Microsoft model was not due to the computer failing to read the answers. The system successfully extracted answers from nearly every response, meaning the poor results were due to the model providing incorrect solutions rather than a failure in the testing process.
Ultimately, the research demonstrates that for anyone looking to run these models on their own hardware, accuracy is not the only metric that matters. A model that is slightly less accurate but uses a fraction of the energy and time might be the better choice for a student or a small team with limited computing resources. The findings suggest that there is no single "best" model for local use. Instead, the right choice depends entirely on the specific task and the constraints of the user's hardware. If the goal is to solve complex calculus problems with the highest possible precision, one model might be preferred. If the goal is to process hundreds of questions quickly while conserving battery life or electricity, a different model becomes the logical choice. This study provides a clear, documented method for making these decisions, moving beyond simple accuracy scores to a more practical understanding of how these tools perform in the real world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.