Provenance-Aware Evaluation of Cloud and Local Large Language Models for Cybersecurity Workloads
This paper audits a cybersecurity inference project to demonstrate that its published cloud-versus-local energy comparisons are invalid due to provenance flaws, including accounting identities masquerading as measurements and dimensional mismatches, thereby establishing a rigorous, provenance-aware protocol for future evaluations.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern digital world, large language models have become powerful tools for security teams, helping them sort through phishing emails, explain software vulnerabilities, and respond to cyberattacks. These artificial intelligence systems work by predicting the next word in a sentence, but they also consume electricity to do so. As organizations decide whether to run these models on their own servers or rent them from massive cloud providers, a critical question arises: which option is better for the environment? To answer this, researchers must measure the energy used by each request. However, measuring energy is not as simple as reading a meter. It requires knowing exactly what hardware was used, how long it ran, and what specific tasks were performed. Without a clear record of where a number came from and how it was calculated, a comparison between two different systems is like comparing a speedometer reading from a car to a guess about how fast a bicycle might go.
A team of researchers at Auckland University of Technology decided to investigate this problem by looking closely at a specific cybersecurity project that had already been completed. They did not run new experiments or measure electricity with new equipment. Instead, they acted as auditors, examining the existing files, logs, and reports to see if the energy claims made in the original study could actually be trusted. Their goal was to trace every number back to its source, checking if the data was a direct observation of reality or a calculation based on assumptions. They found that the study was trying to compare two very different things: a cloud service where the internal workings are hidden, and a local computer where the researchers could see the processes but had made a fundamental mistake in how they recorded the data.
The researchers discovered that the energy figures for the cloud portion of the study were not measurements of electricity at all. In the cloud spreadsheet containing 360 records, every single energy value was derived by taking the total number of words generated by the model and multiplying it by a fixed number. This meant the energy figure was a mathematical identity, a rule applied to the text, rather than a reading from an electrical meter. The cloud provider did not share any data about how much power their servers actually used for these requests. Consequently, the strong link between the number of words and the energy cost was an artifact of the calculation method, not a physical observation of how much electricity the cloud consumed.
The situation was different but equally problematic for the local experiments, where the model ran on the researchers' own computers. Here, the team used a software tool designed to estimate carbon emissions. This tool correctly calculated how much carbon dioxide equivalent was produced, measured in kilograms. However, the researchers then treated this carbon number as if it were a measure of electrical energy in kilowatt-hours. They multiplied the carbon weight by a thousand and called it energy, and then used that incorrect number to calculate the cost of electricity. This was a confusion of units, similar to weighing a bag of apples and then claiming that weight tells you how much water is in a swimming pool. Because of this error, the local energy numbers could not be compared to the cloud numbers, nor could they be trusted as a true measure of power consumption.
The audit also revealed that the two sides of the study were not testing the same things. The cloud requests involved a different type of model and allowed the answers to be much longer than the local requests, which were cut off at a shorter length. The local study also lacked a clear record of the exact software versions and hardware settings used, making it impossible to know if the results would be the same if the experiment were repeated. Furthermore, the study did not include a way to verify if the security tasks were actually solved correctly, only that they were completed. Without knowing if the answers were good, it is impossible to say if the energy used was efficient or wasted.
When the researchers applied a strict framework for tracking the history of data, known as provenance, they found that the archive could not support a ranking of which deployment was more energy-efficient. The cloud numbers were estimates based on text length, and the local numbers were based on a unit conversion error. The study could not determine if the cloud was better, if the local setup was better, or if they were even comparable. The researchers concluded that future studies must be designed differently. They need to use the same tasks for both systems, measure electricity with proper meters or verified data, and keep a clear, unbroken chain of evidence that shows exactly how every number was derived. Until such a study is done, claims about the environmental benefits of one approach over the other remain unproven.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.