TokenPowerSandbox: Evidence-Gated CPU-First Screening for Energy-Aware LLM Serving
TokenPowerSandbox is an evidence-gated, CPU-first screening framework that combines interpretable projection with limited GPU probing to achieve high energy prediction accuracy for LLM serving while explicitly demonstrating the limitations of energy models in certifying latency performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
When a large language model answers a question, it performs a massive amount of mathematical work on a specialized computer chip. This process consumes electricity, and the amount of power used changes depending on how long the question is, how many people are asking questions at once, and how the software is tuned to handle the load. For companies running these systems, knowing exactly how much energy a specific setting will cost is crucial. They want to save money and reduce their environmental footprint, but they also must promise their users that answers will appear quickly. The challenge is that testing every possible setting on the actual hardware is slow and expensive, while trying to guess the results using a standard computer is often dangerously inaccurate.
Researchers have developed a new method called TokenPowerSandbox to solve this problem. Instead of trying to replace the expensive hardware with a cheap guess, the team created a strict workflow that treats computer predictions as tentative ideas that must be proven by real-world evidence. They built a system that first uses a standard computer to screen out obviously bad options, then takes a very short, quick measurement on the real chip to check if the prediction holds up, and finally runs a full, complete test only on the most promising candidates. This approach ensures that decisions are based on verified facts rather than confident guesses that might be wrong in specific situations.
The researchers tested this system on a single high-end graphics card, the NVIDIA H100, running a specific language model. They began by creating a simple model on a standard computer that could estimate energy use based on the length of the text and the number of users. To make sure this model was fair, they first trained it on a small set of six different workloads. Then, they locked the model's settings so it could not learn anything new, and tested it on a completely separate set of workloads it had never seen before. The results were impressive for energy: the computer's estimates were within about six percent of the actual energy used on the real chip, and it correctly ranked which settings were the most efficient in nearly every case.
However, the study revealed a critical limitation that a simple average would have hidden. While the computer was excellent at predicting energy, it failed to predict how long it would take for the first word of an answer to appear when the system was under light load. In these low-traffic situations, the computer's guesses were wildly off. The researchers built a safety mechanism into their system to handle this. The system is programmed to admit when it is out of its depth. If the traffic is too light, the system refuses to make a prediction about speed and instead demands a real measurement. This "abstention" rule prevented the team from making a costly mistake based on a false sense of confidence.
To find the best settings, the team compared twelve different configurations. The computer-only screen correctly identified which settings used the least energy, but it completely failed to identify which settings were fast enough to meet user requirements. In fact, the computer's ranking of speed was almost the exact opposite of reality. By adding a short, quick measurement step, the team was able to fix this. This brief check on the real hardware corrected the speed rankings and successfully identified the single best configuration that was both energy-efficient and fast enough. Without this quick check, they would have chosen a setting that looked good on paper but would have been too slow for real users.
The final step was to prove that the chosen setting actually worked in a new, independent test. The researchers locked in their selection and ran it against a pre-chosen expert baseline on the same hardware. The new setting used about 1.4 percent less energy per thousand words generated and reduced the time to the first word by more than 21 percent. These numbers might seem small, but in the world of massive data centers, they represent significant savings. The study did not claim to solve the problem for all computers or all models, nor did it claim to work for complex networks of many chips. Instead, it established a reliable foundation for how to make these decisions safely.
The core lesson from this work is that cheap predictions are only useful when they are paired with a clear understanding of their limits. A computer can simulate the energy cost of a task with high accuracy, but it cannot yet simulate the complex delays that happen when a system is busy. The researchers showed that by combining a simple model with a few targeted, real-world checks, and by having the system know when to stop guessing, you can find the best settings without wasting time or money. This method provides a trustworthy way to navigate the trade-off between saving energy and keeping services fast, ensuring that the final decision is backed by evidence rather than hope.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.