💻 computer science

LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference

This paper proposes a human-in-the-loop framework that leverages LLMs with specialized feedback prompting to rapidly and efficiently optimize runtime parameters for energy-efficient model inference, outperforming traditional search methods like Sobol sampling in both convergence speed and final energy consumption.

Original authors: Katelyn Crumpacker, Dimitrios Nikolopoulos

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Katelyn Crumpacker, Dimitrios Nikolopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very powerful, high-performance car (the AI model) that you need to drive to get a job done. The problem is, this car is a gas guzzler. It drinks a massive amount of fuel (energy) every time you turn the key, and if you leave it idling or drive it inefficiently, you're wasting money and hurting the environment.

Usually, to make this car more efficient, you'd have to hire a team of mechanics to tweak the engine, change the tires, and adjust the fuel mixture. They would try one setting, drive a mile, check the fuel gauge, then try another setting. This "trial and error" process can take days, and they might not even find the perfect setting.

This paper introduces a new mechanic: an AI "Co-Pilot" (a Large Language Model or LLM) that learns how to tune the car's engine in real-time, using far fewer test drives.

Here is how the researchers did it, broken down into simple concepts:

1. The Problem: The "Black Box" of Tuning

Running AI models requires many "knobs" to be turned (like how much memory to use, how many requests to handle at once, or how fast the processor should run). Turning these knobs changes how much energy the computer uses.

The Old Way: You guess a setting, run the test, see the result, and guess again. It's like trying to find the perfect temperature for a shower by turning the dial randomly until you don't burn yourself.
The Issue: This takes too long and uses too much energy just to figure out the settings.

2. The Solution: The "Smart Co-Pilot"

The researchers created a system where a second AI (the Co-Pilot) watches the first AI (the Car) work.

The Loop: The Co-Pilot suggests a new setting (like "turn the memory knob up a bit"). The system runs the test. The Co-Pilot looks at the result ("Wow, that used less fuel!") and uses that information to make a smarter suggestion for the next turn.
The Human Touch: A human acts as the "traffic controller." They set the rules (e.g., "Don't break the engine") and make sure the Co-Pilot stays on track, but they don't manually turn the knobs themselves.

3. The Secret Sauce: Better "Instructions"

The researchers found that how you ask the Co-Pilot matters.

The "Bare Bones" Prompt: They tried giving the Co-Pilot very simple instructions, like "Here is the data, guess the next setting." This worked okay, but the Co-Pilot was a bit scattered.
The "Enhanced" Prompt: They gave the Co-Pilot a structured guide. They said, "Here is the background, here is the goal, here is what happened last time, and here is exactly how to think about the next step."
The Result: The "Enhanced" Co-Pilot was like a seasoned race car driver who knows the track. It found the most fuel-efficient settings much faster (in about 3.4 tries) compared to the "Bare Bones" version (which took about 5.2 tries) and much faster than random guessing methods.

4. The Test Drive

They tested this on two different types of "cars":

Text Processing: Using a system called vLLM to read and write text.
Image Processing: Using a system called PyTorch to look at pictures and answer questions about them.

The Results:

Faster Tuning: The AI Co-Pilot found the best settings in fewer attempts than traditional math-based methods (which had to try dozens of random combinations).
Less Fuel: The final settings used significantly less energy per word or image processed.
Bonus Speed: Interestingly, by focusing on saving energy, the car actually got faster. It processed more words per second while using less power. It's like finding a driving route that saves gas and gets you to the destination sooner.

5. The Cost of the Co-Pilot

You might ask, "Does the Co-Pilot itself use a lot of energy?"
The researchers calculated that the energy used by the Co-Pilot to figure out the settings is very small. It's like the cost of a single cup of coffee. Once you've driven the car about six times (or processed six batches of work), the fuel saved by using the Co-Pilot's settings completely pays back the cost of the coffee. After that, you are purely saving energy.

Summary

In short, this paper shows that we can use one AI to teach another AI how to run more efficiently. Instead of blindly guessing or running expensive, slow tests, a smart, guided AI can quickly learn the "sweet spot" for energy usage, saving power and time for everyone.

1. Problem Statement

Large Language Models (LLMs) are increasingly integrated into real-world workflows, but their inference processes consume significant energy. A critical challenge is selecting optimal runtime parameters (e.g., batch sizes, memory utilization, precision) to minimize energy consumption (specifically Joules per token) without sacrificing performance.

Current Limitations: Traditional optimization methods (e.g., grid search, random search, Bayesian optimization) often require exhaustive exploration of the parameter space, taking days to converge. They lack the ability to dynamically adapt to specific hardware constraints or system feedback efficiently.
Goal: The authors aim to determine if an LLM can act as an adaptive decision agent to iteratively propose runtime configurations that minimize energy consumption faster than traditional black-box optimization techniques.

2. Methodology

The authors propose a human-in-the-loop framework where an external LLM (specifically Claude) guides the optimization of inference parameters for other LLMs.

A. System Architecture

Target Systems: The framework was tested on two distinct inference backends:
1. vLLM: Serving the Llama-3.2-3B model for text-based workloads.
2. PyTorch: Serving the Qwen3-VL-4B-Instruct multimodal model for image-text workloads.
Hardware: Experiments were conducted on a server with a single NVIDIA Tesla V100 (16GB VRAM).
Measurement: Energy consumption (Joules), wall time, and throughput were measured using NVIDIA's Management Library (NVML) to sample GPU power usage.

B. The Optimization Loop

Search Space: The parameters fall into four categories:
- Memory utilization (e.g., block size, memory fraction).
- Batch and execution control (e.g., max sequences, batch size).
- Precision and computation (e.g., fp16 vs. fp32).
- Hardware constraints (e.g., GPU power limits).
Feedback Mechanism: After each inference run, energy metrics are collected. These metrics are fed back into the LLM via a prompt to guide the next configuration proposal.
Prompting Strategies: The authors compared two prompting approaches:
- Baseline (Unstructured): Provides minimal information (previous parameters and metrics) to the LLM.
- Enhanced (Structured): Uses specific prompting techniques including:
  - Repetition: Reiterating the goal.
  - Background Information: Providing hardware specs and parameter descriptions.
  - Search-Process Constraints: Guiding the LLM to focus on a single task (minimizing energy) rather than multi-task optimization.
  - Task Framing: Clearly defining the input/output structure.

C. Baselines

Sobol Sampling: A deterministic quasi-random sampling method used as a baseline to ensure uniform coverage of the search space.
Default Configuration: The standard settings of the inference engines.

3. Key Contributions

LLM-Guided Optimization Framework: A novel approach using an external LLM as an adaptive agent to iteratively tune runtime parameters based on system feedback, replacing exhaustive search.
Energy-Aware Prompt Engineering: The design of a specialized "Enhanced Prompt Template" that significantly improves the LLM's ability to converge on energy-efficient configurations compared to unstructured prompts.
Cross-Backend Validation: Empirical evaluation across two different inference systems (vLLM and PyTorch) and two modalities (text and multimodal), demonstrating the generalizability of the approach.
Reduced Search Overhead: Demonstration that LLM-guided tuning converges to near-optimal energy configurations with significantly fewer evaluations than traditional black-box optimization.

4. Key Results

A. Convergence Speed

vLLM (Text): The Enhanced Prompt converged to the target threshold (1.80 J/token) in an average of 3.4 iterations, compared to 5.2 iterations for the Baseline Prompt. This difference was statistically significant ( $p=0.0086$ ).
PyTorch (Multimodal): The Enhanced Prompt reached the energy per token threshold in 4.2 iterations, while the Baseline required 6 iterations (and failed to reach the threshold in all 5 runs within the limit).

B. Final Energy Efficiency

vLLM: The Enhanced Prompt consistently achieved lower minimum energy per token than the Baseline across all 5 experimental runs (e.g., 1.68 J/token vs. 1.75 J/token in the best case).
PyTorch: The Enhanced Prompt achieved significantly lower energy per image embedding (27.5 J/image) compared to Sobol sampling (82.5 J/image).

C. Comparison with Sobol Sampling

vLLM: Sobol sampling required evaluating 30 configurations (taking ~24 hours) to find a result (1.72 J/token) comparable to what the LLM achieved in an average of 3.4 prompts.
PyTorch: Sobol sampling suffered from a high failure rate (55/64 runs resulted in Out-of-Memory errors due to fp32 precision settings), whereas the LLM successfully navigated these constraints.

D. Performance vs. Energy Trade-off

Contrary to the typical trade-off where energy efficiency reduces performance, the optimized configurations increased throughput by 76% (from 33.5 to 59 tokens/s) while reducing energy. This indicates a shift toward a more Pareto-efficient operating region.

E. Optimization Overhead

The energy cost of the optimization process itself (LLM prompts) was estimated at ~43,200 J. The system reaches a "break-even point" (where savings exceed overhead) after approximately 5.65 workloads (1,000 prompts each), making the overhead negligible for sustained usage.

5. Significance

This work highlights the potential of AI-assisted system optimization. It demonstrates that LLMs, when guided by structured prompts and human oversight, can act as intelligent agents to solve complex, high-dimensional optimization problems in HPC environments.

Scalability: The method reduces the computational cost of finding optimal configurations, which is crucial as parameter spaces grow in multi-node, heterogeneous systems.
Adaptability: Unlike static optimization methods, this approach can dynamically adapt to specific hardware constraints and error states (e.g., OOM errors) in real-time.
Future Impact: The results suggest a pathway toward self-optimizing inference systems that continuously adapt to workload and hardware changes to maximize energy efficiency without human intervention.