GreenServ: Energy-Efficient Context-Aware Dynamic Routing for Multi-Model LLM Inference

This paper introduces GreenServ, a dynamic, context-aware routing framework that leverages a multi-armed bandit approach to optimize the trade-off between accuracy and energy efficiency by intelligently directing queries to the most suitable model within a heterogeneous pool of large language models.

The Gist

Imagine you run a massive, high-tech restaurant called GreenServ. This restaurant has a very special menu: instead of just one chef, it has a kitchen full of 16 different chefs, ranging from a lightning-fast but simple line cook to a slow, world-renowned master chef who can cook anything but takes hours and uses a ton of electricity.

In the past, most restaurants (and AI companies) had a lazy rule: "No matter what you order, we always send it to the Master Chef."

• If you ordered a simple glass of water, the Master Chef would still stop everything, put on his apron, and use his expensive, energy-hungry tools to make it.
• The Problem: This wastes a massive amount of energy and money. It's like using a sledgehammer to crack a nut.

GreenServ is a new, smart Maitre d' (Host) that changes the game. Instead of blindly sending every order to the Master Chef, GreenServ looks at your order, figures out exactly what you need, and sends it to the perfect chef for that specific job.

Here is how GreenServ works, broken down into simple steps:

1. The Smart Scan (Context Awareness)

When you walk up to the counter and say, "I want a summary of this news article," GreenServ doesn't just hear the words. It quickly scans your request using three "superpowers":

• Task Type: Is this a math problem? A joke? A translation? (Like asking, "Is this a soup order or a steak?")
• Semantic Cluster: What is the general topic? Is it about space, cooking, or law? (Like asking, "Is this a fancy dinner or a quick lunch?")
• Complexity: Is the sentence long and confusing, or short and simple? (Like asking, "Is the steak rare or well-done?")

2. The Learning Waiter (The "Bandit" Strategy)

GreenServ uses a clever learning trick called a Multi-Armed Bandit. Imagine a row of slot machines (the chefs). You don't know which one pays out the best (best accuracy) for which coin (your specific order).

• The Old Way: You just pick one machine and stick with it forever.
• GreenServ's Way: It tries different machines. If you order a simple math problem, it might try the "Line Cook" first. If the answer is good, it remembers: "Hey, the Line Cook is great for math and uses very little electricity!" If the answer is bad, it learns: "Okay, next time, send math to the Master Chef."

It learns on the fly while you are ordering. It doesn't need to spend weeks in a classroom (offline training) before it starts working. It just learns as it goes.

3. The Energy vs. Taste Balance

GreenServ has a special dial called $\lambda$ (Lambda).

• If you turn the dial to "Taste Only," GreenServ will always send your order to the Master Chef, no matter how much energy it burns.
• If you turn it to "Energy Only," it will send everything to the Line Cook to save power, even if the food might be a bit plain.
• The Sweet Spot: GreenServ finds the perfect middle ground. It sends simple tasks to the cheap, fast chefs and hard tasks to the expensive masters.

The Results: A Win for Everyone

The researchers tested this system with 16 different AI models (chefs) and thousands of questions (orders). Here is what happened:

• Compared to Random Guessing: GreenServ got 22% more answers right while using 31% less electricity.
• Compared to "Always Use the Best Chef": It saved a massive amount of energy without sacrificing much quality.
• Speed: The time GreenServ took to decide which chef to use was so fast (less than 8 milliseconds) that it was like a blink of an eye compared to the time it took the chefs to actually cook.

Why This Matters

Think of the current AI boom like a gold rush. Everyone is building bigger and bigger models (bigger chefs), but they are burning through electricity like there's no tomorrow. GreenServ is the smart traffic cop for this rush.

It proves that you don't need the biggest, most expensive tool for every job. By matching the right tool to the right job, we can make AI cheaper, faster, and much greener for the planet. It's about working smarter, not harder.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) are computationally expensive, with inference energy consumption posing a significant barrier to sustainable deployment. Current industry practices often rely on static, "one-model-fits-all" strategies, where all queries are routed to a single large model regardless of task complexity. This approach is inefficient because:

• Resource Waste: Simple tasks (e.g., basic translation) do not require massive models, yet they consume excessive energy.
• Dynamic Landscape: The ecosystem of open-source models is vast and rapidly evolving; static calibration cannot adapt to new models or shifting query distributions.
• Limitations of Existing Routing: Existing dynamic routing methods often rely on proxy metrics (e.g., API costs, token counts) rather than actual energy consumption, lack continual learning capabilities (requiring expensive offline retraining for new models), or suffer from high latency due to cascading approaches (trying small models first, then escalating).

Goal: Develop a routing framework that dynamically assigns queries to the most suitable model from a heterogeneous pool to optimize the trade-off between inference accuracy and energy efficiency, while supporting online integration of new models without extensive offline calibration.

2. Methodology: The GreenServ Framework

GreenServ is a context-aware, dynamic routing system that uses a Contextual Multi-Armed Bandit (MAB) approach. It operates in three main stages:

A. System Architecture

1. Query Context Generator: Extracts lightweight features from incoming queries to create a context vector ($x_t$).
   • Task Type: Classified via a lightweight Logistic Regression model on semantic embeddings (e.g., QA, Summarization).
   • Semantic Cluster: Determined via online K-Means clustering of full query embeddings to group similar semantic domains.
   • Text Complexity: Calculated using the Flesch Reading Ease formula and binned into categories.
2. Router Agent Trainer: Uses the LinUCB (Linear Upper Confidence Bound) algorithm.
   • Treats each available LLM as an "arm."
   • Learns a policy $\pi: x_t \rightarrow m_t$ that maps the context vector to the optimal model.
   • Reward Function: Combines accuracy and energy consumption into a scalarized reward: $r_t = (1-\lambda) \cdot \text{Accuracy} - \lambda \cdot \text{Energy}$. The parameter $\lambda$ allows users to tune the trade-off between accuracy and energy.
3. Online Deployment:
   • Routes queries in real-time.
   • Monitors actual GPU power draw (in Watt-hours) and latency.
   • Updates the MAB parameters online after every inference, allowing the system to learn from partial feedback without needing ground truth for unselected models.

B. Key Technical Innovations

• Direct Energy Measurement: Unlike prior work using proxy costs, GreenServ measures actual GPU power consumption.
• Zero-Calibration Integration: New models can be added to the pool and immediately integrated into the routing strategy via online learning, eliminating the need for offline calibration.
• Contextual Bandits: By leveraging query context (task, semantics, complexity), the system avoids the "cold start" and suboptimal routing of static or non-contextual baselines.

3. Key Contributions

1. Adaptive Routing Framework: A novel system balancing accuracy and energy using MABs, capable of integrating new models dynamically.
2. Multi-Feature Context Representation: A structured context vector combining task type, semantic clustering, and text complexity, proven to significantly improve routing decisions.
3. Comprehensive Evaluation: Extensive benchmarking against static baselines (random, largest, smallest, most accurate) and other MAB strategies ($\epsilon$-Greedy, Thompson Sampling) across 5 tasks and 16 diverse LLMs.
4. Empirical Validation: Demonstrated that dynamic routing can surpass the static Pareto frontier (the theoretical limit of single-model performance).

4. Experimental Results

The system was evaluated on a pool of 16 open-access LLMs (ranging from 0.5B to 34B parameters) across five benchmark tasks (MMLU, HellaSwag, Winogrande, GSM8K, CNN/DailyMail).

• Performance vs. Baselines:
  • Compared to Random Routing, GreenServ achieved a 22% increase in accuracy while reducing cumulative energy consumption by 31%.
  • Compared to the Largest Model (Yi-34B), it reduced energy by 64% with comparable accuracy.
  • Compared to the Smallest Model (Qwen-0.5B), it improved accuracy by 90-100% with only a 400% energy increase (still far more efficient than the largest model).
• Trade-off Control: By adjusting $\lambda$, GreenServ could operate anywhere along the accuracy-energy spectrum, consistently reaching or exceeding the static Pareto front.
• Contextual Feature Impact: Ablation studies revealed that Task Type was the most informative feature, reducing median cumulative regret to ~400. While adding all features was beneficial, the task type alone drove the majority of the performance gain.
• Adaptability: When a new high-performing model (Gemma-3-12B) was introduced mid-experiment, GreenServ successfully identified and integrated it into its routing strategy within ~100 queries, shifting selection frequency away from older models.
• Overhead: The routing and feature extraction overhead was negligible, averaging 7.77 ms per query. This is insignificant compared to inference latencies (36ms–200ms).
• External Validation: Evaluated on RouterBench (36k queries), GreenServ achieved a peak accuracy of 75.7% and an average accuracy of 71.7%, outperforming other contextual routing algorithms.

5. Significance and Conclusion

GreenServ addresses a critical bottleneck in the sustainable deployment of LLMs. By moving away from static, resource-heavy inference strategies, it demonstrates that dynamic, context-aware routing can significantly reduce the carbon footprint of AI services without sacrificing quality.

• Sustainability: It provides a practical pathway to reduce the massive energy demands of LLM inference, which currently exceeds training energy in cumulative volume.
• Scalability: The ability to integrate new models online without retraining makes the framework future-proof for the rapidly expanding open-source model ecosystem.
• Practicality: The low computational overhead and reliance on direct energy measurements make it suitable for real-world production environments.

The paper concludes that GreenServ effectively solves the multi-objective optimization problem of LLM inference, offering a blueprint for energy-efficient, high-performance AI services.