🤖 AI

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

This paper introduces Green-Aware Routing (GAR), a constrained multi-objective optimization framework that minimizes CO2 emissions for LLM inference by dynamically routing requests to heterogeneous models while strictly adhering to accuracy and latency service-level objectives.

Original authors: Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a busy delivery service for a company that uses a fleet of different-sized trucks to deliver packages (answers to questions). Some trucks are tiny and fuel-efficient but slow or bad at carrying heavy loads. Others are massive, powerful super-trucks that can handle anything instantly but guzzle fuel like there's no tomorrow.

In the world of Artificial Intelligence, these "trucks" are Large Language Models (LLMs). Currently, when a user asks a question, the system usually picks a truck based only on speed or cost. It doesn't care much about how much "carbon pollution" that truck creates, even though the "fuel" (electricity) gets dirtier at certain times of day or in certain places.

The paper you shared introduces a new system called GAR (Green-Aware Routing). Think of GAR as a super-smart traffic controller who doesn't just look at speed and price, but also at the environmental impact of every single delivery.

Here is how GAR works, broken down into simple concepts:

1. The Problem: The "Dirtiest" Truck is Often the Default

Right now, if you ask a question, the system might send it to the biggest, most powerful truck just to be safe, or the fastest one to save time.

The Issue: Sometimes, a smaller truck could do the job just as well, but the system doesn't know that until after it's already sent the big, polluting truck.
The Grid Factor: Imagine the electricity grid is like a river. Sometimes the water is clean (renewable energy like wind/solar), and sometimes it's muddy (fossil fuels). The "muddy" times change depending on the time of day and where you are. GAR knows this. It tries to send the "dirty" trucks only when the water is clean, or better yet, sends the "clean" trucks when the water is muddy.

2. The Solution: The "Green-Aware" Traffic Controller

GAR is a decision-maker that looks at three things before picking a truck:

Will it get the job done right? (Accuracy)
Will it be fast enough? (Latency)
How much pollution will it create? (Carbon Emissions)

It uses a special set of rules (mathematical constraints) to ensure it never picks a truck that is too slow or too inaccurate. But among the trucks that are fast and accurate enough, it picks the one that creates the least amount of pollution.

3. How It Makes Decisions (The "Lightweight Predictors")

GAR doesn't want to waste time actually driving the truck to see if it works. That would be too slow. Instead, it uses crystal balls (lightweight estimators) trained on past data to guess:

"If we send this question to the 7B model, will it get the answer right?"
"How long will it take?"
"How much carbon will that specific trip create right now?"

These guesses happen in less than a millisecond. Based on these guesses, GAR picks the "greenest" option that still meets the quality standards.

4. The "Rolling Budget" (The Carbon Wallet)

Imagine you have a monthly budget for pollution. You can't just save up all your pollution for the end of the month; you have to stay within limits every day.

GAR-PD (The Smart Manager): This is the most advanced version of GAR. It keeps a "sliding window" of the last 100 deliveries. If the team has been using too much carbon recently, the system automatically becomes stricter, forcing the selection of cleaner trucks. If the team has been very green, it allows a little more flexibility. It balances the books in real-time.

5. The Results: Doing More with Less Smoke

The authors tested this system using six different types of difficult tasks (like math problems, science quizzes, and reading comprehension) and a fleet of five different AI models (ranging from small to huge).

The Big Winner: The standard "Biggest Truck" approach (Largest LLM) was very accurate but created a huge amount of pollution (2.75 grams of CO2 per request).
The "Smallest Truck" approach: This saved a lot of pollution but got the answers wrong way too often.
GAR-PD: This was the sweet spot. It got the answers right almost as well as the biggest truck (97% as good) but cut the pollution by 74%. It managed to be fast, accurate, and green all at once.

Summary Analogy

Think of it like ordering food delivery.

Old Way: You always order from the most famous, expensive restaurant because you want the best food, even if they are far away and use a gas-guzzling motorcycle.
GAR Way: You ask a smart app: "I want a burger that tastes good and arrives in 20 minutes." The app checks the weather (is the grid clean?), checks the restaurants (which one can do it?), and picks the one that uses an electric bike and is nearby. You get the same tasty burger, but the planet is happier.

In short: GAR is a new way to route AI requests that treats "saving the planet" as a primary goal, not an afterthought, ensuring we get smart answers without burning the world down to get them.

Technical Summary: Green-Aware Routing (GAR) for LLM Inference

1. Problem Statement

The deployment of Large Language Models (LLMs) at scale necessitates routing architectures that balance response quality, tail latency, and computational cost across heterogeneous model pools. While existing routing frameworks optimize for accuracy and latency (often treating monetary cost as secondary), they largely ignore environmental impact. This is a critical gap because:

Grid Carbon Intensity Variability: Carbon intensity varies significantly (up to 10×) across regions and time periods due to renewable energy availability and fossil fuel dependencies.
Model Heterogeneity: Energy consumption varies drastically based on architectural choices, parameter scaling, and token generation patterns.
Misalignment of Objectives: Optimizing for monetary cost or latency alone can systematically direct traffic toward high-carbon periods or carbon-intensive regions, even when multiple models offer equivalent accuracy.

Current systems typically handle carbon emissions only via post-hoc measurement and reporting, lacking real-time, per-request routing decisions that treat carbon minimization as a primary optimization objective alongside formal service-level guarantees.

2. Methodology: Green-Aware Routing (GAR)

The paper introduces GAR, a constrained multi-objective optimization framework designed to minimize per-request CO2 emissions while satisfying explicit accuracy floors and p95-latency Service-Level Objectives (SLOs).

2.1 Problem Formulation

For a query $x$ arriving at time $t$ , the router manages a pool of models $\mathcal{M}$ . Before inference, the system generates lightweight predictions for:

Correctness ( $\hat{p}_m$ ): Probability of a correct answer.
Latency ( $\hat{\ell}_{m,p95}$ ): Estimated p95 response time.
Emissions ( $\hat{c}_m$ ): Estimated CO2, calculated as $energy(m, tokens) \times grid\_intensity(t, region)$ .

The framework defines a feasible set $F(x, t)$ containing all models that satisfy the dataset-specific accuracy floor $\tau_d(x)$ and the p95 latency target $L$ . Within this set, the router selects the model minimizing predicted emissions. To ensure robustness, safety margins are applied to carbon and latency estimates.

2.2 Rolling Carbon Budgets

For streaming workloads, GAR maintains a sliding window of realized emissions over the last $W$ requests. This ensures the average emissions stay within a per-request budget $B$ , preventing long-term carbon debt accumulation while allowing short-term flexibility.

2.3 Algorithms and Policies

The paper proposes several strategies to operationalize this framework:

GAR-PD (Online Primal-Dual): An algorithm inspired by Shalev-Shwartz & Singer (2007) that treats carbon-aware routing as an online Lagrangian problem. It uses a dual variable $\lambda_t$ (shadow price) to balance constraint satisfaction with environmental optimization. The selection objective minimizes a weighted sum of quality loss, latency, and carbon emissions, dynamically updated based on the rolling window sum of realized emissions.
GAR-Fixed: Enforces a hard per-request carbon cap. If the cap renders the feasible set empty, it falls back to the lowest-carbon model available.
GAR- $\epsilon$ : Preserves accuracy within a tolerance $\epsilon$ of the best-performing model in the feasible set while minimizing carbon.
GAR-Target: Sets per-dataset target accuracy floors via binary search to match desired macro accuracy levels, routing to the lowest-carbon model meeting these targets.

2.4 Implementation Details

The framework relies on lightweight pre-route estimators trained on historical data (using logistic regression for accuracy, quantile regression for latency, and linear models for carbon). These predictors run in parallel, completing in under 1 millisecond per query, enabling real-time decision-making without additional inference passes.

3. Key Contributions

Formalization: The paper formalizes carbon-aware routing as a constrained multi-objective optimization problem, providing the first principled framework for per-request CO2 minimization in LLM serving with formal quality guarantees.
Algorithm Development: It develops GAR-PD, a practical online primal-dual routing algorithm for rolling carbon budgets, alongside heuristic variants (GAR-Fixed, GAR- $\epsilon$ , GAR-Target) for diverse deployment scenarios.
Adaptive Optimization: The authors establish adaptive constraint optimization techniques that achieve high feasibility coverage while limiting accuracy degradation, enabling substantial emission reductions without compromising user experience.
Empirical Validation: Comprehensive experiments across standard NLP benchmarks (MMLU, HellaSwag, GSM8K, WinoGrande, SQuAD, ARC) with heterogeneous LLM pools (7B–70B) demonstrate the framework's effectiveness.

4. Experimental Results

The evaluation compares GAR against rule-based baselines (Largest/Smallest LLM), accuracy-optimizing baselines (AccMax), and an Oracle upper bound.

Performance Trade-offs: GAR-PD achieves a macro accuracy of 0.737 with 0.712 g CO2/request.
- Compared to the Largest LLM (0.741 accuracy, 2.750 g/req), GAR-PD reduces emissions by 74% while maintaining comparable accuracy.
- Compared to the Smallest LLM (0.592 accuracy, 0.476 g/req), GAR-PD improves accuracy by 15 percentage points for a modest carbon increase.
- GAR-PD achieves at least 97% of the optimal solution (Oracle-Feasible).
Ablation Studies: Removing components significantly degrades performance:
- Removing the carbon estimator increases emissions by 152% (1.794 g/req).
- Removing feasibility gates causes a 9.5-point accuracy drop and a 23% rise in CO2.
- Removing the latency estimator causes mean latency to jump 147%, violating SLOs.
Pareto Optimality: GAR-PD occupies a strong position on the accuracy-CO2 trade-off frontier, closely tracking the Pareto frontier while remaining practical.

5. Significance and Claims

The paper claims that GAR provides a practical, theoretically grounded approach to sustainable LLM inference. By treating carbon as a primary optimization objective rather than an afterthought, the framework demonstrates that significant reductions in AI infrastructure carbon footprints can be achieved without sacrificing service quality or reliability.

The authors position this work as foundational research. They explicitly acknowledge limitations, noting that their carbon intensity modeling relies on publicly available grid data which may lag real-time conditions by several hours, and that the current framework excludes financial cost considerations. Future work is proposed to integrate real-time carbon APIs, extend the framework to training-time scenarios, and investigate dynamic model ensemble management.