Temperature-Aware Scheduling of LLM Inference in Large-Scale Geo-Distributed Edge Data Centers with Distributed Optimization

This paper proposes a temperature-aware, distributed optimization approach using the alternating direction method of multipliers to co-optimize energy, carbon, water, and latency costs for LLM inference across geo-distributed edge data centers in Australia, leveraging ambient temperature diversity to enhance sustainability and efficiency.

The Gist

Imagine you have a fleet of giant, super-smart robots (Large Language Models, or LLMs) that are constantly chatting with people, writing code, and answering questions. These robots live in "data centers," which are basically massive warehouses filled with computers.

Here's the problem: These robots are thirsty, hot, and expensive to run.

1. They get hot: Like a car engine on a summer day, the computers get incredibly hot. To keep them from melting, we need giant air conditioners.
2. They are thirsty: The air conditioners use a lot of water to cool down, and the power plants that generate the electricity also use water.
3. They leave a carbon footprint: The electricity they use often comes from burning coal or gas, which pollutes the air.

Usually, data center managers treat all their warehouses the same. They think, "We need to cool these computers, so we'll just turn on the AC everywhere." But this paper says: "Wait a minute! Not all warehouses are in the same weather!"

The Big Idea: "The Weather-Smart Robot Manager"

The authors of this paper came up with a clever way to schedule these robot jobs. Think of it like a smart delivery service for a pizza chain, but instead of pizza, they are delivering "answers" from AI.

The Analogy: The Pizza Chain

Imagine you run a pizza chain with 20 stores across Australia.

• Store A is in a freezing cold town in the south.
• Store B is in a scorching hot town in the north.
• Store C is in a mild, rainy city.

If you get an order for a pizza, where do you send it?

• The Old Way: You just send it to the nearest store, regardless of the weather. If the nearest store is in the heat, their ovens (and air conditioners) have to work overtime, using tons of gas and water.
• The New Way (This Paper): You look at the weather map. You send the order to the cold town. Why? Because the air outside is already cold! The store there doesn't need to run its air conditioner as hard. It saves money, saves water, and creates less pollution.

How It Works in the Real World

The researchers built a "brain" (a mathematical algorithm) that does this for AI robots across Australia. Here is what it optimizes:

1. Temperature Awareness: It checks the temperature of every data center. If a center is in a cool place, it sends more work there because cooling is cheap and easy. If a center is in a hot place, it sends less work there to avoid overheating and wasting energy.
2. The "Time-to-First-Token" (TTFT): This is just a fancy way of saying, "How long until the robot starts talking?" The system makes sure the robot answers quickly, so you don't have to wait.
3. The "Four-Way Balance": The system tries to find the perfect sweet spot between:
   • Money: Keeping electricity bills low.
   • Pollution: Keeping carbon emissions low.
   • Water: Keeping water usage low.
   • Speed: Keeping the response time fast.

The Results: Why It Matters

The researchers tested their "Weather-Smart Manager" against two other popular methods (one that just tries to be fast, and one that tries to be balanced but ignores the weather).

• The Winner: Their new method was the champion.
• The Score: It managed to keep the response speed just as fast as the others, but it drastically reduced the cost, the pollution, and the water usage.

The Takeaway

Think of this paper as teaching data centers to stop fighting the weather and start using it.

Instead of forcing a hot computer to cool down in a hot city (which is hard and expensive), the system moves the work to a cool city where nature does the heavy lifting. It's like choosing to walk in the shade on a hot day instead of walking in the sun; you get to your destination just as fast, but you arrive much cooler and with more energy left to spare.

By doing this, we can keep our AI smart and fast without burning the planet or draining our water supplies.

Technical Summary

Here is a detailed technical summary of the paper "Temperature-Aware Scheduling of LLM Inference in Large-Scale Geo-Distributed Edge Data Centers with Distributed Optimization."

1. Problem Statement

The rapid growth of Large Language Models (LLMs) has shifted the primary environmental burden from model training to the inference phase. While training is resource-intensive, inference consumes approximately 25 times more computational resources annually and generates a carbon footprint 1,400 times greater than training in large-scale data centers. Furthermore, LLM inference significantly exacerbates water consumption due to cooling requirements.

Current workload scheduling strategies often fail to address these sustainability issues effectively because:

• They treat cooling system efficiency as a location-independent constant, ignoring the significant impact of ambient temperature on cooling energy consumption.
• They do not simultaneously optimize for multiple conflicting objectives: Energy Cost, Carbon Emissions, Water Consumption, and Time-to-First Token (TTFT).
• Existing methods (e.g., mixed-integer linear programming) often struggle with the scalability required for geo-distributed edge environments or fail to account for the low-latency requirements of inference workloads.

2. Methodology

The authors propose a Temperature-Aware Distributed Optimization framework tailored for geo-distributed edge data centers (specifically modeled across Australia).

A. System Modeling

The paper establishes a comprehensive mathematical model for edge data centers that accounts for:

• IT Energy Consumption: Calculated based on node states (ON, IDLE, OFF) relative to Thermal Design Power (TDP).
• Cooling Energy Consumption: Unlike traditional models, this approach uses a temperature-dependent Coefficient of Performance (CoP). It explicitly models the energy required for Computer Room Air Conditioning (CRAC) and chillers, noting that cooling efficiency varies drastically with ambient temperature (e.g., PUE shifts from 1.30 at 35°C to 1.05 at -3.9°C).
• Water Consumption: Modeled as the sum of:
  • Evaporative loss (cooling).
  • Blowdown (wastewater handling).
  • Grid water intensity (water used for power generation).
• Carbon Emissions: Calculated based on grid carbon intensity for electricity, plus the carbon footprint associated with water generation and wastewater processing.
• LLM Inference Metrics:
  • Memory Footprint: Includes model parameters and Key-Value (KV) cache growth per token.
  • Time-to-First Token (TTFT): Includes model loading overhead and network bandwidth constraints.

B. Optimization Algorithm

The core of the solution is a Distributed Optimization algorithm based on the Alternating Direction Method of Multipliers (ADMM).

• Objective: Co-optimize energy costs, carbon emissions, water consumption, and TTFT.
• Approach: The problem is decomposed to allow distributed computation across geo-distributed sites, making it scalable for large networks of edge data centers.
• Temperature Awareness: The algorithm dynamically routes inference requests to data centers where the ambient temperature allows for higher cooling efficiency, thereby reducing the total energy and water load.

3. Key Contributions

1. Temperature-Aware Framework: A novel scheduling approach that leverages geographical temperature diversity to minimize cooling overhead in LLM inference.
2. Multi-Objective Formulation: A unified problem formulation that simultaneously optimizes four critical metrics: Energy Cost, Carbon Emissions, Water Consumption, and TTFT.
3. Comprehensive Environmental Modeling: A detailed model that integrates carbon and water intensity of both the power grid and the cooling infrastructure, varying by location and time.
4. Distributed Scalability: The use of ADMM enables the solution to scale effectively across 20+ geo-distributed edge data centers without requiring a centralized, computationally prohibitive solver.

4. Results

The proposed method was evaluated against two state-of-the-art baselines:

• Helix: A mixed-integer linear programming approach.
• Splitwise: A queue-based heuristic.

Experimental Setup:

• Scope: 20 edge data centers across Australia.
• Capacity: 200 compute nodes per data center.
• Workload: Real-world LLM token request traces (ChatGPT and ChatGPT-4).

Key Findings:

• Superior Performance: The proposed "Opt-Balance" solution outperformed both Helix and Splitwise across all metrics.
• Efficiency Gains: Compared to Splitwise, the proposed method achieved:
  • Lower carbon emissions.
  • Reduced energy costs.
  • Decreased water consumption.
  • Competitive TTFT: It maintained latency performance comparable to Splitwise, whereas Helix consistently underperformed in latency.
• Single-Objective vs. Multi-Objective: While single-objective optimizations (e.g., Opt-Carbon) excelled in their specific metric, the balanced approach (Opt-Balance) provided the best overall trade-off, proving that a holistic view yields better system-wide sustainability.

5. Significance

This paper addresses a critical gap in sustainable AI infrastructure. By demonstrating that ambient temperature is a critical variable in cooling efficiency, the study proves that geo-distributed edge computing can be a powerful tool for sustainability.

The significance lies in:

• Scalability: Providing a practical, distributed algorithm for managing massive LLM workloads without central bottlenecks.
• Holistic Sustainability: Moving beyond simple "energy efficiency" to a multi-dimensional view that includes water and carbon, which are increasingly regulated and costly.
• Practical Application: The results suggest that cloud providers can significantly reduce operational costs and environmental impact simply by intelligently routing inference requests based on real-time weather and grid data, rather than just server load.

In conclusion, the study validates that temperature-aware scheduling is essential for the future of sustainable LLM deployment, offering a viable path to reduce the massive environmental footprint of generative AI.