🤖 AI

CloudCons: A Comprehensive End-to-End Benchmark for Cloud Resource Consolidation

This paper introduces CloudCons, a comprehensive end-to-end benchmark using diverse real-world cloud datasets to demonstrate that while foundation models excel at zero-shot forecasting accuracy, their superior performance does not automatically translate to better decision utility in resource consolidation, highlighting the critical role of predictive quantile selection in balancing efficiency and reliability.

Original authors: Xiaobin Zhang, Lefei Shen, Mouxiang Chen, Zhuo Li, Hongkai Li, Han Fu, Jianling Sun, Xiaoxue Ren, Chenghao Liu

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaobin Zhang, Lefei Shen, Mouxiang Chen, Zhuo Li, Hongkai Li, Han Fu, Jianling Sun, Xiaoxue Ren, Chenghao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, busy hotel (the Cloud Data Center). Your goal is to fit as many guests (computer programs, or VMs) into the fewest number of hotel rooms (physical servers) as possible to save money, but you must never overbook a room to the point where guests can't sleep (violating Service Reliability).

Currently, most hotel managers are very cautious. They leave many rooms empty just in case a guest shows up unexpectedly. This is called "over-provisioning," and it means the hotel is running at only 15–20% capacity, which is a huge waste of money.

To fix this, managers try to predict when guests will arrive and leave so they can move people around efficiently. This is the "Forecast-then-Optimize" strategy: first guess the future, then make the move.

The Problem: The "Crystal Ball" Trap

Recently, a new generation of super-smart AI "crystal balls" (called Foundation Models) has arrived. These models are trained on massive amounts of data from all over the world. They are amazing at predicting the future in a "zero-shot" way, meaning they can guess the weather in a city they've never visited without needing to study that specific city first.

However, the researchers behind this paper asked a critical question: "Just because the crystal ball is accurate at predicting the weather, does that mean it helps the hotel manager make better decisions?"

They found that existing tests only checked how accurate the crystal ball was at guessing numbers. They didn't check if those guesses actually helped the manager save money or keep guests happy.

The Solution: CloudCons (The Hotel Simulator)

The authors built a new testing ground called CloudCons. Think of it as a high-tech flight simulator for hotel managers. Instead of just checking if the AI guessed the right number, it simulates the entire process:

The Data: They fed the simulator real-world data from three giant "hotels": Huawei Cloud, Microsoft Azure, and Google Borg. These hotels have very different types of guests:
- Some are predictable (like a daily commuter).
- Some are chaotic and bursty (like a sudden flash sale).
- Some are noisy and jittery (like a party with random music).
The Test: They pitted three types of "forecasters" against each other:
- The Old School: Simple statistical rules (like "if it rained yesterday, it will rain today").
- The Deep Learners: Complex AI that needs to be trained specifically for this hotel.
- The Foundation Models: The new super-AIs that know everything about every hotel.
The Outcome: They didn't just look at prediction errors. They looked at the real-world result: Did we save money? Did we crash the system?

The Big Surprise

The paper reveals a counter-intuitive finding: Being the best at predicting the future doesn't automatically mean you are the best at managing the hotel.

The "Super-AI" Paradox: The fancy Foundation Models were indeed the most accurate at predicting the numbers. However, when the hotel manager used those predictions to pack the rooms, they didn't always save more money or prevent crashes than the simpler, cheaper models.
The "Over-Confidence" Risk: In some chaotic environments, the super-AIs were so confident in their "average" prediction that they packed the rooms too tightly. When a sudden spike happened (a "pulse"), the system crashed.
The "Simple" Winner: For very predictable, regular workloads, the old-school statistical models were actually just as good as the super-AIs, but much cheaper to run.

The Secret Lever: The "Safety Margin"

The paper discovered that the key to success isn't just which AI you use, but how you listen to it.

Most modern AIs don't just give one number (e.g., "50 guests tomorrow"). They give a range of possibilities (e.g., "It could be 40, 50, or 60"). The researchers found that the manager can choose which number to trust:

The "Gambler" Approach (Low Quantile): Trust the lower end of the prediction (e.g., "It will be 40"). This packs the hotel tight to save money, but if 60 people show up, you have a disaster.
The "Paranoid" Approach (High Quantile): Trust the upper end (e.g., "It could be 60"). This leaves empty rooms (wasting money) but guarantees no one is turned away.

The Golden Rule:

If you are running a cost-sensitive service (where saving money is the #1 goal), you should pick a "middle-of-the-road" prediction.
If you are running a mission-critical service (where a crash is unacceptable, like a hospital system), you must pick the "paranoid" high-end prediction, even if it means leaving some rooms empty.

Summary

The paper argues that we shouldn't just chase the most accurate AI crystal ball. Instead, we need to understand how that crystal ball's predictions translate into real-world decisions. Sometimes, a "good enough" prediction combined with a smart safety margin is better than a "perfect" prediction that leads to a risky decision. The authors provide a guide on how to tune that safety margin to balance saving money with keeping the lights on.

Technical Summary: CloudCons

Problem Statement

Cloud data centers currently suffer from low resource utilization (15%-20% global average CPU usage) due to conservative over-provisioning required to guarantee Service Level Agreements (SLAs). While the "forecast-then-optimize" paradigm has emerged to address this by anticipating future demands for proactive resource consolidation, existing evaluation methods are insufficient. Current benchmarks focus exclusively on prediction error metrics (e.g., MAE, MSE), failing to verify whether superior forecasting accuracy translates into better decision utility for downstream consolidation tasks. Furthermore, the emergence of time series foundation models (e.g., Moirai, Chronos, TimesFM) promises zero-shot generalization, but their practical efficacy in the specific context of cloud resource consolidation remains unverified. Existing benchmarks also lack multi-source diversity, comprehensive model coverage (often omitting foundation models or deep learning baselines), and a holistic evaluation framework that bridges the gap between prediction error and decision quality.

Methodology

The authors propose CloudCons, a comprehensive end-to-end benchmark designed to evaluate forecasting models within the specific context of cloud resource consolidation. The framework consists of three core modules:

Multi-Cloud Datasets: The authors constructed high-quality datasets by processing real-world workload traces from three major providers: Huawei Cloud (Huawei2025), Microsoft Azure (Azure2019), and Google Borg (Borg2019-d and Borg2019-e). These datasets were cleaned, interpolated, and aggregated into multiple granularities (5-min, 30-min, 1-hour). The workloads exhibit diverse characteristics, including synchronized diurnal rhythms, stochastic pulse-like bursts, and high-frequency noise.
Forecast-then-Optimize Workflow: The benchmark simulates a two-stage process:
- Stage I (Demand Forecasting): Models generate future demand sequences ( $H$ $H$ steps ahead) based on historical usage. The evaluation covers a spectrum of models:
  - Statistical: AutoARIMA, AutoETS, AutoTheta.
  - Deep Learning: DeepAR, TFT, DLinear, PatchTST (trained from scratch on each dataset).
  - Foundation Models: Moirai 2, Chronos 2, TimesFM 2.5, Sundial, TOTO, FlowState-9.1M, Kairos 50M (evaluated in a zero-shot setting).
- Stage II (Resource Optimization): Predicted demands serve as inputs to optimization algorithms to generate consolidation decisions (VM placement on Physical Machines). The problem is formulated as a dynamic bin-packing problem. Optimization methods include Heuristics (First Fit Decreasing, Best Fit Decreasing), Meta-heuristics (Ant Colony Optimization), and Exact Solvers (Gurobi/MILP).
Comprehensive Evaluation Suite: Moving beyond single error metrics, the benchmark evaluates performance across five dimensions:
- Prediction Error: MASE and CRPS.
- Resource Efficiency: Utilization ratio (Util).
- Load Balance: Peak-to-Average Ratio (PAR).
- Service Reliability: Violation Rate (VR) and Violation Severity (VS).
- Uncertainty Quantification: Prediction Interval Coverage Probability (PICP), Mean Prediction Interval Width (MPIW), and Winkler Score, assessed via Monte Carlo simulation at the PM level.

Key Contributions

End-to-End Framework: CloudCons integrates heterogeneous real-world workloads into a unified pipeline that bridges forecasting and optimization, breaking the limitation of evaluating forecasting models solely on prediction tasks.
Decision-Centric Benchmarking: It establishes a new baseline for assessing the practical utility of time series foundation models in driving consolidation decisions, rather than just their predictive accuracy.
Systematic Analysis of Quantile Selection: The work provides actionable guidelines on how the selection of predictive quantiles acts as a critical lever to balance the trade-off between resource efficiency and service reliability.

Experimental Results

The experiments address three research questions:

Forecasting Accuracy (RQ1): Foundation models (e.g., Chronos 2, TimesFM 2.5) demonstrate superior zero-shot forecasting accuracy across diverse and complex cloud environments (Huawei2025, Azure2019), outperforming statistical baselines and deep learning models trained from scratch. However, in highly regularized environments (Borg2019-e), traditional deep learning models like PatchTST achieve accuracy comparable to foundation models.
Decision Utility (RQ2): A pivotal finding is the misalignment between forecasting accuracy and decision utility. Despite superior prediction metrics, foundation models do not inherently translate into better consolidation outcomes. In some cases (e.g., Azure2019), deep learning models like PatchTST and DeepAR outperform foundation models in reliability metrics (VR, VS) by better capturing peak loads, even if their overall prediction error is higher. In highly regularized workloads, traditional cost-efficient models often provide more pragmatic solutions than foundation models.
Quantile Sensitivity (RQ3): The selection of predictive quantiles ( $\alpha$ $α$ ) significantly impacts the efficiency-reliability trade-off.
- $\alpha \in [0.4, 0.6]$ (Median-based): Suitable for cost-sensitive applications, maximizing resource utilization but risking higher violation rates in volatile environments.
- $\alpha \geq 0.8$ (High-quantile): Critical for mission-critical services. Shifting to higher quantiles significantly mitigates resource contention risks with limited impact on utilization in unpredictable scenarios.
- Uncertainty: Foundation models like Chronos 2 show exceptional probabilistic calibration, while others (e.g., FlowState) may exhibit distributional misalignment, leading to deceptive safety margins.

Significance and Claims

The paper claims that CloudCons fills a critical gap in the literature by shifting the evaluation focus from pure prediction error to decision utility. The authors assert that high forecasting accuracy is not a sufficient condition for optimal resource consolidation. Consequently, the benchmark offers vital insights for real-world deployment:

Model Selection: Foundation models are compelling for complex, volatile environments due to zero-shot generalization, but traditional models remain viable and potentially more efficient for stable, periodic workloads.
Operational Guidelines: The study provides concrete recommendations for calibrating predictive quantiles. Operators should not rely solely on median forecasts; instead, they must strategically select quantiles based on their specific reliability requirements (e.g., using $\alpha \geq 0.8$ for high-reliability services) to balance the trade-off between resource efficiency and service reliability.

The work concludes that bridging the gap between forecasting and decision-making requires evaluation metrics that reflect downstream system performance, rather than just statistical error.