💻 computer science

Predictive and Adaptive Resource Scheduling for Kubernetes–Ceph Hyperconverged Infrastructure on Proxmox VE

This paper proposes and evaluates a predictive, adaptive scheduling model that integrates workload forecasting with Ceph-aware storage decisions to significantly reduce resource contention and I/O latency in Kubernetes–Ceph hyperconverged infrastructure running on Proxmox VE, achieving superior load balancing and performance compared to the default Kubernetes scheduler.

Original authors: Doston Khasanov, Abdurauf Abdullaev, Halimjon Khujamatov, Temirbek Toshtemirov, Alisher Mamatov, Razvan Craciunescu

Published 2026-07-23

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Doston Khasanov, Abdurauf Abdullaev, Halimjon Khujamatov, Temirbek Toshtemirov, Alisher Mamatov, Razvan Craciunescu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling city where the roads, the power grid, and the water supply all live inside the same neighborhood. In the world of modern computing, this is called "hyperconverged infrastructure." Instead of having separate buildings for servers (computing), hard drives (storage), and network cables, everything is packed tightly together on the same physical machines. It's efficient and saves space, but it creates a tricky traffic problem. If a massive delivery truck (a data-heavy program) tries to drive down a street while a construction crew (a storage task) is working right next to it, everything gets stuck.

To manage this digital city, we use a system called Kubernetes. Think of Kubernetes as the city's traffic controller. Its job is to decide which building (server) gets which new delivery truck (software program, or "pod"). However, the standard traffic controller is a bit old-fashioned. It only looks at what is happening right now. It sees a building is empty and says, "Great, send the truck there!" It doesn't realize that the building's power grid is already struggling because of a nearby construction project, or that the truck is about to arrive with a load that will cause a traffic jam in five minutes. This reactive style often leads to some buildings being crushed under too much work while others sit idle, and the "roads" (data storage) get clogged, slowing everything down.

This is the puzzle a team of researchers from universities in Uzbekistan and Romania decided to solve. They asked: What if our traffic controller could peek into the future? What if it could predict where the traffic jams will be before they happen and move the trucks accordingly? In their paper, they built a smarter, "predictive and adaptive" system that doesn't just react to the present but plans for the immediate future. By combining a simple prediction tool with a new way of placing software, they managed to smooth out the chaos in their test city, proving that sometimes, a little bit of foresight is better than a super-complex brain.

The Problem: The Reactive Traffic Controller

In the digital world, the researchers set up a "hyperconverged" testbed using three main tools: Proxmox VE (the foundation that holds the servers), Ceph (the storage system that acts like a giant shared hard drive), and Kubernetes (the traffic controller).

In a standard setup, Kubernetes plays it safe. It waits until a server is running a program, checks how much CPU (brainpower) and memory (short-term memory) is being used, and then decides where to put the next program. It's like a traffic cop who only sees the cars currently on the road. If a server looks free, the cop sends a new car there. But in a hyperconverged system, the "car" might need to access the "storage" (the hard drive) on that same server. If the cop doesn't know that the server's storage is already busy, the new car gets stuck, causing a delay.

The researchers found that this "reactive" approach leads to three big headaches:

Resource Contention: Too many programs fighting for the same CPU or storage at the same time.
Load Imbalance: Some servers are sweating buckets (running at 95% capacity) while others are napping (running at 40% capacity).
Storage Latency: The time it takes to read or write data gets slower because the storage system is overwhelmed.

The Solution: A Crystal Ball and a Flexible Map

The team proposed a new model that acts like a traffic controller with a crystal ball and a flexible map. Their system has four main parts working together in a loop:

The Crystal Ball (Prediction): Instead of just looking at the current moment, the system uses a mathematical trick called "Exponentially Weighted Moving Average" (EWMA). Think of this as a weather forecaster who looks at the last few days of rain to guess if you need an umbrella tomorrow. It predicts how much CPU and storage a program will need in the next few minutes. They found that a specific setting for this "forecast" worked best, predicting CPU needs with about 8.4% error, memory with 5.1% error, and storage needs with 12.3% error.
The Flexible Map (Adaptive Scheduling): Once the system knows what's coming, it doesn't just dump the program on the first empty server. It calculates the "predicted load" for every server. It asks, "If I put this program here, will the server be overloaded in five minutes?" If the answer is yes, it skips that server and finds a better spot.
The Storage-Aware Decision: This is the secret sauce. The system doesn't just look at CPU; it also checks the health of the Ceph storage (the "OSD" or storage daemons). If a server has a busy storage drive, the system knows to send the heavy data programs elsewhere, even if the CPU looks free.
The Dynamic Adjustment: If a program suddenly needs more power than expected, the system can automatically adjust its limits without crashing or restarting, keeping the flow smooth.

The Experiment: A Three-City Test

To see if this worked, the researchers built a real, small-scale city using three physical computers (nodes). Each node had a powerful processor, 32 GB of RAM, and two fast NVMe SSDs (super-fast hard drives). They filled this city with different types of traffic:

CPU-heavy trucks: Programs that just crunch numbers (like stress-ng).
Storage-heavy trucks: Programs that read and write massive amounts of data (like fio).
Mixed traffic: Databases and web apps that do a bit of everything (like YCSB).

They ran two scenarios side-by-side for 30 minutes.

Scenario A (The Old Way): Standard Kubernetes with no prediction.
Scenario B (The New Way): Their predictive, adaptive model.

The Results: Smoother Streets, Faster Delivery

The results were a clear win for the new model, and the numbers tell a vivid story of improvement.

1. Balancing the Load:
In the old way, the traffic was wildly uneven. One server was screaming under the pressure, running at 95.64% capacity, while another was barely working at 39.73%. The "imbalance" (the difference between the busiest and quietest server) was a chaotic 53.29%.
With the new model, the traffic smoothed out perfectly. The busiest server dropped to 67.41%, and the quietest one woke up to 56.33%. The imbalance plummeted to just 10.98%. That is a 79.4% reduction in chaos. The new system kept all servers in a tight, happy band of 54% to 68% usage, meaning no one was overworked and no one was bored.

2. Speeding Up Storage:
Because the new system knew where the storage was busy, it avoided clogging the roads. The average time it took to apply storage changes (latency) dropped from 1.11 ms to 1.00 ms. While that sounds like a tiny difference, in the world of data, it means the system is more consistent and reliable. The "worst-case" delay (the 95th percentile) also improved from 1.15 ms to 1.00 ms, showing that the system handled the heavy traffic much better.

3. No Extra Cost:
The researchers were careful to note that they didn't need to rebuild the entire city or use expensive, complex AI brains. They used standard tools (Kubernetes and Prometheus APIs) and a simple prediction method. They proved that you don't need a super-complex algorithm to get great results; you just need to connect the dots between computing and storage.

The Takeaway: Integration Over Complexity

The most exciting part of this paper isn't just that it worked, but why it worked. The researchers argue that the problem wasn't that the old traffic controller was too dumb; it was that it was looking at the problem in only one dimension. It saw the CPU but ignored the storage.

By simply adding a layer of "foresight" and "storage awareness" to the existing system, they achieved massive gains. They showed that architectural integration (making the computing and storage talk to each other) is more powerful than just making the algorithms more complicated. Even a simple, lightweight prediction tool, when combined with smart placement, can fix the biggest bottlenecks in a hyperconverged system.

In the end, this paper suggests that the future of efficient computing isn't necessarily about building bigger, smarter brains, but about making sure the different parts of the system are holding hands and looking ahead together. The traffic controller doesn't need to be a genius; it just needs to know what's coming around the corner.

Technical Summary: Predictive and Adaptive Resource Scheduling for Kubernetes–Ceph Hyperconverged Infrastructure on Proxmox VE

Problem Statement
Hyperconverged infrastructure (HCI) integrates compute, storage, and networking on the same physical nodes, yet standard Kubernetes scheduling operates reactively based on instantaneous resource metrics. This reactive approach lacks awareness of the underlying Ceph storage state, leading to resource contention, load imbalance, and elevated I/O latency when compute and storage workloads compete for the same physical resources. Existing research has largely addressed scheduling and storage optimization in isolation or focused on disaggregated architectures, failing to provide an integrated framework that couples short-horizon workload prediction with storage-aware placement on a single HCI control plane.

Methodology
The authors propose a unified, closed-loop control model that operates entirely through standard Kubernetes and Prometheus APIs without modifying core system components. The architecture consists of four interconnected stages:

Monitoring: Direct metric collection from Kubernetes, Ceph, and Proxmox VE via APIs.
Prediction: A lightweight workload demand forecaster using an Exponentially Weighted Moving Average (EWMA) algorithm. This component predicts resource demands (CPU, memory, I/O) for the next scheduling interval based on historical data. The study identified a smoothing factor ( $\alpha$ ) of 0.3 as empirically optimal, achieving Mean Absolute Percentage Errors (MAPE) of 8.4% for CPU, 5.1% for memory, and 12.3% for I/O.
Decision-Making (Adaptive Scheduling): The scheduler utilizes predicted demand rather than current load to make placement decisions. It minimizes load imbalance ( $\Delta_k$ ) and ensures projected resource usage does not exceed node capacity.
Adaptation (Storage-Aware Placement): The model incorporates Ceph OSD load and apply latency into the scoring function for pod placement. This prevents storage-intensive workloads from being placed on nodes already burdened by high I/O or storage latency, effectively coupling compute scheduling with storage state.

The system was implemented and evaluated on a real three-node testbed running Proxmox VE 9.1.4, Ceph 19.2.3, and Kubernetes 1.29.15. Experiments involved mixed workloads including CPU-intensive (stress-ng), I/O-intensive (fio), and transactional (YCSB) tasks.

Key Results
Compared to the default Kubernetes scheduler, the proposed model demonstrated significant performance improvements:

Load Balancing: CPU load imbalance was reduced by 79.4%, dropping from an average of 53.29% to 10.98%.
Utilization Efficiency: The maximum node utilization decreased from 95.64% to 67.41%, while minimum utilization increased from 39.73% to 56.33%, effectively eliminating resource hotspots and idle nodes.
Storage Performance: Average Ceph OSD apply latency improved from 1.11 ms to 1.00 ms, with the 95th percentile latency also dropping from 1.15 ms to 1.00 ms.
Stability: The Ceph cluster maintained a HEALTH_OK status throughout the evaluation, and the model successfully managed transient load spikes without requiring pod migrations or system restarts.

Significance and Claims
The paper argues that the primary bottleneck in HCI performance is not a lack of algorithmic sophistication in prediction, but rather the inherent reactive, single-layer architecture of default Kubernetes scheduling. The authors claim that architectural integration across compute and storage layers yields greater performance gains than increasing algorithmic complexity alone.

Key contributions highlighted by the authors include:

Unified Framework: A structural description of a predictive and adaptive framework that coalesces demand forecasting, adaptive scheduling, dynamic resource allocation, and storage-aware placement.
Cross-Layer Coordination: A mechanism that optimizes placement decisions by incorporating predicted I/O demand and Ceph OSD load, preventing storage-heavy workloads from landing on already busy nodes without modifying core system components.
Practical Applicability: The model relies on standard APIs and lightweight heuristics (EWMA) rather than complex machine learning models, making it deployable in real-world open-source HCI environments.
Empirical Evidence: The study provides reproducible evidence from a physical testbed confirming that cross-layer integration significantly improves CPU balancing and reduces storage latency under realistic mixed workloads.

The authors conclude that even simple, lightweight prediction mechanisms, when combined with adaptive scheduling and storage-aware decision-making, can achieve substantial performance improvements (79.4% reduction in imbalance) that surpass those reported by more complex machine learning-based approaches in related studies.