💻 computer science

History-First Reliability-Gated Fusion for Sampled-Peak System and GPU Memory Utilization Prediction in HPC Clusters

This paper presents a history-first, reliability-gated fusion framework that combines exact-script historical caching with a LoRA-adapted Qwen2.5-Coder encoder to significantly improve the prediction accuracy of system and GPU memory utilization in HPC clusters while reducing computational overhead through selective inference.

Original authors: Pan Chang, Xueru Chen, Guangchao Hu

Published 2026-09-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pan Chang, Xueru Chen, Guangchao Hu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, humming halls of high-performance computing, where supercomputers solve problems too complex for any single machine, a quiet but critical challenge persists: resource management. These clusters are like massive cities of processors, memory banks, and graphics accelerators, all working in concert. To keep the city running smoothly, administrators must decide how much power to allocate to each task before it even begins. Currently, they rely on the job's own request—a user or a script asking for a certain amount of memory or time. However, these requests are often conservative guesses, made to ensure the job doesn't crash. When everyone asks for more than they need, the city becomes fragmented; valuable space sits empty while other tasks wait in line. The goal of modern research in this field is to move beyond these rough guesses and predict exactly how much a job will actually use, allowing the system to pack tasks more tightly and efficiently.

The core difficulty lies in the nature of the work itself. A job's behavior is not just a number; it is a story written in code. Sometimes a user runs the exact same script they used yesterday, and the result is predictable. Other times, they tweak a single line of code, change a data file, or run on a different type of computer, and the resource usage can shift dramatically. Traditional methods that look only at past numbers often fail when the script changes, while methods that try to read the code from scratch can be slow and computationally expensive. The question researchers faced was whether they could build a system that knows when to trust the past and when to read the new code, blending these two sources of information to make a precise prediction before the job even starts.

A team of researchers from East China Normal University, Renmin University of China, and New York University Shanghai set out to solve this by creating a new kind of prediction engine for high-performance computing clusters. They focused on two specific resources: the system memory that holds data for general processing, and the video memory, or VRAM, that graphics cards use for heavy calculations. Their approach, which they call a "history-first reliability-gated cascade," acts like a smart traffic controller. Instead of forcing every job to undergo a complex analysis, the system first checks if there is a reliable record of that exact same script running successfully in the past. If the history is clear and trustworthy, the system uses that past data immediately, skipping the heavy lifting. This is the "bypass" mechanism, designed to save time and energy when the answer is already known.

However, the system is not blind to change. If the script is new, or if the past history is too scattered to be reliable, the system activates a sophisticated code-reading tool. This tool, based on a specialized artificial intelligence model trained to understand programming languages, analyzes the submitted script, the user's identity, and the specific requests made for the job. It reads the code to understand the intended logic, looking for clues about how much memory or graphics power the job will actually need. The researchers did not simply let the AI guess; they built a second layer of decision-making that happens after the AI has spoken. This layer compares the AI's prediction with the available historical data. If the history is strong, it pulls the final prediction toward the past record, but only within a safe, calculated range to prevent wild swings. If the history is weak, it lets the AI's reading of the code take the lead. This dynamic blending ensures that the system adapts to the specific situation of every single job.

To test this method, the researchers examined a real-world dataset from a production cluster containing nearly 20,000 completed jobs over an eleven-and-a-half-month period. They compared their new system against several existing methods, including simple lookups of the last time a user ran a job and standard machine learning models that rely only on metadata. The results showed that their hybrid approach was the most accurate. For predicting system memory usage, the new method reduced the average error by nearly five percent compared to the best previous method. For predicting graphics memory usage, the improvement was even more significant, cutting the average error by nearly fourteen percent. The system was particularly effective at identifying jobs that would stay within a safe margin of error, a crucial factor for administrators who need to avoid overloading the machines.

The study also revealed important nuances about how these predictions work. The researchers found that the system's success depended heavily on the type of job. When a user ran the exact same script repeatedly, the simple history-based approach was often just as good as the complex AI, proving that past performance is a strong indicator for repetitive tasks. However, when the script changed or no exact history existed, the code-reading AI became essential, providing insights that pure history could not offer. The system learned to recognize these different regimes, switching strategies automatically. In terms of speed, the researchers measured the time it took for the system to process a single job on a high-end graphics card. Without any shortcuts, the analysis took about thirty-one milliseconds, a fraction of a second that fits well within the operational needs of a busy computing cluster. By using the bypass mechanism, the system avoided this heavy analysis for nearly half of the jobs, further improving efficiency.

The researchers were careful to note the limits of their findings. The study was conducted on a specific cluster with a specific mix of hardware and workloads, so the results reflect that particular environment. They also emphasized that their predictions are based on jobs that completed successfully; the system does not predict failures or crashes, only the peak usage of resources for jobs that run to completion. Furthermore, the predictions are intended as planning aids to help administrators make better decisions, not as a guarantee that the system can be overloaded safely. The data used for the study included real executable scripts, which contain sensitive information, so the researchers could not release the raw data publicly, though they have made the code and derived data available for academic review under specific agreements.

Ultimately, this work demonstrates that the future of resource prediction lies not in choosing between history and code, but in intelligently fusing them. By creating a system that knows when to trust the past and when to read the present, the researchers have provided a practical tool for making high-performance computing clusters more efficient. The method suggests that with the right combination of reliability checks and adaptive learning, we can move closer to a state where computing resources are utilized with precision, reducing waste and allowing more complex scientific work to be done. The findings offer a clear path forward for managing the growing demands of modern computing, proving that a little bit of history, guided by a deep understanding of code, can go a long way.

Technical Summary: History-First Reliability-Gated Fusion for Sampled-Peak System and GPU Memory Utilization Prediction in HPC Clusters

1. Problem Statement

High-performance computing (HPC) clusters frequently suffer from capacity fragmentation due to conservative resource requests (CPU, memory, GPU, walltime) submitted by users. While these requests prevent resource exhaustion, persistent overestimation weakens admission and placement decisions. Existing predictors rely heavily on scheduler metadata, user identity, or recent utilization history. While efficient, these methods fail to capture changes in executable logic, command-line arguments, or embedded code unless explicitly reflected in specific scheduler fields. Conversely, relying solely on code models (e.g., Large Language Models) imposes inference costs even when reliable historical data for the exact same script exists. Furthermore, fixed historical weights can be detrimental when workloads change or when no exact-script predecessor is available.

The core challenge is to predict sampled-peak resource utilization (specifically system memory and GPU VRAM) at submission time by effectively fusing executable job scripts, submission-time requests, user identity, and strictly event-time-eligible history (jobs completed before the current submission), while minimizing inference costs and handling cold-start scenarios.

2. Methodology: History-First Reliability-Gated Cascade

The authors propose a history-first reliability-gated cascade that dynamically decides whether to bypass a code encoder in favor of historical data or to fuse both sources. The system operates on a chronological event-time protocol where labels are only available after job completion.

A. Core Components

Multimodal Code Predictor (Qwen2.5-Coder-1.5B):
- A LoRA-adapted Qwen2.5-Coder-1.5B-Instruct backbone encodes executable scripts with a 4096-token context (60% head, 40% tail retention).
- It fuses script embeddings with user embeddings (64-dim) and standardized metadata (requested CPU, memory, GPU, walltime, memory/core) into a 256-dimensional space.
- Two prediction heads output standalone estimates for memory and VRAM utilization ratios.
Robust Event-Time History Experts:
- Constructs history based on jobs completed before the current submission.
- Maintains four candidate estimators: Last-1, Median-3, Median-5, and an Exponentially Time-Decayed Mean (168-hour half-life).
- Prioritizes exact-script history (same SHA-256 redacted script) over same-user history.
Pre-Inference Reliability Gate (Bypass Mechanism):
- A lightweight gradient-boosted classifier ( $C_{r,e}$ ) evaluates whether a specific history candidate is reliable enough to bypass the Qwen encoder.
- Condition: Bypass is permitted only if exact-script history exists and the classifier predicts the history estimate is within a 0.10 tolerance of the Qwen prediction.
- If bypassed, the system uses the historical estimate directly, saving inference cost.
Post-Inference Dynamic Fusion:
- If the encoder is invoked, a second gradient-boosted regressor ( $G_{r,e}$ ) learns a per-job weight ( $\gamma_{jr}$ ) to blend the Qwen output ( $Q_{jr}$ ) and the history estimate ( $H_{jr}$ ).
- Trust Region: For exact-script history, a dispersion-aware trust region limits deviations from the historical value, clipping the final prediction to $[H_{jr} - d_{jr}, H_{jr} + d_{jr}]$ , where $d_{jr}$ depends on the Median Absolute Deviation (MAD) of the history.
- The final output is clipped to $[0, 1]$ .

B. Training and Selection Protocol

Data: 19,321 effective jobs from an 11.5-month production trace (Aug 2025–July 2026) on a heterogeneous Slurm cluster.
Splitting: Chronological split (Train/Val/Test) with a "completion-boundary purge" to ensure no future labels leak into training or validation.
Policy Selection: Validation blocks are used to jointly select the best history expert and bypass threshold for each resource. These policies are frozen before test replay.
Loss: Masked Smooth L1 loss with resource-specific weighting (VRAM weighted twice as heavily as memory).

3. Key Contributions

Formulation: A novel formulation for submission-time sampled-peak prediction integrating executable scripts, metadata, and strictly event-time-eligible history with explicit resource-valid masks.
Architecture: A history-first reliability-gated cascade that combines:
- Conservative exact-script bypass (avoiding encoder cost when history is reliable).
- Per-job post-inference fusion (adapting the code-history mix).
- A dispersion-aware trust region to prevent large deviations in repetitive regimes.
Empirical Validation: Evaluation on 19,321 jobs across five training seeds, demonstrating significant reductions in Mean Absolute Error (MAE) compared to strong baselines.
Efficiency Analysis: Characterization of implementation trade-offs, including routing rates, cache hit rates, and measured inference latency (31.31 ms on RTX 4090 D).

4. Results

The proposed cascade was evaluated against baselines including XGBoost, TextCNN, BiLSTM, and a "Hierarchical Same-Script Last-1" (which prioritizes exact history but lacks dynamic fusion).

Performance Metrics (Mean ± Std Dev over 5 seeds):
- Memory: MAE 0.0818, RMSE 0.1777, Acc@10% 79.46%.
- VRAM: MAE 0.0823, RMSE 0.1634, Acc@10% 77.11%.
Improvements:
- Relative to Hierarchical Same-Script Last-1, the cascade reduced mean MAE by 4.83% (Memory) and 13.70% (VRAM).
- VRAM MSE and RMSE decreased by 36.61% and 20.38% respectively.
Statistical Significance:
- Paired day-block bootstrap intervals (across 67 days) consistently support lower VRAM MAE, MSE, and RMSE for the proposed method across all seeds.
- Memory improvements and Acc@10% differences were statistically inconclusive (intervals included zero), though the mean MAE was lower.
Subgroup Analysis:
- The cascade excels when exact-script history is absent (dynamic fusion provides value).
- When exact-script history is present, the "Hierarchical Same-Script Last-1" remains highly competitive, particularly for VRAM, validating the "history-first" philosophy.
Efficiency:
- Routing: 49.71% of test jobs required an encoder forward after routing and exact-cache reuse (down from 51.60% without bypass).
- Latency: Uncached batch-one inference averaged 31.31 ms on an NVIDIA GeForce RTX 4090 D.

5. Significance and Claims

The paper claims that reliability-aware fusion with selective encoder execution is a practical direction for sampled-peak resource prediction.

Complementarity: The results support the view that code models and historical data are complementary. Code models provide value when history is unavailable or behavior changes, while history is superior for repetitive, exact-script workloads.
Operational Feasibility: The system demonstrates that integrating a large code model (Qwen2.5-Coder-1.5B) into a production pipeline is feasible, provided that a gating mechanism minimizes unnecessary inference (bypassing the encoder for ~35% of jobs and utilizing exact-cache reuse for another ~24% of routed jobs).
Modesty: The authors explicitly state that point-error improvements do not guarantee safe oversubscription or immediate scheduler-level savings. They frame the predictors as planning aids rather than complete oversubscription policies, noting that underprediction and overprediction have different operational consequences. The study characterizes the observed environment and does not claim universal superiority across all HPC clusters or workloads.

In summary, the paper presents a robust, event-time-constrained framework that effectively balances the predictive power of code language models with the efficiency and reliability of historical data, achieving state-of-the-art accuracy on production HPC traces while maintaining operational efficiency.