💻 computer science

OmniPilot: An Uncertainty-Aware LLM Inference Advisor for Heterogeneous GPU Clusters

OmniPilot is an uncertainty-aware inference advisor for heterogeneous GPU clusters that predicts serving costs and abstains from unreliable configurations using conformally calibrated quantile models and out-of-distribution detection, thereby optimizing economic utility with high accuracy across diverse hardware and precision settings.

Original authors: D. Balamurugan, Thomas W. Bush

Published 2026-07-03✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: D. Balamurugan, Thomas W. Bush

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, shared spaceship (a GPU cluster) carrying hundreds of different passengers (AI models) who all want to travel to different destinations. Before the ship can even leave the dock, you have to decide: Which engine should we use? How many engines should we link together? And what kind of fuel should we burn?

If you guess wrong, the ship might sputter, run out of fuel, or crash before it even takes off. This is the problem OmniPilot solves.

Here is the story of OmniPilot, explained simply:

1. The Problem: Guessing is Expensive

In the world of AI, running a "Large Language Model" (like the one you are talking to right now) is like driving a race car. You have to choose the car (the GPU), how many drivers share the wheel (Tensor Parallelism), and how much detail the car needs to process (Precision).

The paper found that if you just guess or use a "one-size-fits-all" rule, you fail 14% of the time. Sometimes the car is too heavy for the engine, or the fuel tank is too small. In a shared spaceship, a failed launch wastes expensive time and money that could have been used by someone else.

2. The Solution: The "Uncertainty-Aware" Co-Pilot

The authors built OmniPilot, a smart advisor that acts like a seasoned co-pilot. Instead of just giving you a single answer, it does two special things:

It predicts the future (with a safety net): It looks at your request and says, "If you use this engine with this fuel, you will likely get this much speed." But it doesn't just guess; it gives you a confidence range. It's like saying, "You'll probably get 100 miles per hour, give or take 5 miles."
It knows when to say "I don't know": This is the most important part. If you ask for a configuration the co-pilot has never seen before (like a brand new type of engine or a weird fuel mix), it doesn't guess. It raises a red flag and says, "Stop! I haven't tested this. I'm not confident enough to recommend it." This is called abstention. It prevents you from crashing the ship on untested ground.

3. How It Learns: The "Training Logbook"

OmniPilot isn't magic; it's a student that learned from a massive logbook of past trips.

It studied 500,000 past jobs from the cluster to understand how often ships fail to launch.
It ran 460 specific test runs on different types of high-tech engines (A100, H100, H200) to learn exactly how they behave.
It learned that the rules are tricky. For example, sometimes using "less detailed" fuel (quantization) makes the car slower, not faster, depending on the engine. A simple rulebook would miss this, but OmniPilot learned these weird quirks.

4. The Results: Smarter Choices

When the researchers tested OmniPilot against standard ways of making decisions:

Accuracy: It picked the best engine/fuel combo 95% of the time.
Cost: It saved a lot of "node-hours" (time and money) by avoiding bad choices.
Safety: When they tried to trick it with a brand-new, untested scenario, the system correctly said, "I don't know," instead of giving a wrong answer.

5. What It Doesn't Do (The Boundaries)

The paper is very clear about what OmniPilot is not:

It doesn't fix the engine while it's running (that's for other tools like vLLM).
It doesn't handle training new AI models (teaching the AI), only running them (inference).
It doesn't work if the ship is broken (hardware failures are out of scope for this specific tool).

The Big Takeaway

OmniPilot is like a smart travel agent for AI. It looks at your destination, checks the weather, reviews the history of every plane in the fleet, and tells you exactly which plane to book to get there fastest and cheapest. Most importantly, if you ask for a flight path it has never flown, it honestly tells you, "I can't recommend that yet," saving you from a disastrous trip.

By combining data-driven predictions with a honest "I don't know" button, OmniPilot helps operators run their AI clusters more efficiently and safely.

Technical Summary: OmniPilot

Problem Statement

Serving Large Language Models (LLMs) on shared, heterogeneous GPU clusters presents a complex pre-submission decision problem. Operators must select the GPU type, tensor-parallel (TP) degree, and numerical precision before committing valuable node-hours. This process is fraught with challenges:

High Failure Rates: Blind configuration selection leads to immediate failures due to hardware incompatibilities or out-of-memory (OOM) exhaustion. In the authors' benchmark runs, 14% of jobs failed immediately.
Non-Intuitive Interactions: Static deployment recipes fail to capture intricate hardware-software dynamics. For instance, 4-bit quantization can be slower than 16-bit on specific runtimes, and Key-Value (KV) cache pressure creates non-monotonic size-by-precision trade-offs.
Data Limitations: Traditional cluster schedulers (e.g., Slurm) lack visibility into workload-specific behaviors (model family, precision, concurrency) and rely on stale telemetry (e.g., 60-second DCGM sampling intervals) that misrepresents active serving performance.
Uncertainty: Existing systems optimize execution within a pre-selected configuration but do not provide calibrated uncertainty estimates for choosing that configuration, leaving operators unable to distinguish between supported and unsupported (Out-of-Distribution) requests.

Methodology

OmniPilot is a launch advisor designed to predict serving costs and recommend optimal configurations while explicitly handling uncertainty. Its architecture consists of three core components:

1. Data Infrastructure

OmniPilot utilizes two distinct data stores to ensure lineage and reproducibility:

Personal Store: A controlled benchmark grid (1,667 rows) covering specific model families, GPUs, and precisions.
Cluster-wide Store: Aggregated telemetry from 500,000 historical jobs, used to derive launch-success priors and demand patterns.
Feature Engineering: The system encodes ~28 features, including log-transformed numerics (model size, concurrency), linear numerics (TP degree, memory), and one-hot encodings for model families, GPU kinds, and quantization formats. Crucially, new axes (e.g., quantization types) are represented in the featurizer before data collection to prevent feature leakage.

2. Inference Cost Model

The core predictor is a set of gradient-boosted regressors using quantile loss, trained in log-space to handle multiplicative errors.

Targets: It predicts eight targets, including aggregate tokens/s, request throughput, Time to First Token (TTFT), cold-start time, peak KV-cache usage, and average power.
Calibration: The model employs Conformalized Quantile Regression (CQR) to generate prediction intervals. It uses a split-conformal additive offset calibrated on held-out cell residuals to achieve 80% coverage on unseen workloads.
Abstention Mechanism: To handle Out-of-Distribution (OOD) requests, OmniPilot uses a support-distance check. If a request falls outside the measured support envelope (defined across seven axes), the system flags it as low-confidence and abstains, regardless of interval width.

3. Decision Utility and Placement

OmniPilot ranks configuration "arms" (combinations of GPU, TP, and precision) using an economic utility metric calibrated to operator preferences.

Utility Function: $Utility = (Throughput \times Runtime \times Value) - (Compute Cost + Queue Wait + Expected Failure Cost + SLO Violation Cost + KV-Saturation Penalty)$ .
Launch-Success Prior: A beta-binomial estimator shrinks per-configuration failure rates toward a pooled cluster rate, accounting for the observation that multi-GPU configurations (e.g., TP=4) have significantly lower launch success (~38%) than single-GPU setups.
KV-Saturation Penalty: A multiplicative throughput penalty is applied when KV-cache occupancy exceeds 0.85, converting preemptions into utility costs.

Key Contributions

Calibrated, Abstention-Aware Advisor: OmniPilot is the first system to combine conformally calibrated quantile cost models with an explicit OOD abstention layer for LLM inference. It predicts performance metrics alongside calibrated intervals and a safety signal to block unsupported requests.
Decision-Coupled Evaluation: The paper introduces a leakage-free evaluation framework using GroupKFold (grouped by workload cell) to compare the advisor against model-free baselines. It demonstrates that optimizing for economic utility (incorporating cost and failure risk) yields significantly better placement accuracy than throughput-only or rule-based policies.
Cluster Telemetry Study (Negative Result): The authors demonstrate that while scheduler metadata (Slurm) is sufficient for deriving demand and launch-success priors, it is insufficient for building a cluster-wide throughput advisor because it records resource allocation (how a GPU was used) rather than the specific software workload (what ran on it).

Results

Evaluated across 460 benchmark runs on A100, H100, and H200 hardware:

Prediction Accuracy: OmniPilot predicts aggregate throughput with a 6.2% Mean Absolute Percentage Error (MAPE) and a log-space $R^2$ of 0.92.
Placement Accuracy: The advisor achieves 95% top-1 accuracy in selecting the optimal configuration, with a mean utility regret of 0.003. This is an order of magnitude better than baselines like "size-threshold" rules (regret 1.97) or "throughput-max" policies (regret 4.73).
OOD Handling: On a holdout set of 5 unsupported cells (new model families or extreme contexts), prediction error rose to 24–46%, and conformal intervals failed to cover the true values. However, the abstention layer successfully flagged all five as low-confidence, preventing risky deployments.
Data Efficiency: Accuracy saturates after approximately 150 training rows once the task axes are covered, indicating that coverage of the configuration space is more critical than raw data volume.

Significance and Claims

The paper claims that OmniPilot fills a critical macro-level gap in multi-user, shared GPU clusters by transforming hardware configuration selection into a calibrated economic decision. Its significance lies in:

Safety and Reliability: By explicitly abstaining on OOD requests, it prevents costly failures that static recipes or "best-effort" schedulers would incur.
Economic Optimization: It moves beyond simple throughput maximization to optimize for a complex utility function that balances cost, failure risk, and SLOs, aligning with real-world operator constraints.
Methodological Rigor: It establishes a framework for "feature-before-data" design and demonstrates that many performance issues are better resolved through refined utility design (e.g., KV-saturation penalties) or measurement precision (e.g., freshness windows) rather than simply increasing model capacity.

The authors remain modest about the scope, noting that the system is currently limited to single-node vLLM inference serving. Training workloads, fault recovery, and multi-node serving architectures are explicitly deferred to future research. The system relies on empirical data from a specific cluster (Kempner AI), and absolute thresholds (e.g., KV saturation knee) are deployment-specific.