🤖 AI

Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

Coral is an adaptive, heterogeneity-aware multi-LLM serving system that jointly optimizes resource allocation and serving strategies across diverse cloud GPUs, achieving up to 2.79 $\times$ cost reduction and 2.39 $\times$ higher goodput by using a lossless two-stage decomposition to enable fast, near-optimal decision-making.

Original authors: Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, K. V. Rashmi

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yixuan Mei, Zikun Li, Zixuan Chen, Shiqi Pan, Mengdi Wu, Xupeng Miao, Zhihao Jia, K. V. Rashmi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-end restaurant kitchen. In the past, everyone believed you needed the most expensive, state-of-the-art head chef and the largest, fastest stove to prepare every single dish. However, in the world of Large Language Models (LLMs) – the AI brains behind chatbots and code generators – this has changed. No single "AI model" is the best at everything. Some excel at writing code, others at chatting, and some are only suitable for simple tasks.

Furthermore, the "kitchen" (the cloud servers) is a chaotic mix of varying equipment. You have super-fast, expensive "H100" stoves, yet these are often sold out. You also have older, cheaper "L4" or "A100" stoves that are available and surprisingly offer just as much cooking power per dollar.

The problem is: How do you run a kitchen with 46 different recipes (models) using a mix of expensive and cheap stoves without wasting money or making customers wait?

This is exactly the problem Coral solves.

The Old Way: The "One-Size-Fits-All" Mistake

Previous systems tried to solve this by treating every dish as a black box. They said: "For the code-writing dish, we need a whole H100 stove. For the chat dish, we need an A100." They did not look inside the recipe to see if they could mix and adapt parts of the stove.

This led to two major problems:

Wasted Money: They often bought the most expensive equipment, even if a cheaper mix would have worked just as well.
Bottlenecks: When the expensive stoves ran out, the entire kitchen came to a halt, even though they had many cheaper stoves sitting idle.

The Coral Solution: The "Intelligent Mix-and-Match" Head Chef

Coral is like a brilliant head chef who realizes that a single dish does not need to be prepared on just one type of stove. Instead, Coral breaks down the cooking process into phases and assigns the perfect stove to each stage.

Here is how it works, using a simple analogy:

1. The Two-Phase Strategy (Offline vs. Online)

Coral splits its thinking into two parts to avoid being overwhelmed:

Phase 1: The Recipe Book (Offline)
Before the restaurant opens, Coral spends time creating a massive "recipe book." For every possible dish (model) and every possible combination of stoves (GPUs), it calculates the absolute best way to prepare it.
- Example: It determines that for the "Qwen-3 235B" dish, the best way to cook the first part (Prefill) is to use three cheap L40S stoves and two expensive H100 stoves in a specific sequence.
- It writes these perfect "recipes" as Serving Templates. This takes a long time but only needs to happen once.
Phase 2: The Daily Shift (Online)
When the restaurant opens and orders come in, Coral does not stop to rethink from scratch how to cook. It simply looks into its Recipe Book.
- It checks: "We currently have 5 L40s and 2 H100s available. We need to prepare 100 Qwen-3 dishes."
- It immediately selects the pre-calculated recipe that fits these specific stoves and meets the customer's speed requirements.
- Since the hard math was done in advance, Coral can make these decisions in seconds, not hours. This allows the kitchen to adapt instantly if a stove fails or a new order floods in.

2. The "Teamwork" Analogy

Imagine you have two teams of workers (two different AI models) and a limited amount of tools (GPUs).

The Old Way: Team A grabs all the best tools because they are the "best" team. Team B is left with broken tools and cannot finish its work.
Coral's Way: Coral looks at the big picture. It realizes that Team A can actually do its job just as well with a mix of good and decent tools if they reorganize their workflow. This frees up the best tools for Team B. Both teams finish their work on time, and no tools are wasted.

Why This Matters

The study tested Coral with 6 different AI models and 20 different types of computer chips (GPUs). The results were impressive:

Cheaper: Coral reduced the cost of operating these AI models by up to 2.79 times compared to the best existing methods. This is like paying $100 for a meal and only $36 for the same quality.
Faster (with scarce resources): When the "kitchen" is overcrowded and tools are hard to find, Coral managed to serve 2.39 times more customers than the competition without slowing down.

The Bottom Line

Coral is a system that stops treating AI models as rigid, monolithic machines. Instead, it treats them like flexible recipes that can be prepared on a mix of old and new equipment. By planning the "perfect mix" in advance and then applying it quickly in real-time, it saves enormous amounts of money and keeps AI running smoothly, even when hardware supply is shaky.

In short: Coral is the intelligent manager who knows how to use a mix of expensive and cheap tools to get the job done faster and cheaper than anyone else.

Technical Summary: Coral – Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

1. Problem Statement

The landscape of Large Language Model (LLM) deployment is increasingly fragmented, with no single model dominating; instead, providers must serve multiple models simultaneously, each with distinct architectures and cost-quality trade-offs. Concurrently, cloud providers offer a heterogeneous mix of GPU generations (from high-end H100s to mid-range A100s and older L4s). While high-end hardware delivers peak performance, mid-range and older GPUs often provide comparable or superior performance per dollar, along with better availability.

The central challenge addressed by Coral is cost-efficient, adaptive serving of multiple LLMs over heterogeneous cloud resources. This requires the joint optimization of two tightly coupled decisions:

Resource Allocation: Determining which nodes to deploy across various cloud regions.
Model Placement: Deciding how to partition each model across these heterogeneous nodes (e.g., via pipeline parallelism and data parallelism).

Existing systems do not address this end-to-end. Prior work either optimizes resource allocation assuming homogeneous replicas (ignoring placement optimization) or optimizes placement for a single model on a fixed set of nodes (ignoring global allocation). A brute-force joint optimization is impractical due to the exponential search space of both dimensions and the requirement for rapid recomputation (minutes, not hours) to adapt to fluctuating demand and resource prices.

2. Methodology

Coral introduces an adaptive, heterogeneity-preserving system that decomposes the joint optimization problem into a lossless two-phase decomposition.

Core Idea: Serving Templates

Coral observes that for a fixed model and a latency Service-Level Objective (SLO), the throughput-optimal model placement on a specific set of nodes depends only on that node combination, regardless of how the rest of the cluster is allocated. This enables shifting the placement search to the offline domain.

Phase 1: Offline Generation of Serving Templates

Process: For each (model, SLO) pair, Coral enumerates a principled subset of node combinations. For each combination, it uses an Integer Linear Program (ILP) to compute the throughput-optimal model placement (layer partitioning and node assignment) that satisfies the latency SLO.
Output: A Serving Template Library. Each template is a reusable artifact specifying a node combination and its optimal placement strategy.
Optimization: The ILP maximizes throughput while adhering to latency constraints. To manage combinatorial complexity, Coral reduces the search space by limiting the maximum number of nodes per template ( $N_{max}$ ) and the total memory ratio ( $\rho$ ) relative to the model size, as larger combinations often incur prohibitive communication overhead.

Phase 2: Online Resource Allocation

Process: The online resource allocator periodically queries the precomputed Serving Template Library. It solves a second, lightweight ILP to determine which templates to instantiate in which regions to meet current throughput demands at minimal cost.
Inputs: Real-time throughput demand, GPU availability, and prices across different regions.
Goal: Minimize total deployment costs plus an initialization penalty (to avoid excessive changes), while satisfying throughput and availability constraints.
Efficiency: By shifting the complex model placement search out of the critical path, the online solver completes in seconds rather than hours, enabling the cluster to adapt to dynamic shifts in demand and supply.

System Implementation

Coral is implemented as a runtime system using ZeroMQ for control, NCCL for intra-instance communication, and vLLM as the execution engine. It supports both disaggregated and aggregated Prefill-Decode (PD) serving. A high-fidelity event-based simulator, validated against real hardware, is used for large-scale evaluation where physical deployment would be cost-prohibitive.

3. Key Contributions

The work claims the following contributions:

Problem Formulation: A formalization of heterogeneity-aware multi-LLM serving as a joint optimization over resource allocation and model placement under model-specific latency SLOs.
Lossless Decomposition: A novel two-phase approach using reusable Serving Templates to shift the complex placement search to the offline domain, reducing online solution times from hours to seconds without compromising optimality.
Scalable Solvers: ILP-based solvers for both offline placement and online allocation, scaling to clusters with thousands of nodes.
End-to-End System: A complete implementation of Coral and a validated simulator for large-scale evaluations.
Empirical Validation: Evaluation across 6 models and 20 GPU configurations, demonstrating significant cost reductions and improved goodput.

4. Results

The evaluation was conducted across diverse models (ranging from 14B to 235B parameters) and GPU configurations (including H100, A100, L40S, L4, A10G) under varying resource availability.

Cost Efficiency: Coral reduces serving costs by up to 2.79× compared to the best baseline system (Homo, which uses homogeneous replicas) and by 2.66× compared to Cauchy (a PD-disaggregated baseline system) in advanced setups. In core setups, reductions were 1.62× and 1.60×, respectively.
Goodput under Scarcity: Under scarce resource availability, Coral delivers up to 2.39× higher goodput (tokens per second meeting SLOs) compared to baselines. This is achieved through joint coordination of resources across models to resolve conflicts, whereas baselines fail to meet demand for specific models.
Solution Time: The online resource allocation ILP solves in an average of 0.24 seconds (core setup) and 9.68 seconds (advanced setup), well within the required adaptation window.
Comparison with Helix: In a single-model regime on a fixed heterogeneous cluster, Coral reduced costs by 21% and improved throughput by 26% compared to Helix, while strictly adhering to latency SLOs.
Sensitivity: Analysis shows that the cost efficiency of generated templates plateaus when the node limit ( $N_{max}$ ) is set to 6 and the memory ratio ( $\rho$ ) to 12, justifying the reduction strategy.

5. Significance

The work argues that joint optimization of resource allocation and model placement is essential for cost-efficient multi-LLM serving. By treating model replicas as black boxes on homogeneous hardware, existing systems leave significant cost savings untapped and struggle under resource constraints. Coral demonstrates that heterogeneity within replicas – mixing different GPU types within a single model replication via hybrid pipeline and data parallelism – enables systems to tailor instances to specific workload characteristics and SLOs.

The significance of Coral lies in its ability to make this joint optimization practical and adaptive. By precomputing optimal placements as reusable templates, it bridges the gap between the theoretical optimality of joint search and the practical necessity of rapid, online reconfiguration. This enables cloud providers to leverage a broader and more cost-effective spectrum of GPU inventories while maintaining strict latency guarantees for diverse, parallel LLM workloads.