🤖 machine learning

Compass: SLO-aware Query Planner for Compound AI Serving at Scale

Compass is a novel SLO-aware query planner that enables efficient, large-scale compound AI serving across diverse deployment environments by decomposing complex planning problems, utilizing selective profiling for cost-effective performance estimation, and employing bipartite matching to maximize service goodput while significantly reducing costs and planning latency.

Original authors: Banruo Liu, Wei-Yu Lin, Minghao Fang, Yihan Jiang, Fan Lai

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Banruo Liu, Wei-Yu Lin, Minghao Fang, Yihan Jiang, Fan Lai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-tech restaurant kitchen. In the past, chefs (AI models) worked alone in one big kitchen (the cloud). But now, "Compound AI" is like a complex meal that requires a whole team: a sous-chef to chop vegetables, a grill master to cook meat, a sauce expert, and a delivery driver, all working together to serve a single dish to a customer.

The problem? You have thousands of orders coming in at once. Some customers want their food in 30 seconds (live video chat), while others are okay waiting a bit longer but want it to be the cheapest option possible (batch code generation). Furthermore, your kitchen isn't just one room; you have a fancy main kitchen (Cloud), a smaller satellite kitchen nearby (Edge), and even some home cooks (Edge devices) who can help.

Compass is the new, super-smart Head Chef (Query Planner) designed to manage this chaos. Here is how it works, using simple analogies:

1. The Problem: Too Many Choices, Not Enough Time

In the old days, if a new order came in, the Head Chef would try to figure out the perfect recipe. But with Compound AI, there are millions of ways to make the dish:

Placement: Should the chopping happen in the main kitchen or the satellite one?
Configuration: Should we use a giant, expensive knife (a huge AI model) or a small, fast one?
Resources: How many cooks should we assign?

Trying to find the perfect plan for every single order takes hours. By the time the Chef figures it out, the customer has already left, or the food is cold. Existing systems were either too slow or only worked for simple, single-dish orders.

2. The Solution: Compass's "Smart Shortcuts"

Compass doesn't try to taste every single possible recipe. Instead, it uses three clever tricks to find a great recipe in seconds:

A. The "Similarity" Shortcut (Search Optimizer)

Imagine you've cooked a "Spicy Tofu" dish for a customer before. Now, a new customer orders "Spicy Tofu with a little less salt."
Instead of starting from scratch, Compass looks at its history. It says, "Hey, I know how the last Spicy Tofu turned out. I can use that knowledge to guess how this new one will taste."

The Metaphor: It's like a chef who remembers that "Recipe A" and "Recipe B" are 90% similar. They don't need to taste-test the whole thing again; they just tweak the salt. This saves massive amounts of time and money (GPU costs).

B. The "Taste-Test" Triage (SLO Profiler)

To know if a recipe is good, you usually have to cook the whole thing and taste it. But cooking a giant AI model is expensive (it costs real money to run on powerful computers).
Compass uses a smart sampling technique.

The Metaphor: Instead of eating the whole pizza to see if it's salty enough, Compass takes a tiny bite. If the first bite is clearly too salty, it stops immediately and says, "Nope, trash this recipe." If the first bite is clearly perfect, it stops and says, "Great, keep this one." It only cooks the whole pizza if it's on the fence. This saves huge amounts of money and time.

C. The "Global Seating Chart" (Multi-Query Planner)

If you plan every order individually, you might accidentally assign all the "Grill Masters" to the same table, leaving other tables empty and hungry.
Compass looks at all the orders at once. It creates a seating chart that balances the load.

The Metaphor: It's like a bouncer at a club who doesn't just let people in one by one. They look at the whole crowd and say, "Okay, we have space at Table A for the big group, and Table B for the small group, so everyone gets served without anyone waiting too long." This ensures the whole kitchen runs efficiently, not just one dish.

3. The Results: Faster, Cheaper, Better

The paper tested Compass on real-world scenarios like live video chat with AI agents, speech-to-text, and code generation. Here is what happened:

Speed: Compass found a good plan in seconds (2.4 to 5.1 times faster than before). This means users don't have to wait for their AI to "think" about how to start.
Cost: It reduced the cost of running these services by 3.8 to 4.5 times. It's like getting the same meal for a fraction of the price by using the right ingredients and the right kitchen.
Quality: It served 2.4 to 5.1 times more customers successfully without breaking the rules (SLOs).

Summary

Think of Compass as the ultimate traffic controller for a city of AI robots. Instead of letting every robot drive wherever it wants (causing traffic jams and delays), Compass looks at the whole map, remembers where it's been before, and directs traffic so that everyone gets to their destination quickly, cheaply, and safely. It turns a chaotic, expensive mess into a smooth, efficient operation.

Technical Summary: Compass: SLO-aware Query Planner for Compound AI Serving at Scale

1. Problem Statement

The emergence of Compound AI systems—which integrate multiple operators (e.g., video samplers, vision encoders, LLMs, speech renderers) into pipelines for applications like live video chat, autonomous driving, and immersive gaming—has shifted ML serving from monolithic model deployment to complex pipeline orchestration. These workloads often span diverse deployment spaces, ranging from edge devices to near-edge clusters and cloud tiers.

Optimizing these systems requires joint planning of three interdependent dimensions for every operator in a pipeline:

Placement: Where to run the operator (edge, near-edge, or cloud).
Configuration: Which model version, precision, or sampling rate to use.
Resource Allocation: How much compute (e.g., GPU share) and bandwidth to provision.

The core challenge is maximizing SLO goodput (the number of queries meeting Service Level Objectives for latency, accuracy, and cost) under resource contention. Existing approaches fail to address this at scale due to:

Combinatorial Explosion: A single query can have thousands of candidate plans due to heterogeneous configurations and placements.
Heterogeneity: Runtime environments vary (e.g., edge vs. cloud speeds), and input data distributions differ across users, causing performance variance even for identical plans.
Planning Latency: Existing planners (e.g., Vulcan, VideoStorm) take minutes to hours to find a feasible plan, rendering them infeasible for real-time, short-lived queries (e.g., 10-second speech recognition).
Local vs. Global Optimization: Single-query planners ignore cross-query resource contention, leading to suboptimal global allocations (e.g., overloading the cloud while leaving edge resources idle).

2. Methodology: The Compass System

Compass is the first SLO-aware query planner designed to handle large-scale, multi-query, multi-tier compound AI workloads. It employs a two-stage planning framework to balance efficiency and decision quality.

A. System Architecture

Compass decomposes the intractable global planning problem into:

Single-Query Search: Identifying a set of Pareto-optimal candidate plans for each individual query.
Multi-Query Scheduling: Selecting the best combination of query-plan pairs to maximize global goodput under resource constraints.

B. Core Components

1. Search Optimizer (Cost-Aware Multi-Objective Search)

To navigate the vast search space efficiently, Compass introduces CAMO (Cost-Aware Multi-Objective) search, an improvement over standard Bayesian Optimization (BO).

Decomposed Surrogates: Instead of a single objective, CAMO maintains two independent surrogate models: one for accuracy ( $f_a$ ) and one for latency ( $f_l$ ).
Utility Function: It proposes new plans by maximizing an acquisition function $U(p)$ that balances the probability of meeting SLOs against the profiling cost (latency to run the inference):
$U(p) = \text{Pr}[f_a(p) \ge A_{slo}] \cdot \text{Pr}[f_l(p) \le L_{slo}] \times \frac{1}{C(f_l(p))}$
Inter-Query Warm-Start: To accelerate search for new queries, Compass leverages latent similarities between queries. It maintains a "committee" of historical surrogate models. If a new query is similar to past ones, the committee votes on the next candidate plan. As the new query's own CAMO model matures (prediction error drops below the committee's), it automatically transitions to self-reliance.

2. SLO Profiler (Efficient Verification)

Profiling (running plans to measure accuracy/latency) is the primary bottleneck. Compass reduces this cost via:

Adaptive Profiling: Instead of fixed sampling, Compass uses Student's t-test to determine the minimum number of samples required to statistically confirm if a plan meets an SLO. If the gap between the plan's estimated accuracy and the SLO is large, it stops early; if the gap is small, it samples more.
Dependency-Shared Profiling: It caches intermediate outputs for shared pipeline segments. If a new plan only changes a downstream operator, the profiler skips re-executing the upstream operators, reusing cached results.
Resource Trimming: The search initially assumes over-provisioned resources to ensure SLO feasibility. The profiler then performs a multi-dimensional binary search to greedily trim resources (e.g., reducing GPU share) until the latency SLO is on the verge of violation, identifying the Pareto-optimal cost frontier.

3. Multi-Query Planner (Global Scheduling)

Once each query has a set of Pareto-optimal plans, Compass performs bipartite matching between the query set and their plan candidates.

Objective: Maximize SLO goodput (for resource-constrained scenarios) or minimize total deployment cost (for elastic cloud scenarios).
Algorithm: A greedy heuristic that ranks plans by their benefit-to-resource ratio (or benefit-to-cost ratio). It iteratively allocates resources to the highest-ranked plans until capacity is exhausted.
Theoretical Guarantee: The authors prove that for cloud-only setups, the greedy approach achieves at least 50% of the optimal goodput. For multi-tier cost minimization, the cost is within a 11/9 competitive ratio of the optimal, plus a small constant dependent on the number of tiers.

3. Key Contributions

First Generic SLO-Aware Planner: Compass is the first system to support compound AI serving across diverse deployment spaces (cloud-only, multi-tier, edge-cloud) while handling both continuous and ephemeral interactions.
Novel Search Mechanism: It introduces a cost-aware, multi-objective search (CAMO) that leverages plan similarities (intra- and inter-query) and adaptive profiling to navigate high-dimensional search spaces efficiently.
Global Decision Quality: By decoupling single-query search from global scheduling, Compass avoids the combinatorial explosion of joint planning while maintaining near-optimal decision quality through Pareto-optimal candidate sets and greedy matching.
Real-World Implementation: The system is implemented in ~2,800 lines of Python, is API-compatible with Kubernetes and LangChain, and supports existing ML stacks with minimal code changes.

4. Experimental Results

Evaluations were conducted on five representative compound AI applications (Live Agent Chat, Agentic Code Generation, Dense Video Captioning, Visual Tracking, Speech Recognition) across realistic multi-tier infrastructure (Cloud, Near-Cloud, Near-Edge, Edge).

Service Goodput: Compass improves SLO goodput by 2.4–5.1× compared to state-of-the-art baselines (Vulcan, VideoStorm), even in their native deployment spaces.
Cost Reduction: It reduces deployment costs by 3.8–4.5× by optimizing resource allocation and placement.
Planning Speed: Compass accelerates planning by 4.2–10.5×, identifying the first SLO-compliant plan in <5 seconds (vs. minutes for baselines) and near-optimal plans in <20 seconds.
Scalability: The system scales to thousands of concurrent queries, achieving performance close to an exhaustive-search oracle (ILP) while maintaining sub-linear scheduling time.
Responsiveness: It enables real-time adaptation to runtime dynamics (e.g., network fluctuations, lighting changes) within seconds.

5. Significance and Claims

The paper claims that Compass addresses a critical bottleneck in the evolution of AI serving: the transition from single-model inference to compound AI pipelines. The authors argue that without a planner capable of handling the combinatorial complexity of multi-operator, multi-tier, and multi-query scenarios, real-time compound AI applications (like live video agents) cannot be deployed efficiently or cost-effectively.

The significance lies in:

Feasibility: Making real-time, SLO-guaranteed compound AI serving tractable where previous approaches were too slow or inaccurate.
Efficiency: Drastically reducing the "planning tax" (time and cost spent on configuration search) which previously dominated the total serving time.
Generality: Providing a unified framework that works across heterogeneous infrastructure, from edge devices to large-scale cloud clusters, without requiring application-specific tuning.

The paper concludes that Compass enables service providers to sustain high-quality, cost-efficient compound AI services at scale, bridging the gap between theoretical pipeline optimization and practical, real-time deployment.