Serving Compound Inference Systems on Datacenter GPUs

JigsawServe is a novel serving framework that optimizes latency, accuracy, and GPU resource costs for compound inference systems by jointly selecting model variants and spatially partitioning GPUs, achieving up to 11.3x higher throughput and significantly improved resource efficiency compared to prior work.

The Gist

Imagine you run a busy, high-end restaurant kitchen. In the past, every order was simple: "Make me a burger." You had one chef for burgers, and that was it.

But today, customers are ordering "Compound Meals."

• "I want a burger, but first, check if the meat is fresh, then grill it, then slice it, then wrap it, and finally, write a poem about the burger on the napkin."

This is what Compound Inference Systems are in the world of AI. Instead of one AI model doing one job, a single request triggers a chain of different AI models (a "task graph") working together.

The problem? Your kitchen (the Datacenter GPU) is huge, but most chefs (AI models) are either too big for the counter or sit around doing nothing while waiting for ingredients. You have a limited number of chefs, but a massive amount of complex orders coming in.

Enter JIGSAWSERVE, the new system proposed in this paper. Think of it as a Master Kitchen Manager who uses a "Jigsaw Puzzle" approach to solve the chaos.

Here is how it works, broken down into simple concepts:

1. The Three Superpowers of JIGSAWSERVE

To handle these complex orders efficiently, JIGSAWSERVE uses three tricks that previous managers didn't use all at once:

• Trick A: The "Menu Flexibility" (Accuracy Scaling)

  • The Old Way: You only have one recipe for "Grilled Burger." It takes 10 minutes and tastes perfect.
  • JIGSAWSERVE: You have a menu of 5 recipes. One takes 10 minutes and is perfect. Another takes 2 minutes and is "pretty good." Another takes 30 seconds and is "okay."
  • The Magic: If the kitchen is swamped, the manager says, "For the burger wrapping step, let's use the 30-second 'okay' recipe because it doesn't ruin the whole meal." But for the "meat freshness check," they use the perfect 10-minute recipe because that's critical. They dial the quality up or down depending on how busy the kitchen is, ensuring the final meal is still good enough without wasting time.

• Trick B: The "Smart Counter" (GPU Spatial Partitioning)

  • The Old Way: You have a giant counter (a powerful GPU). You assign one chef to the whole counter. If that chef is small, 80% of the counter sits empty. If the chef is huge, they can't fit.
  • JIGSAWSERVE: Imagine the counter is made of Lego bricks. The manager can slice the counter into tiny, isolated islands. One small island for a tiny chef, a medium island for a medium chef.
  • The Magic: Now, instead of one chef hogging the whole counter, you can fit four different chefs on one counter, working on four different parts of the order at the same time, without bumping into each other. This is called Spatial Partitioning.

• Trick C: The "Map of the Order" (Task-Graph-Informed Budgeting)

  • The Old Way: The manager gives every step of the order the same amount of time and resources. "Everyone gets 5 minutes!"
  • JIGSAWSERVE: The manager looks at the whole order map. "Wait, the 'poem writing' step is fast and doesn't need a fancy chef. But the 'grilling' step is the bottleneck. Let's give the grilling step 3 chefs and the poem step 1 tiny chef."
  • The Magic: They allocate resources based on the flow of the order, not just a flat rule.

2. The "Jigsaw" Analogy

Why is it called JIGSAWSERVE?

Imagine you have a giant puzzle (the total computing power of the datacenter).

• Old systems tried to force big, square puzzle pieces (whole GPUs) into the puzzle. They didn't fit well, leaving huge gaps of wasted space.
• JIGSAWSERVE realizes that the puzzle pieces (AI tasks) come in all different shapes and sizes. Some are tiny, some are huge.
• The system acts like a puzzle solver that picks the perfectly shaped pieces (model variants) and cuts the big puzzle board (GPUs) into custom shapes (spatial partitions) so that every single inch of the board is filled perfectly with no gaps.

3. The Results: A Miracle in Efficiency

The researchers tested this in a real datacenter (a kitchen with 4 super-computer GPUs).

• The Result: JIGSAWSERVE could handle 11.3 times more orders than the best previous system.
• The Cost: It used only 43% of the available GPU power to meet the same goals.
• The Quality: The food (accuracy) was still delicious (meeting accuracy goals), and the wait times (latency) were almost never late.

4. Why This Matters

Before this, if you wanted to run complex AI (like an AR assistant that sees an object, describes it, and speaks to you), you needed a massive, expensive server farm, and it was often inefficient.

JIGSAWSERVE proves that by being smart about how we slice the hardware and flexible about the quality of the AI models, we can:

1. Save massive amounts of money and electricity.
2. Serve more users with the same equipment.
3. Make complex AI applications (like self-driving cars or VR assistants) faster and more reliable.

In short: JIGSAWSERVE is the ultimate kitchen manager that knows exactly which chef to hire, how to cut the counter, and when to serve a "good enough" meal versus a "perfect" one, ensuring the restaurant runs at maximum efficiency without ever serving a bad meal.

Technical Summary

Here is a detailed technical summary of the paper "Serving Compound Inference Systems on Datacenter GPUs" by Sriram Devata, Rahul Singh, and Sarita Adve.

1. Problem Statement

Emerging applications in domains like Extended Reality (XR) and multi-agent systems are shifting from single-model inference to Compound Inference Systems. In these systems, a single user request triggers a Directed Acyclic Graph (DAG) of multiple ML models, where each model performs a specific task.

Serving these systems efficiently on datacenter GPUs presents two unique challenges not fully addressed by prior work:

1. End-to-End Budget Apportionment: Latency and accuracy Service Level Objectives (SLOs) are defined for the entire system, not individual tasks. The system must dynamically decide how to split these budgets among tasks, allowing for the selection of different model variants (trade-offs between speed and accuracy) for each task.
2. Heterogeneous Resource Allocation: Different tasks in the graph have varying resource requirements. Traditional systems often allocate whole GPUs to tasks, leading to underutilization. Modern GPUs (e.g., NVIDIA H100) offer spatial partitioning mechanisms (MIG and MPS) to slice GPUs into smaller, isolated instances, but existing compound inference frameworks do not leverage this granularity effectively.

2. Methodology: JIGSAWSERVE

The authors propose JIGSAWSERVE, a serving framework that jointly optimizes for latency, accuracy, and cost (GPU resources) by adaptively choosing model variants and performing fine-grained spatial partitioning.

Core Components

• Registration Handler: Accepts the compound inference system definition, including the task graph, available model variants (with their accuracy/latency profiles), and end-to-end SLOs.
• Profiler: Performs offline profiling to determine the 95th-percentile latency and throughput for all combinations of model variants, GPU segment types (MIG instances with MPS concurrency), and batch sizes.
• Controller (The Optimizer):
  • Formulates the configuration selection problem as a Mixed Integer Linear Programming (MILP) problem.
  • Decision Variables: Selects the number of instances for each model variant, the specific GPU segment type (MIG size + MPS concurrency), and the batch size for every task.
  • Constraints:
    • Latency: Ensures the worst-case path latency (including queuing) meets the end-to-end SLO.
    • Throughput: Calculates demand propagation through the DAG using multiplicative factors (e.g., one object detection triggering multiple re-identifications) and ensures total throughput meets demand.
    • Resources: Limits total GPU slices used to the available hardware capacity.
    • Accuracy: Uses a Pipeline Accuracy Score (PAS) to ensure the weighted average accuracy of the path meets the global SLO.
  • Objective: Maximize overall system accuracy while minimizing total GPU slices used.
• Frontend: Handles client requests, tracks demand, and routes requests to the first task. It triggers the controller to re-optimize when demand patterns shift.
• Runtime Mechanisms: Implements batching (waiting for max batch size or timeout) and early dropping (dropping stale requests that cannot meet SLOs) to manage tail latency.

Key Innovations

JIGSAWSERVE is the first system to combine three critical features:

1. Per-task Accuracy Scaling: Choosing different model variants (e.g., ResNet-18 vs. ResNet-152) for different tasks based on their impact on the global accuracy budget.
2. GPU Spatial Partitioning: Utilizing NVIDIA MIG to create isolated slices and MPS to run multiple processes within a slice, maximizing hardware utilization.
3. Task-Graph-Informed Budgeting: Using the DAG structure to propagate demand and allocate resources, rather than treating tasks in isolation.

3. Key Contributions

1. Novel Framework: Introduction of JIGSAWSERVE, which unifies accuracy scaling, spatial partitioning, and task-graph-aware resource budgeting.
2. Analytical Evaluation: A comprehensive comparison of configuration search spaces. The authors demonstrate that spatial partitioning (S) provides the highest standalone gain (5.25× capacity over unoptimized), but the full integration of Accuracy + Spatial + Task-graph (A+S+T) yields the best results (21.6× capacity over unoptimized).
3. Empirical Validation: Extensive testing on a 4-GPU H100 cluster with three real-world workloads (Social Media, Traffic Analysis, AR Assistant).
4. Significance of Joint Optimization: The paper proves that sacrificing any single feature (e.g., removing spatial partitioning or task-graph awareness) significantly degrades efficiency.

4. Results

The evaluation was conducted on a Dell PowerEdge server with four NVIDIA H100 GPUs, simulating traffic patterns from a Twitter trace.

• Maximum Serviceable Demand:

  • JIGSAWSERVE (A+S+T) achieves 21.6× higher maximum demand compared to an unoptimized system.
  • Compared to the closest prior work (Loki, equivalent to A+T), JIGSAWSERVE serves 11.3× more requests.
  • Spatial partitioning (S) alone was the most impactful single feature, providing a 5.25× gain.

• Resource Efficiency & SLO Compliance:

  • GPU Utilization: JIGSAWSERVE consumes only 43.3% of available GPU resources on average while meeting SLOs.
  • SLO Violations: The system maintains an average latency SLO violation rate of <0.6%.
  • Comparison: Baselines like S+T and A+T suffered >10% SLO violations and required >2× the resources of JIGSAWSERVE in high-demand scenarios. A+S (without task-graph awareness) required 33% more resources and had 6.7% SLO violations.

• Configuration Insights:

  • The system dynamically selects model variants based on task criticality (e.g., using a high-accuracy YOLO variant for AR but a lower-accuracy one for traffic analysis if the downstream task is less sensitive).
  • It optimally selects MIG instance sizes (e.g., 1/7 MIG) and MPS concurrency levels to minimize fragmentation.

5. Significance

This work fundamentally shifts the paradigm for serving compound AI workloads in datacenters:

• Efficiency: It demonstrates that treating compound inference as a holistic optimization problem, rather than a collection of independent models, drastically reduces hardware costs and increases throughput.
• Hardware Utilization: It validates that modern GPU spatial partitioning (MIG/MPS) is essential for datacenter efficiency, moving away from the "one GPU per model" paradigm.
• Future Directions: The authors argue that the ML community should continue releasing multiple model variants per architecture to enable this flexibility, and GPU vendors must prioritize spatial partitioning mechanisms. The framework is also adaptable to other vendors (e.g., AMD) and future work could explore carbon-intensity optimization and dynamic request rerouting.

In summary, JIGSAWSERVE proves that joint optimization of model selection, fine-grained hardware partitioning, and graph-aware resource budgeting is critical for the scalable and cost-effective deployment of complex AI applications.