Peformance Isolation for Inference Processes in Edge GPU Systems

This paper evaluates NVIDIA's GPU isolation mechanisms (MPS, MIG, and Green Contexts) on A100 and Jetson Orin platforms to determine their effectiveness in ensuring predictable inference times for safety-critical edge applications, finding that while MIG offers high isolation, Green Contexts provide a promising low-overhead alternative for edge devices despite lacking memory isolation.

The Gist

Imagine you have a super-fast, high-powered chef (the GPU) in a kitchen. This chef is incredibly talented at cooking complex dishes (running Deep Learning models) to help make important decisions, like helping a self-driving car see a pedestrian or a medical device diagnose an illness.

In the world of "safety-critical" applications, it's not enough for the chef to just be good at cooking; they also need to be predictable. If the chef takes 2 seconds to chop an onion today, they must take exactly 2 seconds tomorrow. If they get distracted by another order and take 5 seconds, that could be dangerous.

The problem arises when you try to make the chef cook multiple dishes at the same time to save money and space. If two orders come in, they might fight over the same cutting board, the same stove, or the chef's attention, causing delays.

This paper tests three different ways to organize this kitchen so that every dish gets cooked on time, even when the chef is busy with multiple orders. They tested these methods on two types of kitchens: a massive, industrial kitchen (NVIDIA A100) and a compact, energy-efficient food truck (Jetson Orin).

Here are the three "kitchen management strategies" they tested:

1. MPS (Multi-Process Service): The "Shared Apron" Approach

• The Analogy: Imagine the chef wears a special apron that allows them to switch between two recipes instantly without taking off their hat or washing their hands. It's a software trick to make the chef feel like they are doing two things at once with very little "switching cost."
• The Result: This works well for making the kitchen faster overall. However, it's like two people sharing a single pair of scissors. If one person grabs them, the other has to wait. It doesn't guarantee that the second person will finish on time if the first person gets too busy. It's good for general speed, but not safe enough for critical emergencies.

2. MIG (Multi-Instance GPU): The "Walled Garden" Approach

• The Analogy: Imagine the chef's kitchen is physically divided by walls into two separate rooms. Each room has its own stove, its own cutting board, and its own chef (or a dedicated portion of the main chef's time). One room cannot touch the other's ingredients or tools.
• The Result: This is the gold standard for safety. Because the rooms are physically separated, if the chef in Room A gets overwhelmed, the chef in Room B doesn't care. They get their own guaranteed time and space.
• The Catch: The walls are rigid. Once you build the wall, you can't easily move it. If one room needs more space and the other needs less, you can't just shift the wall; you have to rebuild it. Also, this "wall" technology is only available in the big industrial kitchens (A100), not the small food trucks.

3. Green Contexts (GC): The "Time-Boxed" Approach

• The Analogy: This is a newer, clever software trick for the small food trucks. Instead of building walls, the manager gives the chef a specific list of "tools" (specific parts of the stove) that they are only allowed to use for a specific dish. The chef knows, "I can only use burners 1 and 2 for this order."
• The Result: This is very flexible and lightweight. It doesn't slow the chef down much.
  • On the Big Kitchen (A100): It works great, acting like a soft version of the walled garden.
  • On the Food Truck (Jetson Orin Nano): It hit a snag. Because the food truck is small and has a strict power limit (like a small generator), when the chef tried to cook two dishes at once, the whole truck started to "overheat" and the power generator slowed down. Even though the chef was told to only use specific burners, the whole truck slowed down because it was running out of power.
  • The Fix: When they tested this on a slightly bigger food truck (Jetson Orin AGX) with a stronger generator, the "Time-Boxed" approach worked perfectly, isolating the dishes just as well as the walled garden.

The Big Takeaways

1. Big Models vs. Small Models: Some dishes (large AI models) are so heavy that they use the whole stove anyway. They are hard to slow down. Other dishes (small models) are light and rely heavily on the speed of the delivery service (memory). If the delivery is slow, these small dishes get delayed easily.
2. The Power Problem: On small, portable devices, power is the biggest enemy. If you try to run two tasks at once, the device might run out of juice, causing the whole system to slow down, breaking the "isolation" promise.
3. The Future: The paper suggests that while "Walled Gardens" (MIG) are the safest, they are too rigid. The "Time-Boxed" approach (Green Contexts) is the future for small devices, but we need to invent a way to give them their own "memory" (like a private pantry) so they don't fight over the delivery service.

In short: If you need absolute safety and have a big machine, build walls (MIG). If you are on a small, battery-powered device, you can use the "Time-Boxed" method (Green Contexts), but you have to be careful not to ask the device to do too much at once, or it will run out of power and slow everything down.

Technical Summary

Technical Summary: Performance Isolation for Inference Processes in Edge GPU Systems

Problem Statement
The increasing adoption of Deep Learning Models (DLMs) in safety-critical domains (e.g., automotive, aerospace) necessitates strict adherence to functional safety standards like ISO 26262. While ensemble strategies—running multiple diverse models to vote on a final prediction—enhance accuracy and robustness, they introduce significant challenges regarding temporal predictability. In high-criticality tasks, a model must not only produce a correct prediction but must do so within guaranteed time constraints. This becomes particularly difficult when multiple inference processes run concurrently on a single Graphics Processing Unit (GPU), where resource contention can lead to unpredictable latency and timing violations. The paper addresses the need to analyze modern GPU isolation mechanisms to determine their efficacy in ensuring predictable inference times for such safety-related applications.

Methodology
The study evaluates three primary isolation mechanisms provided by NVIDIA:

1. Multi-Process Service (MPS): A software-based solution unifying CUDA contexts to reduce context-switching overhead.
2. Multi-Instance GPU (MIG): A hardware-based solution creating physically isolated partitions of compute units (GPCs) and memory.
3. Green Contexts (GC): A newer software-based solution (CUDA 12.4+) allowing fine-grained allocation of Streaming Multiprocessors (SMs) without memory isolation.

The experimental methodology involved two distinct hardware platforms:

• General-Purpose GPU: NVIDIA A100 (40 GB memory), used to test MPS and MIG.
• Edge Systems: Jetson Orin Nano and Jetson Orin AGX (SoC platforms), used to test MPS and GC.

The evaluation utilized six neural network architectures of varying complexity (ConvNeXt Base/Large, MobileNetV2, ResNet18, ViT B 16, ViT L 32), optimized for the specific hardware (TensorRT for Jetson). The methodology included:

• Partitioning Impact Analysis: Measuring throughput, memory usage, and power consumption as resources (GPCs or SMs) were incrementally allocated.
• Maximum Inference Frequency (IMS) Search: An iterative algorithm (Algorithm 1) to determine the maximum stable inference frequency for each model under isolated conditions.
• Isolation Benchmarking: A two-process experiment where a "process of interest" runs at its maximum stable IMS while a second "interfering" process gradually increases its load. The study quantified isolation by measuring the occurrence of timeouts (inferences failing to complete within their allocated time) as contention increased.

Key Results

• MIG (A100): Provided the highest level of temporal isolation. For large, compute-bound models (e.g., ConvNeXt Large), MIG maintained performance even with reduced resources. For smaller, memory-bound models, MIG showed some overhead but generally maintained near-zero timeouts until extreme contention. However, MIG partitions are rigid; once created, they cannot be dynamically resized.
• MPS (A100 & Jetson): Offered moderate improvements over standalone GPU usage by reducing context-switching costs, particularly benefiting larger models. However, it failed to provide true temporal isolation; as the interfering process load increased, the process of interest experienced significant timeouts, as MPS does not isolate memory bandwidth or SM scheduling.
• Green Contexts (Jetson Orin Nano): While GC allowed fine-grained SM allocation with negligible throughput overhead, it failed to provide robust isolation on the Orin Nano. The study identified that the device's power constraints (max ~20W) caused the GPU frequency to drop drastically when two processes ran in parallel, leading to performance degradation and timeouts regardless of the isolation mechanism. This suggests that on power-constrained edge devices, hardware limitations (thermal/power throttling) can override software partitioning benefits.
• Green Contexts (Jetson Orin AGX): When tested on the Orin AGX (50W power limit, higher frequency headroom), GC performance improved significantly. Without the power-induced frequency throttling seen on the Nano, GC achieved isolation levels approaching those of MIG on the A100, demonstrating that GC is a viable alternative for edge devices provided power constraints are managed.
• Model Sensitivity: Compute-bound models (ConvNeXt, ViT) showed greater resilience to resource reduction, while memory-bound models (ResNet18, MobileNetV2) were more sensitive to contention, especially in environments lacking memory isolation.

Significance and Claims
The paper claims that while no current technology offers perfect temporal predictability across all scenarios, distinct trade-offs exist:

• MIG remains the most robust mechanism for safety-critical applications due to its hardware-level isolation of both compute and memory, though its lack of dynamic flexibility limits its adaptability to variable workloads.
• Green Contexts emerge as a promising, low-overhead alternative for edge devices, offering fine-grained control. The study highlights that their effectiveness is contingent on the underlying hardware's ability to sustain power and frequency requirements under load.
• MPS is deemed unsuitable for high-criticality systems requiring strict isolation but remains a practical tool for general-purpose workloads seeking efficiency.

The authors conclude that future work must focus on developing memory isolation at the context level for Green Contexts and creating algorithms for dynamic resource allocation that can maintain strict isolation while adapting to changing workload requirements. This research positions Green Contexts as a potential primary alternative for parallelizing processes on edge GPUs, provided hardware limitations are addressed.