BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs

BrownoutServe is a novel serving framework for Mixture-of-Experts (MoE) based large language models that enhances inference efficiency and maintains service level objectives under bursty workloads by introducing "united experts" to reduce access latency and employing a dynamic brownout mechanism for adaptive token processing.

The Gist

Imagine you run a massive, high-end custom furniture workshop (this is your Large Language Model, or LLM).

In a traditional workshop, every time a customer orders a chair, you have a team of 60 specialized carpenters (called Experts). Even if the customer just needs a simple leg, you might call in 2 or 3 different carpenters to make sure it's perfect. This is the Mixture-of-Experts (MoE) architecture. It's great because you don't need to pay all 60 carpenters for every single chair; you only pay the few you use.

However, you have a problem: The Rush Hour.

The Problem: The "Bursty" Traffic Jam

Imagine it's 9:00 AM. Suddenly, 500 customers walk in at once, all wanting custom tables.

1. The Bottleneck: Your 60 carpenters are now overwhelmed. Some are super busy (the "Hot" experts), while others are standing around doing nothing (the "Cold" experts) because the work isn't distributed evenly.
2. The Chaos: Because everyone is trying to grab tools and talk to each other, the workshop gets clogged. The time it takes to finish a chair (latency) skyrockets.
3. The Promise Broken: You promised customers they'd get their chairs in 10 minutes (your SLO or Service Level Objective). But because of the rush, it's taking 30 minutes. You are failing your customers.

Existing systems try to solve this by just adding more carpenters (buying more GPUs), but that's expensive, slow to set up, and often too late for the rush.

The Solution: BrownoutServe

The paper introduces a new system called BrownoutServe. It uses two clever tricks to keep the workshop running smoothly during the rush, inspired by how power companies handle blackouts.

Trick 1: The "Super-Carpenter" (United Experts)

Instead of having 60 individual carpenters, imagine you train a few "Super-Carpenters."

• Each Super-Carpenter is a master who has studied the techniques of 4 or 8 regular carpenters combined into one person.
• Why does this help? Instead of calling 4 different people to do a job, you just call one Super-Carpenter. They can do the work of all four at once.
• The Result: You reduce the number of people you have to coordinate. The workshop moves faster because there's less "talking" and more "doing."

Trick 2: The "Brownout" Strategy (Smart Sacrifice)

This is the coolest part. In power systems, a "brownout" is when the grid is overloaded, so they dim the lights in non-essential areas to keep the hospital running.

BrownoutServe does the same thing for your furniture orders:

• The Scenario: The rush is too big. You can't finish every detail perfectly in time.
• The Choice: The system looks at the incoming orders. For the most critical parts of the chair, it uses the original, perfect carpenters. But for the less critical parts (or if the rush is insane), it routes some work to the Super-Carpenters.
• The Trade-off: The Super-Carpenter is 95% as good as the original team, but they are twice as fast.
• The Magic: The system has a smart manager (the SLO-Aware Controller) that watches the clock.
  • If the line is moving too slow: It immediately switches more work to the Super-Carpenters to speed things up.
  • If the line is moving too fast: It switches back to the original experts to ensure maximum quality.

It's like a traffic cop who dynamically changes the speed limit. If traffic is gridlocked, they lower the limit to keep cars moving. If traffic is light, they raise it to let people drive faster.

The Results

The authors tested this in a real "workshop" (using powerful computer chips called GPUs):

• Speed: They processed 2x more orders per hour than the standard system (vLLM).
• Reliability: They reduced the number of customers who waited too long (SLO violations) by 90%.
• Quality: The furniture was still 95% as good as before. The tiny drop in perfection was a small price to pay to avoid a total system crash.

In a Nutshell

BrownoutServe is a smart manager for AI models that knows when to "cut corners" just enough to keep the line moving. It combines the knowledge of many experts into "Super-Experts" and dynamically decides which orders get the VIP treatment and which get the "Super-Expert" treatment, ensuring that even during the craziest rushes, the system doesn't break, and customers don't wait forever.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) utilizing the Mixture-of-Experts (MoE) architecture offer significant advantages in reducing memory footprint and training costs by activating only a subset of parameters (experts) per token. However, existing MoE inference systems (e.g., vLLM, DeepSpeed-MoE) face critical challenges in real-world deployment, particularly under bursty workloads:

• Expert Load Imbalance: Token routing often results in "hot" experts handling massive loads while "cold" experts remain underutilized, leading to poor GPU parallelism and resource waste.
• Static Configuration: Current systems rely on static model placement and lack dynamic adaptation to sudden traffic spikes.
• SLO Violations: Under bursty traffic, latency spikes cause frequent violations of Service Level Objectives (SLOs). Traditional horizontal scaling (spawning new instances) is too slow (30–90s cold starts) and memory-intensive to handle short-lived traffic bursts effectively.
• Bottleneck Dominance: The MoE module accounts for the majority of inference latency (approx. 81% in prefill and 94% in decoding), making it the primary target for optimization.

2. Methodology: BrownoutServe

The authors propose BrownoutServe, a novel serving framework designed for small-to-medium GPU clusters. It introduces two core mechanisms to optimize efficiency and maintain SLOs:

A. United Experts Model

To address load imbalance and reduce access overhead, the authors introduce "United Experts."

• Concept: Multiple original experts are grouped, and their knowledge is distilled into a single "united expert" of equivalent parameter size.
• Mechanism: During training, a united expert acts as a student model, learning from the hidden states of its group of original experts (teacher models) using Mean Squared Error (MSE) loss.
• Benefit: This reduces the number of distinct experts accessed during inference, increases the batch size processed by each active expert, and fully leverages GPU parallelism, thereby reducing latency.

B. Dynamic Brownout Mechanism

Inspired by power grid "brownout" strategies (temporarily reducing non-essential load to prevent total failure), the system dynamically degrades the processing of certain tokens during high load.

• Three Strategies:
  1. Zero-Brownout: All tokens processed by original experts (highest accuracy, highest latency).
  2. Full-Brownout: A threshold determines a subset of tokens processed by original experts; the rest are ignored (lowest latency, significant accuracy loss).
  3. Partial-Brownout (Proposed): A hybrid approach. Tokens not processed by original experts are routed to their corresponding United Experts rather than being ignored. This balances latency reduction with accuracy preservation.
• Routing Logic: Tokens are sorted by load. High-load tokens are assigned to original experts ($S_1$), while lower-priority tokens are routed to united experts ($S_2$).

C. SLO-Aware Latency Control (SALC) Algorithm

To dynamically manage the trade-off between accuracy and latency, BrownoutServe employs the SALC algorithm.

• Function: It continuously monitors real-time inference latency (P90) and adjusts the brownout threshold (the proportion of tokens sent to original experts).
• Logic:
  • If latency exceeds the SLO, the threshold is multiplicatively shrunk (rout more tokens to united experts).
  • If latency is well below the SLO (below a "warning line"), the threshold is linearly increased (rout more tokens to original experts to recover accuracy).
• Goal: Maximize prediction accuracy while strictly ensuring latency remains within the SLO bounds.

3. Key Contributions

1. Unified Framework: Development of BrownoutServe, an end-to-end inference framework integrating United Experts and dynamic brownout mechanisms specifically for MoE-based LLMs.
2. Novel Concepts: Introduction of "United Experts" to consolidate knowledge and the "Brownout Approach" to dynamically trade accuracy for latency, including three distinct strategies (Zero, Full, Partial).
3. Adaptive Control: Design of the SLO-Aware Latency Control (SALC) algorithm, which automatically tunes the brownout configuration in real-time without manual intervention.
4. Implementation: A practical implementation (approx. 5.5k lines of Python) utilizing Triton for optimized kernels (PagedAttention, FlashAttention) and integration with existing optimizations like Continuous Batching.

4. Experimental Results

The system was evaluated using Qwen1.5-MoE-A2.7B-Chat on a 4x NVIDIA A100-40GB GPU cluster, comparing against vLLM (both native and non-fused versions) under ShareGPT and Alpaca workloads.

• Throughput Improvement:
  • Compared to vLLM (non-fused), BrownoutServe achieved up to 2.07× higher throughput on ShareGPT and 2.06× on Alpaca.
  • Compared to vLLM (native/fused), it achieved 1.32× improvement on Alpaca.
• SLO Violation Reduction:
  • Under bursty traffic (2x load surge), BrownoutServe reduced SLO violations by 90.28% compared to vLLM on the ShareGPT dataset.
  • On Alpaca, violations were reduced by 89.72%.
• Accuracy Trade-off:
  • Using optimal configurations (e.g., 8-way united experts with a 0.4 threshold), the system incurred an average accuracy loss of only ~5% while maintaining strict SLO compliance.
  • The SALC algorithm successfully kept latency oscillations low (67% reduction in variance compared to baseline).

5. Significance

BrownoutServe addresses a critical gap in the deployment of MoE-based LLMs: the inability to handle bursty, unpredictable traffic without massive resource over-provisioning or severe latency spikes.

• Cost Efficiency: By enabling small-to-medium GPU clusters to handle traffic bursts through intelligent resource reallocation (rather than spawning new instances), it significantly lowers infrastructure costs.
• Reliability: It guarantees SLOs even during traffic spikes, which is crucial for latency-sensitive applications like chatbots and real-time recommendation systems.
• Scalability: The framework is orthogonal to existing optimizations (like PagedAttention) and can be integrated into current serving stacks, offering a practical path to scalable, reliable MoE inference.

In summary, BrownoutServe transforms the MoE inference paradigm from a static, resource-heavy process into a dynamic, SLO-aware system that intelligently sacrifices marginal accuracy to ensure service reliability during peak loads.