Technical Summary: DiLaServe

Problem Statement

Diffusion Language Models (DLMs) have emerged as a promising alternative to autoregressive language models (LLMs), offering the ability to generate multiple tokens in parallel during each denoising step. This parallelism theoretically enables significantly higher inference throughput compared to the sequential token generation of autoregressive models. However, deploying DLMs in production serving systems presents unique challenges that prevent the realization of these throughput gains while meeting strict latency Service Level Objectives (SLOs).

The primary challenges identified are:

1. Speed-Quality Trade-off: DLMs use a confidence-based denoising process where tokens are unmasked only if their predicted confidence exceeds a threshold. Lowering this threshold reduces the number of denoising steps (improving speed) but risks unmasking incorrect tokens (degrading quality). Existing systems lack mechanisms to dynamically adjust this threshold per request to balance SLO adherence with generation quality.
2. Cluster-Level Latency-Throughput Trade-off: DLMs are computationally intensive per step and benefit from Tensor Parallelism (TP). However, increasing TP degree reduces per-step latency but reduces the total number of model instances a cluster can support, thereby lowering aggregate throughput. Static cluster configurations fail to adapt to fluctuating workloads, leading to either excessive queueing delays under high load or suboptimal latency under low load.
3. Approximate KV Caching Heterogeneity: Unlike autoregressive models, DLMs cannot use exact KV caching due to bidirectional attention. Recent approximate KV caching techniques introduce non-uniform computational costs: "cache steps" (reusing KV states) are cheap, while "recompute steps" (refreshing KV states) are expensive. Serving systems must account for this step-level heterogeneity to accurately estimate latency and load.

Methodology

The authors present DiLaServe, a cluster-level serving system designed specifically for DLMs. DiLaServe integrates three core mechanisms to address the challenges above:

1. Dynamic Confidence Threshold Adjustment

DiLaServe treats the confidence threshold as a control knob for each request.

• SLO-Aware Selection: For each request, the system predicts the remaining number of denoising steps required under different candidate thresholds. It selects the highest confidence threshold that ensures the estimated remaining latency fits within the request's remaining SLO budget.
• Adaptive Load Control: To prevent the aggregate load from exceeding cluster capacity (which would cause queueing delays and SLO violations), the system limits the maximum confidence threshold available to each request. It calculates the worst-case aggregate load assuming all requests use their maximum allowed threshold and ensures this sum remains within the cluster's processing capacity.
• Step Prediction: A lightweight, offline-trained gradient boosting model predicts the remaining denoising steps based on runtime features (unmasking progress, token confidence statistics, and output length). These predictions are updated online as the request progresses, significantly reducing prediction error compared to one-shot estimates.

2. Fine-Grained Scheduling

DiLaServe schedules requests at the granularity of individual denoising steps.

• Instance Selection: The scheduler assigns the next denoising step to the model instance with the lowest predicted step completion time, considering both the instance's current load and its Tensor Parallelism degree.
• Migration: Requests can be migrated between instances between steps. Since denoising steps are stateless without KV caching (or only require KV state at recompute steps), migration is lightweight. This enables fine-grained load balancing and reduces tail latency.

3. Cluster Reconfiguration via Integer Linear Programming (ILP)

To adapt to changing workload patterns, DiLaServe periodically solves a two-stage ILP:

• Stage 1 (Capacity Planning): Determines the optimal number of model instances at each TP degree to maximize the average confidence threshold (and thus quality) while ensuring steady-state capacity sufficiency and SLO satisfaction.
• Stage 2 (Reconfiguration Planning): Computes a migration plan to transition from the current cluster configuration to the target configuration with minimal overhead, prioritizing the preservation of existing instances to minimize service disruption.

4. Support for Approximate KV Caching

DiLaServe explicitly models the heterogeneity introduced by approximate KV caching. It distinguishes between "cache steps" and "recompute steps" when estimating request load, cluster capacity, and remaining latency. Crucially, it restricts request migration to occur only at recompute steps to avoid the overhead of rebuilding KV caches on the destination instance.

Key Contributions

• DiLaServe System: The first cluster-level serving system designed specifically for Diffusion Language Models, addressing the unique speed-quality and latency-throughput trade-offs inherent to the architecture.
• Joint Optimization: A novel approach that jointly optimizes per-request confidence thresholds, cluster-wide load control, and dynamic cluster reconfiguration (TP degree allocation).
• Approximate KV Caching Integration: A framework that incorporates the non-uniform costs of approximate KV caching into scheduling and reconfiguration decisions, enabling efficient serving without sacrificing the benefits of caching.
• Step Prediction: An online step prediction mechanism that leverages runtime features to accurately estimate remaining work, enabling precise SLO-aware threshold selection.

Results

The authors evaluated DiLaServe on 16-GPU clusters using real-world LLM serving traces (derived from ShareGPT and BurstGPT) and standard accuracy benchmarks (GSM8K, MMLU-Pro, MBPP). The system was tested against baselines adapted from existing serving systems (INFaaS and Llumnix) that use fixed confidence thresholds.

• SLO Attainment: DiLaServe improved SLO attainment by up to 56.6 percentage points compared to baselines under high load. In real-world trace evaluations, it achieved a 30.2 percentage point improvement.
• Latency: End-to-end request latency was reduced by up to 46%.
• Quality: These improvements were achieved with minimal accuracy degradation, typically less than 1% (e.g., a 0.9% drop in accuracy under high load, and a 0.09-point drop in LLM-judge scores on conversational data).
• Ablation Studies: The authors demonstrated that both the dynamic threshold adjustment and the system-wide load control are critical; removing load control reduced SLO attainment by up to 42.4 percentage points in high-load scenarios. Similarly, reconfiguration significantly improved performance across varying load levels.

Significance

The paper claims that DiLaServe demonstrates the feasibility of serving Diffusion Language Models at scale while meeting strict latency requirements. By explicitly modeling the unique characteristics of DLMs—specifically the confidence-based denoising process and approximate KV caching—DiLaServe unlocks the throughput potential of these models without sacrificing generation quality or SLO adherence. The work suggests that dynamic, quality-aware control mechanisms are essential for the next generation of language model serving systems, moving beyond static configurations to adaptively balance speed, quality, and resource utilization.