💻 computer science

HADIS: A Hybrid Architecture for Query-Aware Diffusion Model Serving

HADIS is a hybrid diffusion model serving system that combines pre-generation routing and post-generation discrimination to dynamically optimize model selection and GPU allocation, significantly improving response quality and reducing SLO violations compared to existing cascade approaches.

Original authors: Qizheng Yang, Tung-I Chen, Siyu Zhao, Ramesh K. Sitaraman, Hui Guan

Published 2026-09-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qizheng Yang, Tung-I Chen, Siyu Zhao, Ramesh K. Sitaraman, Hui Guan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, computers have learned to paint. By studying millions of images and the words people use to describe them, artificial intelligence systems can now generate entirely new pictures from a simple text prompt. When a user asks for "a surreal cityscape representing time and memory," the computer does not simply retrieve a photo; it constructs an image pixel by pixel, a process that requires immense computing power and time. This capability has moved from research labs into everyday tools used by artists and designers, but it faces a stubborn logistical problem: how to serve these requests quickly and cheaply without sacrificing the beauty of the final image.

The core difficulty lies in the unpredictability of the task. Some requests are simple, like "a banana on a white plate," which the computer can solve quickly. Others are complex, requiring the machine to juggle intricate details and abstract concepts, a process that takes much longer. An approach that runs every single request through the most powerful, high-quality version of the software would guarantee a beautiful result, but it would be incredibly slow and expensive, clogging the system with unnecessary work. Conversely, using a faster, simpler version for everything would be efficient but often produce blurry or nonsensical images for difficult requests. The challenge for engineers is to build a system that can instantly recognize which requests are easy and which are hard, routing them to the right tool without wasting time or money.

Researchers at the University of Massachusetts Amherst have developed a new system called HADIS to solve this exact problem. Their work addresses a specific flaw in how current systems handle these requests. Existing methods often force every single request to pass through a fast, lightweight version of the software first, checking the result afterward to see if it is good enough. If the result is poor, the system discards it and starts over with the slow, powerful version. This approach wastes a significant amount of computing power on requests that were always going to be too difficult for the fast version to handle. It is akin to asking a junior artist to sketch a masterpiece, only to realize halfway through that the task requires a master, and then throwing away the sketch to start fresh.

To fix this, the team designed a hybrid architecture that combines two different strategies. The first part is a "router" that looks at the text prompt before any image is generated. Based on the words used, it predicts whether the request is likely to be easy or hard. If the request seems too complex, the system bypasses the fast, lightweight version entirely and sends the request directly to the powerful, high-quality model. This saves time and resources by avoiding the futile attempt to solve a hard problem with a simple tool. The second part of the system is a "discriminator" that acts as a quality inspector. If a request does go through the fast version, this inspector checks the final image. If the image is not up to standard, the system catches the mistake and reroutes it to the powerful model before the user ever sees the poor result. This two-step safety net ensures that easy requests are handled quickly, while difficult ones get the attention they need without wasting effort on failed attempts.

The researchers tested this system on a cluster of sixteen powerful graphics processors, using real-world data from thousands of image requests. They compared HADIS against several existing methods, including systems that only use a fast model, systems that only use a slow model, and systems that try to switch between them without a smart router. The results were striking. By intelligently deciding which model to use and when, HADIS improved the quality of the generated images by up to thirty-five percent compared to the other systems. At the same time, it reduced the number of requests that failed to finish on time by a factor of twenty-seven to forty-five. This means the system is not only producing better pictures but is also far more reliable in meeting the speed expectations of users.

A key insight from the study was that adding more layers of complexity to the system was unnecessary. The researchers found that a simple two-stage setup—one fast model and one slow model, managed by their hybrid router and inspector—was just as effective as more complicated arrangements with three or more models. This discovery allowed them to simplify the system's decision-making process, making it faster to adapt to changing demands. The system constantly monitors the flow of requests and adjusts its settings in real time, ensuring that it always uses the right balance of speed and quality.

The success of HADIS demonstrates that the future of serving artificial intelligence lies not just in building bigger models, but in building smarter ways to use them. By understanding the nature of the request before it is processed and verifying the result after, the system avoids the waste that plagues current technologies. This approach allows computers to serve more users with higher quality results, using the same amount of energy and hardware. As generative AI becomes more integrated into daily life, systems like HADIS will be essential for ensuring that these powerful tools remain fast, affordable, and reliable for everyone.

Technical Summary: HADIS – A Hybrid Architecture for Query-Aware Diffusion Model Serving

1. Problem Statement

Text-to-image (T2I) diffusion model serving faces a fundamental tension in balancing response quality, throughput, and latency Service-Level Objectives (SLOs). Existing query-aware cascade systems (e.g., DiffServe) route all queries through a lightweight model first, using a discriminator to reroute low-quality outputs to a heavyweight model. This approach suffers from two critical inefficiencies specific to diffusion models:

Wasted Computation on Hard Queries: All queries, including those predicted to be difficult, must pass through the lightweight stage before quality can be assessed. Since diffusion trajectories offer no reliable intermediate quality signals, hard queries inevitably require the heavyweight model, making the initial lightweight pass a sunk cost.
Inability to Reuse State: Unlike Large Language Models (LLMs) where speculative decoding allows token reuse, diffusion models lack cross-model state transfer. Architectural heterogeneity (e.g., UNet vs. DiT) and the nature of the denoising process prevent reusing intermediate states from a rejected lightweight output. A rejected query forces the heavyweight model to regenerate from scratch, incurring the full cost of both stages.

Furthermore, existing systems often fix the model pair (lightweight/heavyweight) regardless of workload dynamics. However, empirical analysis shows that the optimal model pair varies significantly with both the latency budget and the current query load, rendering static configurations suboptimal.

2. Methodology

HADIS (Hybrid Architecture for Diffusion Model Serving) addresses these challenges through a hybrid cascade architecture and joint runtime optimization.

2.1 Hybrid Cascade Architecture

HADIS combines two complementary mechanisms to minimize wasted computation and maximize quality:

Pre-Generation Router: A lightweight, rule-based prompt analyzer estimates query difficulty before generation. Queries predicted to be "hard" bypass the lightweight model entirely and are sent directly to the heavyweight model.
Post-Generation Discriminator: A CLIP-based image evaluator assesses the output of the lightweight model. If the quality is insufficient (a "false-easy" case), the query is rerouted to the heavyweight model.

This design ensures that hard queries avoid unnecessary lightweight inference, while easy queries that are misclassified by the router are caught before being returned as low-quality results.

2.2 Empirical Insight: Two-Model Sufficiency

The system relies on a key empirical observation: two-model cascades capture the Pareto frontier of deeper cascades. Profiling experiments demonstrated that adding a third stage to the cascade offers negligible quality improvement over a well-tuned two-stage cascade but introduces significant system overhead (extra inference passes, queuing, and routing decisions). Consequently, HADIS restricts its search space to pairs of models, drastically reducing the complexity of the optimization problem.

2.3 System Design and Optimization

The HADIS system consists of a Controller, a Query Router, and Workers.

Offline Profiling: The system prunes the candidate model pool and profiles all valid two-model pairs across a grid of hardness ( $\theta$ ) and quality ( $\tau$ ) thresholds. This generates a compact lookup table of Pareto-optimal configurations.
Online Resource Management: At runtime, a Resource Manager solves a Mixed-Integer Linear Program (MILP) periodically. The MILP jointly optimizes:
1. Model Pair Selection: Choosing the best lightweight/heavyweight pair from the lookup table.
2. Threshold Tuning: Adjusting the router's hardness threshold and the discriminator's quality threshold.
3. Resource Allocation: Determining the number of workers and batch sizes for each model to satisfy demand within the SLO.

The MILP uses an exponentially weighted moving average (EWMA) to estimate demand and incorporates queueing delays to ensure SLO compliance.

3. Key Contributions

HADIS System: A diffusion model serving system that integrates pre-generation routing with post-generation verification, dynamically selects model pairs at runtime, and jointly optimizes configuration via an online MILP.
Pareto Frontier Reduction: An empirical demonstration that two-model cascades are sufficient to achieve the Pareto frontier of deeper cascades, enabling tractable online optimization.
Formal Analysis: A theoretical proof showing that, under mild accuracy conditions, the hybrid architecture strictly dominates both "router-only" and "discriminator-only" alternatives at equal expected cost. The analysis defines feasibility regions in ROC space where the hybrid approach is robust to imperfect routing.
Comprehensive Evaluation: Evaluation on a 16-GPU cluster using real-world traces, demonstrating significant improvements over state-of-the-art baselines.

4. Experimental Results

Evaluated on a 16-GPU cluster (NVIDIA L40S) using real-world Azure Functions traces and ImageNet-1K prompts:

Response Quality: HADIS improved response quality (measured by Fréchet Inception Distance, FID) by up to 35% compared to baselines. On average, it reduced FID by 7–23% over systems like DiffServe, Proteus, and INFaaS.
SLO Violations: HADIS reduced SLO violations by 2.7× to 45× compared to baselines. Specifically, it outperformed DiffServe by 2.7×, Proteus by 4.2×, and INFaaS by 8×.
Ablation Studies:
- Hybrid vs. Single Component: The hybrid design outperformed both router-only and discriminator-only designs across the latency-quality spectrum.
- Dynamic Pair Selection: Systems that fixed the model pair (like DiffServe) incurred higher SLO violations and lower quality compared to HADIS's dynamic selection.
- Resource Allocation: The online MILP approach significantly outperformed cached planning strategies, which suffered from transient quality drops and violation spikes during load shifts.

5. Significance and Claims

The paper claims that HADIS resolves the fundamental inefficiency in cascade serving where hard queries are forced through lightweight stages. By combining pre-generation bypass with post-generation catching, HADIS eliminates the "worst of both worlds" scenario found in single-component systems.

The authors emphasize that the system's ability to jointly optimize the model pair, routing thresholds, and GPU allocation is critical. Fixing any single dimension (as done in prior work) leads to suboptimal performance. The paper asserts that the hybrid architecture is robust to imperfect routing accuracy, provided the discriminator is sufficiently accurate, and that the two-model cascade assumption holds empirically for the evaluated model pool, making the optimization space tractable for real-time adaptation.

The work is positioned as a necessary evolution for generative AI inference, moving beyond static variant selection or simple cascades to a system that adapts to both workload dynamics and query-specific difficulty.