🤖 AI

HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization

HeraSys is an LLM serving system that optimizes the end-to-end performance of concurrent multi-tenant workflows by eliminating cross-workflow computational redundancy through structural node merging and employing a load-aware joint scheduling policy to significantly reduce tail latency while increasing throughput.

Original authors: Size Li, Zhiqing Tang, Hongrui Liang, Jianxiong Guo, Jiong Lou, Tian Wang, Weijia Jia

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Size Li, Zhiqing Tang, Hongrui Liang, Jianxiong Guo, Jiong Lou, Tian Wang, Weijia Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling kitchen where a team of chefs is trying to cook hundreds of complex meals at once. In the world of Artificial Intelligence, these "meals" are tasks performed by Large Language Models (LLMs)—the super-smart computer brains that write stories, answer questions, and solve problems. Usually, when you ask an AI to do something, it's treated like a single, isolated order. But in the real world, applications are becoming more like elaborate banquets: a single request might involve searching for information, checking a database, and then writing a summary, all in one go. These are called "workflows."

The problem is that when many people order these complex banquets at the same time, the kitchen gets chaotic. If two customers ask for the same ingredient to be chopped, the kitchen might chop it twice, wasting time and energy. Worse, if one customer orders a massive, slow-cooking stew, the chefs might get stuck on it, leaving everyone else waiting for their quick salads to be served. This is the challenge of "serving" AI: how do you manage a flood of complex, overlapping requests without the system grinding to a halt or leaving some people waiting forever?

This is where a new system called HeraSys steps in. Think of HeraSys as a revolutionary kitchen manager who doesn't just look at one order at a time, but watches the entire kitchen floor to see how different orders can help each other. Instead of treating every request as a separate island, HeraSys looks for overlaps. If two different customers need the same document analyzed or the same tool used, HeraSys says, "Hey, let's do that once and share the result!" It also acts like a traffic cop, ensuring that the quick, simple orders don't get stuck behind the slow, heavy ones, and that the slow ones don't get starved of attention.

The researchers behind HeraSys built a system that breaks down these complex AI tasks into tiny, fine-grained pieces. They found that by merging duplicate steps and smartly scheduling who gets the kitchen's resources (like powerful computer chips), they could make the whole system much faster. In their tests, they showed that HeraSys could cut the time it takes for the slowest requests to finish by up to 2.17 times (meaning they were more than twice as fast) and increased the total number of tasks the system could handle by up to 1.85 times compared to the best existing systems.

The paper argues against the old way of doing things, where systems treat every request as an independent, isolated job. They show that this "isolation" creates unnecessary waste and bottlenecks. Instead, they propose a collaborative approach where the system actively looks for shared work. They also argue against simple scheduling rules like "first come, first served," which can cause long tasks to block everyone else, or "shortest job first," which can make long tasks wait forever. HeraSys suggests a balanced, "load-aware" strategy that reserves some resources for the heavy tasks while prioritizing the quick ones, ensuring fairness and speed for everyone.

The results, measured through extensive experiments on real hardware, suggest that this fine-grained, collaborative approach is a significant step forward. By fusing redundant steps and dynamically adjusting how tasks are grouped and executed, HeraSys manages to keep the average wait time low while preventing the "tail" of slow, frustrating delays that often plague complex AI systems. It's a bit like realizing that if you organize a group of friends to clean a house, you shouldn't just assign each person a room; you should notice that two people are both vacuuming the same hallway and have them work together, while making sure the person washing the windows doesn't get stuck waiting for the vacuum cleaner.

Technical Summary: HeraSys

Problem Statement

The proliferation of Large Language Models (LLMs) has shifted application development from simple, single-turn interactions to complex, multi-tenant agentic workflows represented as Directed Acyclic Graphs (DAGs). Existing serving systems primarily optimize performance within a single workflow or focus on intra-workflow dependencies (e.g., LLM inference batching). However, these approaches treat concurrent workflows as independent instances, leading to two critical inefficiencies in multi-tenant environments:

Computational Redundancy: Concurrent queries often share structural commonalities, such as embedding the same document chunks, retrieving identical data, or invoking the same tools with similar contexts. Current modular designs lack a global view to detect these overlaps, forcing the system to execute redundant tasks.
Scheduling Inefficiency and Unfairness: Real-world workloads exhibit high heterogeneity, mixing short tasks (e.g., simple QA) with long tasks (e.g., multi-round reasoning). Traditional scheduling policies like First-Come-First-Served (FCFS) suffer from Head-of-Line (HOL) blocking, where long tasks delay short ones. Conversely, strictly prioritizing short tasks (e.g., Shortest Job First) can cause starvation for long tasks, degrading fairness and tail latency.

Methodology

The authors propose HeraSys, a fine-grained end-to-end serving system designed to optimize concurrent multi-workflow scenarios through a dual-layer optimization architecture.

1. Pre-Execution Optimization: Semantic Graph Fusion

HeraSys breaks request-level isolation by analyzing the global context of incoming queries before execution.

Node Semantic Signature: The system assigns a unique hash signature to each task node based on its operator type, parameters, input data, and the signatures of its predecessors. This allows the system to identify logically independent but computationally equivalent tasks across different queries.
Topological Fusion: For concurrent queries arriving within a short time window, the system fuses redundant nodes. If a node in a new query matches a node already in the ready queue, the new node is removed from the execution graph, and its downstream dependencies are redirected to the existing node.
Asynchronous Cache Reuse: For requests outside the time window, the system reuses results from completed nodes or waits for currently executing nodes to finish, sharing the output.
Priority Adjustment: Fused nodes are assigned a higher scheduling priority, calculated as the sum of the contributions of all merged logical nodes, to accelerate the progress of multiple downstream paths simultaneously.

2. Runtime Optimization: Load-Aware Joint Scheduling

The runtime scheduler dynamically manages execution order and resource allocation to balance average latency with fairness for long-tail tasks.

Dynamic Remaining Time Estimation: The system estimates the remaining service time ( $\hat{S}_q$ ) for each query by combining offline profiling with online execution progress. Long decoding operators are decomposed into sequential sub-nodes to enable fine-grained preemption and accurate tracking.
Node Contribution Metric: A metric ( $\phi_v$ ) quantifies a node's contribution to query progress based on downstream workload, path length, and parallel branches.
Resource Skewing and Partitioning: To prevent starvation, the system classifies queries as "long" or "short" based on a dynamic threshold derived from the distribution of remaining times. It enforces a minimum resource reservation for heavy nodes in long queries, ensuring they receive a guaranteed share of resources while short tasks are prioritized for low latency.
Joint Priority Function: The scheduler orders ready nodes using a composite priority function $\pi_v$ $π_{v}$ that balances three factors:
1. Dynamic Shortest Remaining Time First (SRTF): Minimizes average latency.
2. Topological Contribution: Prioritizes critical nodes that unlock downstream work.
3. Anti-Starvation: Boosts the priority of heavy nodes in long tasks.
Adaptive Execution: The system dynamically adjusts batch sizes and pipeline decomposition granularity based on system load. Under high load, it increases batching to improve throughput; under low load, it increases pipeline splitting to exploit data parallelism and reduce latency.

Key Contributions

Dual-Layer Optimization Architecture: HeraSys introduces a system that overcomes traditional request-level isolation by integrating pre-execution graph fusion with runtime dynamic scheduling, achieving joint optimization of resource utilization and scheduling efficiency.
Cross-Query Graph Node Reuse: A mechanism to identify and eliminate redundant computational subgraphs across workflows via semantic signature matching, significantly reducing computational overhead.
Load-Aware Joint Scheduling Strategy: A strategy that integrates dynamic task classification, biased resource allocation, and adaptive batching to effectively reduce end-to-end latency while ensuring fairness for long-tail tasks.
Implementation and Evaluation: The authors implemented HeraSys on a server with NVIDIA RTX 4090 GPUs using vLLM as the backend and demonstrated its effectiveness through extensive comparisons with state-of-the-art baselines.

Experimental Results

The authors evaluated HeraSys against LangGraph, LlamaIndex, and the state-of-the-art system Ayo across diverse workloads (RAG with/without reranking, Web Search, and mixed scenarios) under varying concurrency levels (6 to 300).

Latency Reduction: HeraSys reduced P99 latency by up to 2.17× and P95 latency by up to 1.94× compared to baselines under strict latency guarantees. In mixed workloads, it achieved a 2.17× improvement in P99 latency.
Throughput Improvement: The system increased serving throughput by up to 1.85× compared to baselines.
Ablation Studies: Experiments confirmed that removing the Graph Fuser reduced throughput by approximately 1.16×, and using only SRTF scheduling (without resource skewing) increased P99 latency by 22.3% compared to the full system, highlighting the necessity of the joint optimization approach.
Fairness: The resource skewing mechanism successfully mitigated starvation issues common in pure SRTF strategies, providing a more predictable Quality of Service (QoS) for long tasks without significantly compromising short task responsiveness.

Significance and Claims

The paper claims that HeraSys provides a significant step forward in LLM serving infrastructure by shifting the focus from isolated request optimization to collaborative, cross-workflow optimization. By exploiting fine-grained structural commonalities and dynamically balancing resource contention, HeraSys addresses the inherent redundancies and scheduling challenges in multi-tenant agentic workflows. The authors posit that this approach offers insights for building more efficient, scalable, and fair LLM application serving systems, ultimately reducing computational resource consumption and operational costs while enhancing user experience.