Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective

1. Problem Statement

Agentic workflows are becoming a dominant workload in modern AI systems. These workflows consist of sequences of interdependent Large Language Model (LLM) calls (e.g., multi-agent debates, iterative refinement, parallel exploration) orchestrated to solve complex tasks.

The Core Challenge:
Existing LLM serving systems (e.g., vLLM) and orchestration frameworks (e.g., LangGraph, AgentScope) suffer from significant inefficiencies when handling these workflows because they treat LLM calls as isolated units or black-box User-Defined Functions (UDFs).

• Redundancy: Agentic workflows often involve speculative execution and parallel exploration, leading to massive redundancy in prompts and intermediate results (e.g., multiple agents sharing the same system prompt or context).
• Lack of Visibility: Current systems lack visibility into the broader workflow structure (Directed Acyclic Graph or DAG). They optimize individual inference calls (local optimization) rather than the entire workflow (global optimization).
• Statefulness: LLMs are inherently stateful (maintaining KV caches). Standard data processing optimizations fail to account for the ability to reuse KV states across different operators and queries within a batch.
• Inefficiency: Treating LLMs as black-box UDFs prevents the system from reordering operations, pruning redundant branches, or proactively caching states, leading to significant computational waste and latency.

2. Methodology: The Helium System

The authors propose Helium, a workflow-aware serving framework that rethinks LLM serving from a data systems perspective. Helium models agentic workloads as query plans where LLMs are treated as first-class operators, enabling the application of classic database query optimization principles.

Key Architectural Components:

A. DSL and Workflow Representation

• Helium uses a Domain-Specific Language (DSL) to define agentic workflows as DAGs.
• Nodes represent operators (LLM calls, data retrieval, transformations), and edges represent data/prompt flow.
• The system distinguishes between static prompt components (system prompts, fixed contexts) and dynamic components (user inputs, outputs from previous operators).

B. Query Optimizer (Logical Plan Optimization)
Helium employs a two-stage optimization process:

1. Initial Plan Pruning:
   • Operator Pruning: Removes dead code or speculative branches that do not contribute to the final output.
   • Common Subgraph Elimination (CSE): Identifies and merges structurally identical subgraphs that share the same inputs, ensuring computations are performed only once.
2. Cache-Aware Logical Optimization:
   • Prompt Caching: Uses a global prompt cache to map deterministic operator inputs to their outputs. If an operator's input matches a cached entry, the LLM call is replaced by a lightweight CacheFetch operator, bypassing the LLM entirely.
   • Proactive KV Caching: Unlike existing systems that passively cache KV states based on incoming requests, Helium pre-warms the KV cache for static prompt prefixes identified during compilation.

C. Templated Radix Tree (TRT)

• To manage the complex structure of prompts across a batch, Helium constructs a Templated Radix Tree (TRT).
• The TRT captures both the prefix structure of static prompts and the dependencies among dynamic parts (outputs from other operators).
• This structure allows the system to identify shared prefixes across different queries and operators, which is crucial for efficient scheduling.

D. Cache-Aware Scheduling

• Helium formulates the scheduling problem as a multi-worker scheduling task aimed at minimizing the total "token steps" (a proxy for wall-clock latency).
• Cost Model: The scheduler considers token usage (prefill vs. decode), precedence delays (dependencies), and KV cache capacity.
• Algorithm: A greedy, DFS-based algorithm traverses the TRT to generate a schedule. It prioritizes subtrees with the greatest dependency depth and groups operators sharing static prefixes into "inner sequences." This ensures that workers process requests with shared prefixes consecutively to maximize KV cache reuse, while interleaving independent tasks to hide latency.

3. Key Contributions

1. Workflow-Aware Serving Layer: Helium is the first system to model agentic workflows as query plans with LLMs as first-class operators, enabling holistic intra- and inter-query optimization.
2. Proactive Caching Strategy: Introduces a mechanism to pre-warm KV caches for static prompt prefixes and maintain a global prompt cache to bypass redundant operators, moving beyond reactive, opportunistic caching.
3. Cost-Based, Cache-Aware Scheduling: Designs a novel scheduling algorithm using a Templated Radix Tree to capture prompt structure and dependencies, maximizing prefix cache reuse across batch agentic workloads.
4. End-to-End Optimization: Demonstrates that bridging AI/ML systems with database query optimization principles (logical rewriting, cost-based scheduling) significantly outperforms state-of-the-art solutions.

4. Experimental Results

The authors evaluated Helium on various microbenchmarks (Map-Reduce, Multi-Agent Debate, Reflection) and a complex end-to-end "Trading" workflow using Qwen3-8B and Qwen3-14B models on NVIDIA H100 GPUs.

• Performance Gains:
  • vs. vLLM (Naive): Up to 100.92× speedup (due to vLLM's sequential execution of operators).
  • vs. OpWise (Operator-by-Operator): Up to 1.58× speedup.
  • vs. LangGraph: Up to 1.83× speedup.
  • vs. AgentScope: Up to 4.32× speedup.
  • vs. Parrot (Prefix-aware): Up to 2.21× speedup (Parrot lacks global scheduling).
  • vs. KVFlow: Up to 1.56× speedup.
  • Complex Workflow (Trading): Up to 39.50× speedup over vLLM and 1.34× over KVFlow.
• Ablation Studies:
  • Disabling Plan Pruning caused the largest performance drop (23.35%), highlighting the importance of removing redundant computations.
  • Disabling Cache-Aware Scheduling caused a 17.66% drop, proving that intelligent task ordering is critical.
• Optimality: Helium's scheduling algorithm achieved an average optimality gap of 0.9% compared to the theoretical optimum (solved via MILP), significantly outperforming other heuristics.
• Overhead: The planning overhead (optimization + scheduling) is negligible (<230ms) compared to the total inference time, and memory footprint for scheduling metadata is significantly lower than SGLang's RadixCache.

5. Significance

This paper represents a paradigm shift in how LLM serving is approached for agentic systems:

• From Reactive to Proactive: It moves beyond optimizing individual inference calls to optimizing the entire workflow graph, anticipating redundancy before execution.
• Bridging Disciplines: It successfully applies decades of database query optimization (CSE, cost-based scheduling, logical rewriting) to the emerging field of LLM agents, treating LLMs as white-box operators rather than black boxes.
• Scalability: By eliminating redundant computation and maximizing hardware utilization through intelligent caching and scheduling, Helium provides a scalable path for deploying complex, multi-agent AI systems in production.

In conclusion, Helium demonstrates that end-to-end optimization across agentic workflows is essential for efficient LLM serving, achieving substantial speedups by treating the workflow as a data processing pipeline rather than a collection of isolated API calls.