🤖 AI

Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective

This paper addresses the overlooked CPU-centric bottlenecks in Agentic AI serving by characterizing execution patterns and proposing two scheduling optimizations, COMB and MAS, which significantly reduce latency and improve resource utilization on heterogeneous CPU-GPU systems.

Original authors: Ritik Raj, Souvik Kundu, Ishita Vohra, Hong Wang, Tushar Krishna

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ritik Raj, Souvik Kundu, Ishita Vohra, Hong Wang, Tushar Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-fast chef (the AI) who can cook complex meals (generate answers) in seconds. But here's the catch: this chef doesn't have hands. To actually get the job done, they need to call a team of assistants (the CPU) to do things like check the weather, search the internet, write code, or organize files.

For a long time, computer scientists have been obsessed with making the chef faster. They've built bigger kitchens (GPUs) and sharper knives. But this new paper argues that we've been ignoring the assistants. In the world of "Agentic AI" (AI that acts on its own), the assistants are actually the ones causing the traffic jams.

Here is a simple breakdown of what the paper discovered and how they fixed it.

1. The Problem: The "Super-Chef" is Waiting on the "Slow Assistant"

The researchers looked at how these AI agents work. They realized that while the AI (the GPU) is incredibly fast at thinking, it spends a huge amount of time waiting for the computer's main processor (the CPU) to do the "boring" but necessary stuff, like:

Searching the web.
Running code to fix a bug.
Reading a huge database of documents.

The Analogy: Imagine a Formula 1 race car (the GPU) sitting at the starting line. It's ready to go at 200 mph. But the driver (the CPU) is stuck in traffic trying to get the car out of the garage. No matter how fast the car is, the whole race is delayed by the traffic.

The paper found that in many AI tasks, the CPU is responsible for up to 88% of the total time the system takes to finish a job. If you have a super-fast GPU but a slow CPU, your expensive GPU just sits there idle, waiting for the CPU to catch up.

2. The Diagnosis: Two Types of Traffic Jams

The researchers tested different AI tasks and found two main ways the system gets clogged up:

The "One-Size-Fits-All" Jam: When you try to process too many requests at once, the CPU gets overwhelmed. It's like trying to serve 100 customers at a coffee shop with only one barista. The barista (CPU) gets stressed, the line gets long, and the expensive espresso machine (GPU) sits empty because the barista can't keep up with the orders.
The "Mixed Crowd" Jam: Imagine a coffee shop where some customers just want a quick black coffee (simple AI tasks), while others want a complex, custom latte that requires the barista to spend 10 minutes grinding beans and steaming milk (complex AI tasks with tools). If the shop lets the complex orders flood in, the simple coffee drinkers wait forever. If the shop prioritizes the simple orders, the complex orders get stuck. The system becomes unfair and inefficient.

3. The Solution: Two New Rules for the Kitchen

To fix these jams, the authors proposed two new "scheduling" strategies (rules for how to organize the work).

Strategy A: COMB (The "Micro-Order" System)

For when everyone is ordering the same complex drink.

Instead of letting the barista try to make 128 complex lattes at once (which causes a panic and a traffic jam), COMB suggests breaking the big order into smaller "micro-batches" of, say, 64.

The Magic: While the barista is making the first 64 lattes, the espresso machine starts heating up for the next batch. They work in a relay race style.
The Result: The CPU isn't overwhelmed, and the GPU isn't sitting idle. The paper showed this could make the system 1.7 to 3.9 times faster for the average user.

Strategy B: MAS (The "VIP & Regular" Lanes)

For when you have a mix of simple and complex orders.

This system creates two separate lines (queues) at the coffee shop:

The "Quick Coffee" Lane: For simple AI tasks that just need the GPU.
The "Complex Latte" Lane: For tasks that need the CPU to do heavy lifting.

The system guarantees that the "Quick Coffee" lane never gets blocked by the "Complex Latte" lane, and vice versa. Even if the shop is flooded with complex orders, the simple orders still get served quickly.

The Result: This prevents the "minority" type of request from being ignored. It made the slowest requests 2.4 times faster in some tests, ensuring fairness.

4. Why This Matters

Money: GPUs (the super-chefs) are very expensive. If they sit idle waiting for the CPU, you are wasting money. These new methods make sure you get your money's worth.
Speed: Users won't have to wait as long for AI to finish a task.
Energy: The paper also found that the CPU uses a surprising amount of electricity (sometimes more than the GPU!). By optimizing the CPU, we can save energy in data centers.

The Big Takeaway

We used to think the bottleneck in AI was the "brain" (the model). This paper proves that for AI that does things (Agentic AI), the bottleneck is actually the "hands" (the tools and the CPU).

By treating the CPU with the same respect as the GPU and organizing the workflow better, we can make AI systems faster, cheaper, and fairer for everyone. It's not just about building a faster car; it's about clearing the traffic so the car can actually drive.

1. Problem Statement

The rise of Agentic AI (systems where Large Language Models orchestrate tool use, reasoning, and planning) has shifted the computational bottleneck from pure GPU-based LLM inference to CPU-centric tool execution.

The Shift: While monolithic LLMs run primarily on GPUs, Agentic AI frameworks (e.g., ReAct, AutoGPT, RAG) rely heavily on external tools (web search, code execution, database retrieval, chemical simulation) that are orchestrated by the CPU.
The Gap: Prior research has focused aggressively on GPU kernel optimization and KV-cache management. However, these strategies are ineffective for Agentic AI because the end-to-end (E2E) latency is often dominated by CPU-bound tool processing (up to 88% in some cases).
The Challenge: Current serving systems often over-subscribe CPU resources to maximize GPU utilization, leading to premature throughput saturation, increased latency, and inefficient resource utilization in heterogeneous CPU-GPU systems.

2. Methodology

The authors employ a two-fold methodology: Compile-time Characterization and Runtime Profiling, followed by the design of scheduling optimizations.

A. Characterization Framework

Compile-time Taxonomy: The paper introduces a three-dimensional classification for Agentic AI workloads (Figure 1):
1. Orchestrator: LLM-orchestrated vs. Host (Python code)-orchestrated.
2. Agentic Path: Static-path (predetermined) vs. Dynamic-path (runtime decision).
3. Repetitiveness: Single-step vs. Multi-step.
Representative Workloads: Five diverse workloads were selected to capture algorithmic diversity: Toolformer (Math), SWE-Agent (Coding), RAG (Haystack), ChemCrow (Chemistry), and Web-Augmented Agent (LangChain).
Hardware Setup: Experiments were conducted on two distinct systems to isolate bottlenecks:
- Sys 1: High-Performance (HP) CPU (Intel 64-core GNR) + Low-Performance (LP) GPU (RTX-Pro 6000).
- Sys 2: HP CPU (Nvidia Grace 72-core) + HP GPU (H200).

B. Runtime Profiling Insights

Latency Analysis: In tool-dominated workloads (e.g., RAG, ChemCrow), CPU tool execution consumes 60–89% of total E2E latency. Interestingly, systems with HP GPUs and LP CPUs can match the performance of HP GPU systems because the bottleneck shifts entirely to the CPU.
Throughput Saturation: CPU parallelization (Multi-processing) is significantly less efficient than GPU parallelization. Increasing batch sizes leads to premature throughput saturation on the CPU due to OS contention and the Python Global Interpreter Lock (GIL), often before the GPU is fully utilized.
Energy: CPU dynamic energy consumption accounts for 40–61% of total system energy in CPU-centric workloads, highlighting a significant, often overlooked, energy cost.

3. Key Contributions: Scheduling Optimizations

Based on the profiling insights, the authors propose two scheduling strategies for open-loop arrival systems:

1. CPU-Aware Overlapped Micro-Batching (COMB)

Target: Homogeneous workloads (single request type).
Mechanism:
- Micro-batching: Instead of processing a large monolithic batch ( $B_{max}$ ), the system caps the batch size ( $B_{cap}$ ) to the optimal CPU parallelism limit (e.g., $B_{cap} \approx 1-2 \times$ CPU cores) to avoid over-subscription.
- Overlapping: It overlaps the execution of adjacent micro-batches. While the CPU processes the tool execution for micro-batch $i+1$ , the GPU simultaneously performs inference for micro-batch $i$ .
Goal: To balance CPU and GPU utilization, reducing the "phase imbalance" where one device sits idle while the other is busy.

2. Mixed Agentic Scheduling (MAS)

Target: Heterogeneous workloads (mix of CPU-heavy and GPU-heavy requests).
Mechanism:
- Elastic Admission Caps: Uses separate execution queues with specific caps for CPU-heavy ( $E_{cap, CPU}$ ) and GPU-heavy ( $E_{cap, GPU}$ ) requests.
- Shared Reserved Queue: A shared queue handles overflow, providing elasticity.
- Protection: Ensures that a dominant request type (e.g., a flood of GPU-only requests) does not monopolize the concurrency budget, starving the minority request type (CPU-heavy tools).
Goal: To maintain fair utilization of both CPU and GPU resources and protect minority request types from high tail latency.

4. Experimental Results

Evaluations were performed on Sys 1 and Sys 2 under open-loop load conditions.

COMB Performance (Homogeneous):
- Latency: Achieved up to 1.7× lower P50 latency in standalone execution.
- Open-Loop Load: Reduced service latency by 3.9× and total latency by 1.8× at P50/P90 percentiles compared to baselines.
- Throughput: Improved throughput by 1.7× by preventing CPU saturation.
MAS Performance (Heterogeneous):
- Fairness: Significantly reduced latency for minority request types. In skewed mixes, MAS reduced total latency for minority requests by up to 2.37× (P50) and 2.49× (P90).
- Robustness: Maintained high throughput while preventing head-of-line blocking caused by bursty arrivals.
Ablation Studies:
- CPU-Constrained Systems: The optimizations remained effective even on a 16-core CPU system, proving generalizability.
- Energy: The study confirmed that CPU-centric workloads incur massive dynamic energy costs (up to 61%), validating the need for CPU-aware optimization.

5. Significance

Paradigm Shift: This is the first work to quantify and analyze the CPU bottleneck as the primary constraint in Agentic AI serving, challenging the GPU-centric optimization dogma.
Design Principles: It establishes that optimal Agentic AI serving requires concurrent CPU-GPU utilization strategies rather than just maximizing GPU throughput.
Practical Impact: The proposed optimizations (COMB and MAS) offer immediate, deployable solutions for data centers to reduce latency, improve fairness in mixed workloads, and lower energy consumption for the next generation of AI agents.
Cost Efficiency: The findings suggest that systems with high-performance CPUs and moderate GPUs can be cost-effective alternatives for Agentic AI, provided the CPU is not the bottleneck.