AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent

1. Problem Definition: Client-Side Agent Optimization

While existing research has heavily focused on server-side optimization (e.g., caching, speculative execution, load balancing) to reduce the cost of serving LLM agents, a critical efficiency gap remains on the client side.

• The Challenge: Developers increasingly build agents by composing heterogeneous components (local tools, remote APIs, and multiple candidate models). They must decide how to allocate resources—specifically, which model to assign to each role in a multi-step pipeline—subject to constraints on cost, latency, and quality.
• The Limitation of Current Approaches: Server-side systems optimize infrastructure for many users but cannot account for application-specific utility functions (e.g., a startup prioritizing cost vs. a medical system prioritizing reliability). Furthermore, standard LLM routing (assigning a model per query based on difficulty) is insufficient for multi-step agents because decisions are coupled across stages. A model's performance in one role (e.g., a "Planner") depends on how its output interacts with downstream roles (e.g., a "Solver").
• The Core Problem: Finding the optimal model combination (assigning a specific model to every role in a pipeline) is a combinatorial optimization problem. The search space grows exponentially with the number of roles ($|M|^N$), and evaluating every combination via brute force is prohibitively expensive.

2. Methodology: The AgentOpt Framework

The authors introduce AgentOpt, the first framework-agnostic Python package for client-side agent optimization. It treats model selection as a black-box optimization problem over full pipeline configurations.

A. Core Abstraction: The "Combo"

Instead of optimizing individual model calls, AgentOpt optimizes combinations ($c = (m_1, m_2, ..., m_N)$), where each $m_i$ is a model assigned to a specific pipeline role. The utility function $J(c)$ considers end-to-end performance, latency, and cost.

B. System Architecture

• Framework Agnostic: AgentOpt does not require rewriting agent code. It intercepts LLM calls at the HTTP transport layer (patching httpx.Client.send) to track metrics uniformly across different frameworks (e.g., LangGraph, AutoGen).
• Attribution: It uses Python contextvars to associate every intercepted API call with the specific data point and model combination currently being evaluated.
• Efficiency Mechanisms:
  • Caching: Reuses responses for identical requests across different combinations to save API costs and time.
  • Parallel Execution: Uses a two-level concurrency scheme to manage API rate limits while maximizing throughput.

C. Search Algorithms

To navigate the exponentially large search space efficiently, AgentOpt implements eight search strategies:

1. Arm Elimination (Default): A multi-armed bandit approach that iteratively evaluates combinations on batches of data and eliminates those with upper confidence bounds below the lower bounds of current leaders.
2. Epsilon-LUCB: Focuses on identifying an $\epsilon$-optimal arm by allocating samples to the most ambiguous pairs.
3. Threshold Successive Elimination: Finds all combinations exceeding a specific quality threshold.
4. Hill Climbing: Explores the neighborhood of a current solution by changing one role assignment at a time.
5. Bayesian Optimization: Fits a surrogate model (e.g., Gaussian Process) to the search space to guide exploration.
6. Brute-Force & Random Search: Baselines for comparison.
7. LM Proposal: Uses a strong LLM to propose a shortlist of promising combinations (found to be unreliable for complex workflows).

3. Key Contributions

1. Formalization of Client-Side Optimization: Defined a distinct systems problem where developers optimize their own agent workflows based on specific utility functions, separate from server-side infrastructure optimization.
2. The "Combo" Abstraction: Demonstrated that model selection in agents is a combinatorial problem over full pipelines, not a sequence of independent routing decisions.
3. AgentOpt Implementation: Released a production-ready, framework-agnostic tool that enables sample-efficient search over model combinations.
4. Algorithmic Evaluation: Provided a comprehensive benchmark of search algorithms, showing that Arm Elimination offers the best trade-off between evaluation cost and solution quality.

4. Experimental Results

The authors evaluated AgentOpt on four benchmarks: HotpotQA, GPQA Diamond, MathQA, and BFCL v3, using 9 different models (including Claude Opus, Qwen, and Ministral).

• Massive Cost Gaps: At matched accuracy levels, the cost difference between the best and worst model combinations ranged from 13× to 32×.
  • Example: On BFCL, Qwen3 Next 80B matched Claude Opus 4.6 in accuracy but was 32× cheaper.
• Counter-Intuitive Optima: The "best" model in isolation is often the worst in a specific role.
  • HotpotQA Case Study: Claude Opus 4.6 (the strongest standalone model) performed worst as a Planner (31.71% accuracy) because it bypassed the solver's search tools. The optimal combination used the cheapest model, Ministral 3 8B, as the Planner and Opus as the Solver, achieving 74.27% accuracy.
• Algorithm Performance:
  • Arm Elimination consistently recovered near-optimal combinations while reducing the evaluation budget by 24–67% compared to brute-force search across three of the four tasks.
  • Bayesian Optimization performed well on smooth landscapes (HotpotQA) but struggled on tasks with sharp, localized optima (MathQA).
  • Hill Climbing failed on HotpotQA (52% find rate) because the optimal solution was non-intuitive and disconnected from local neighbors.

5. Significance and Impact

• Paradigm Shift: The paper argues that efficiency in the agentic era cannot be solved solely by better servers. Developers must actively optimize the composition of their agents.
• Context-Aware Selection: It proves that model quality is role-dependent. A model's value is determined by its interaction with the entire workflow, not just its standalone capabilities.
• Practical Utility: AgentOpt provides a principled, automated way for developers to find the Pareto frontier of performance, cost, and latency, enabling the deployment of highly efficient, cost-effective AI agents without manual trial-and-error.
• Future Direction: This work lays the foundation for broader client-side optimizations, including adaptive tool selection and dynamic scheduling, moving beyond static model assignment.

Code & Benchmarks: The framework is open-source and available at https://agentoptimizer.github.io/agentopt/.