Technical Summary: Learning to Configure Agentic AI Systems (ARC)

Problem Statement

Configuring Large Language Model (LLM)-based agent systems involves selecting from a vast combinatorial design space comprising workflows, tools, token budgets, and prompts. Current approaches rely on static heuristics or "kitchen sink" strategies that apply a fixed configuration regardless of query difficulty. This "one-size-fits-all" approach leads to two primary issues:

1. Performance Degradation: Long contexts often suffer from the "lost-in-the-middle" phenomenon, where models fail to attend to relevant information.
2. Inefficiency: Static systems waste computational resources by applying heavy, iterative workflows (e.g., multi-hop reasoning with web search) to trivial queries, leading to unnecessary latency and cost.

The core challenge is that the design space is too large for brute-force search or manual tuning. For instance, a modest system with a few workflow patterns, tools, and budget levels can yield tens of thousands of structural configurations, which expands to over $10^5$ when including prompt variations. Furthermore, agent workflows are not atomic actions; they are temporally extended processes with variable durations and costs, making standard Markov Decision Process (MDP) formulations inadequate.

Methodology: ARC (Agentic Resource & Configuration learner)

The authors propose ARC, a hierarchical reinforcement learning framework that formulates agent configuration as a Semi-Markov Decision Process (SMDP). In this formulation, each configuration acts as a temporally extended "option" that determines how an agent system processes a query.

1. SMDP Formulation

Unlike standard MDPs where actions have fixed durations, ARC models configurations as options with variable execution times (number of LLM calls) and costs.

• State ($S$): A concatenation of semantic embeddings of the query and hand-crafted features (e.g., query length, numerical density).
• Options ($O$): Each option represents a complete configuration $(\omega, t, b, p)$, where $\omega$ is the workflow, $t$ are tools, $b$ is the token budget, and $p$ is the prompt.
• Reward: A utility function balancing correctness against cost: $U(q, c) = I[\hat{a} = a] - \lambda C_{cost}(c)$. The reward includes penalties for token usage, runtime, and duration (number of LLM calls).

2. Hierarchical Policy Decomposition

To manage the combinatorial complexity, the policy $\pi_{arc}$ is decomposed into two levels:

• Structure Policy ($\pi_{struct}$): Selects the workflow topology, tool allocation, and budget tiers.
• Prompt Policy ($\pi_{prompt}$): Autoregressively composes specific prompt instructions from a library of semantic fragments based on the structural choice.
• Action Masking: To handle structural invalidity (e.g., assigning tools to a single-agent "Direct" workflow), the system applies action masking to prune invalid configurations during training.

3. Training Pipeline

The training process combines Reinforcement Learning (RL) with Supervised Fine-Tuning (SFT):

1. RL Phase: Uses Proximal Policy Optimization (PPO) with shaped rewards to explore the design space. The reward function is decomposed into terms for correctness, step penalties, token penalties, and tool shaping to address the mismatch between tool allocation and actual usage.
2. SFT Refinement: After RL converges, the system filters the replay buffer for "elite" trajectories (correct answers with high rewards). A supervised fine-tuning phase then distills these successful strategies into the policy networks. This step stabilizes learning, reduces variance, and provides a performance floor.

4. Theoretical Guarantees

The paper establishes two theoretical results:

• Convergence: The SMDP formulation admits a unique optimal value function $Q^*$, ensuring the RL target is well-defined.
• Policy Concentration: The SFT phase guarantees that the final policy concentrates on high-reward configurations, ensuring a minimum performance threshold.

Key Contributions

1. SMDP Formulation: The authors cast agent configuration as an SMDP problem, treating workflows as temporally extended options to naturally handle variable computational footprints.
2. ARC Framework: Introduction of a lightweight, hierarchical policy that dynamically selects query-specific configurations (workflow, tools, budget, prompts) without updating the backbone LLM.
3. Hybrid Training: A pipeline combining masked PPO with SFT on elite trajectories to stabilize learning under sparse rewards and improve consistency.
4. Empirical Validation: Extensive experiments demonstrating that learning per-query configurations is a superior alternative to static designs.

Experimental Results

The authors evaluated ARC across reasoning, tool-use, and agentic benchmarks using Qwen 2.5 7B and Gemini 2.5 Flash Lite.

• Reasoning Tasks: ARC improved average reasoning accuracy by 31.3% over budget-matched tool-augmented LLMs. On GSM8k, ARC achieved 88.6% accuracy, outperforming strong baselines like GEPA (83.6%) and RL Episodes (85.2%).
• Tool-Use Tasks: ARC increased tool-use accuracy by 13.95%. On HotpotQA, it matched AutoGen (34.1%) and outperformed the strongest RL baseline by 6.3%.
• Agentic Tasks: On $\tau$-Bench (Airline), ARC doubled the Pass$^1$ success rate from 9.0% to 18.0%, whereas fixed prompt optimization (DSPy) reduced performance.
• Efficiency: ARC occupies Pareto-optimal regions in the accuracy-cost trade-off, achieving higher accuracy at lower costs compared to fixed or search-based baselines.
• Generalization: The learned policies showed moderate transferability across reasoning tasks (e.g., GSM8k to DROP) and zero-shot scaling across model sizes (7B to 72B), indicating that structural priors are largely invariant to scale.

Significance and Claims

The paper claims that learning per-query agent configurations is a powerful alternative to "one-size-fits-all" designs. The significance of this work lies in demonstrating that:

• Adaptability: An agent system can dynamically adapt its computational footprint to task difficulty, balancing accuracy and efficiency.
• Scalability: A lightweight controller can manage a massive combinatorial design space effectively without retraining the underlying LLM.
• Practicality: The combination of RL exploration and SFT refinement provides a tractable method for navigating complex agent architectures, offering a practical direction for building flexible and compute-efficient LLM-based agent systems.

The authors emphasize that while prompt semantics can dominate in knowledge-intensive domains, the structural optimization provided by ARC remains essential for resource allocation and handling complex, multi-step interactions.