Technical Summary: From Intent to Execution: Composing Agentic Workflows with Agent Recommendation

1. Problem Statement

The rapid proliferation of specialized AI agents (both general-purpose and domain-specific) has created a combinatorial challenge in Multi-Agent Systems (MAS): efficiently identifying the "best" agent for a specific task within a vast, growing registry. Traditional approaches to designing agentic workflows suffer from significant manual overhead, requiring human intervention for:

• Plan Composition: Manually designing the sequence of tasks.
• Agent Selection: Manually mapping tasks to specific agents.
• Execution Graph Creation: Manually constructing the logic for agent interaction.

Furthermore, existing retrieval methods (keyword-based or simple LLM context searches) are inadequate for large-scale agent ecosystems. They often fail to capture semantic nuances, struggle with the "needle in a haystack" problem as registries expand, and lack the ability to optimize for global constraints (e.g., latency, cost, security) rather than just local task similarity.

2. Methodology: The AutoMAS Framework

The authors propose AutoMAS, an automated framework designed to convert user intents into functioning multi-agent applications with minimal human intervention. The system architecture comprises four primary components:

2.1 Planner

The planner accepts a user intent in natural language and generates a structured plan. Unlike linear sequences, the plan is represented as a Finite State Machine (FSM) to accommodate non-sequential logic, recursion, and dynamic paths. The planner can leverage episodic and semantic memories (databases of previous plans) to refine the generated plan via Retrieval-Augmented Generation (RAG).

2.2 Variable Call Graph (VCG)

The VCG acts as a dynamic routing mechanism. It maps the tasks in the FSM to agents retrieved from a registry. Crucially, the VCG supports multiple execution routes based on:

• Redundancy: Alternative agents for the same task to ensure fault tolerance.
• Dynamic Optimization: Selecting routes based on real-time environmental factors (e.g., network shutdowns) or user constraints (e.g., lowest latency vs. lowest cost).

2.3 Agent Recommender (AR)

The core of the system is a two-stage hybrid retrieval system designed to map natural language task descriptions to agents in a registry.

• Stage 1: Fast Retriever: Utilizes hybrid search (combining dense vector embeddings with sparse keyword matching, e.g., BM25) to quickly identify a broad set of candidate agents (high recall).
• Stage 2: LLM-based Re-ranker: Takes the top candidates from Stage 1 and re-orders them using a Large Language Model. This stage provides a nuanced semantic understanding of the task-agent fit, prioritizing precision at the top of the list.
• Agent Description Enrichment: To improve retrieval quality, the system generates synthetic queries from agent descriptions at ingestion time. These queries are appended to the original description, and their embeddings are averaged to create a more robust semantic representation for the vector store.

2.4 Critique Mechanism

A third, optional stage involves a Critique Agent that performs a holistic review of the agent recommendations against the overall plan and user constraints.

• In-the-loop (Task-specific): Evaluates if the recommended agent fits the specific task description.
• Outside-the-loop (Plan-specific): Evaluates the global optimality of the agent selection, ensuring input-output compatibility across the Variable Call Graph and adherence to constraints like cost, latency, and safety.
• The critique does not re-rank agents but can suggest refinements to the task plan or agent selection if the current configuration is suboptimal.

2.5 Human-in-the-Loop (HIL)

The framework supports human intervention at both design time (to improve the application structure) and runtime (to resolve ambiguities or correct detected errors via supervising nodes in the execution graph).

3. Key Contributions

1. Automated MAS Composition: A complete pipeline that automates the transition from user intent to an executable agent workflow, replacing manual composition steps.
2. Two-Stage Hybrid Retrieval: A scalable architecture for agent selection that combines fast vector/keyword retrieval with LLM-based re-ranking, addressing the scalability limitations of context-window-based approaches.
3. Agent Description Enrichment: A technique to enhance the semantic representation of agents by generating and averaging synthetic queries derived from agent descriptions during the indexing phase.
4. Global Critique Mechanism: The integration of a critique agent that evaluates agent selection not just locally, but against the global workflow topology and user constraints, improving the robustness of the final composition.
5. Dynamic Execution Graphs: The use of a Variable Call Graph to enable redundant and dynamic routing, allowing the system to adapt to runtime failures and optimization constraints.

4. Experimental Results

The authors evaluated the system using the ToolE dataset (for agent recommendation) and the TaskBench dataset (for end-to-end planning and composition).

4.1 Agent Recommendation Performance

• Embedding & Re-ranking: The combination of large embeddings (text-embedding-3-large) and an LLM re-ranker (gpt-4o or o1) significantly improved precision (nDCG@1) compared to retriever-only baselines.
• Enrichment & Filtering: Agent description enrichment (averaging embeddings of synthetic queries) consistently improved Recall and mAP. Adding a filtering step (where the re-ranker discards irrelevant candidates) yielded the highest gains, achieving a Recall of 0.8398 and mAP of 0.8173.
• Comparison: The proposed approach outperformed the state-of-the-art Re-Invoke method, which relies on providing the full tool list as context to an LLM. The AutoMAS approach achieved higher recall and precision while being more scalable to larger agent registries.

4.2 End-to-End Planning Performance

Evaluated on TaskBench (covering Daily Life, Hugging Face, and Multimedia APIs):

• Semantic Metrics: Using LLM-as-a-Judge, the system achieved a 59.3% overall match rate and a 0.742 step-wise ratio, indicating that while exact wording may differ, the semantic intent and tool selection were largely correct.
• Structured Metrics: Under structured prediction evaluation (focusing on tool names, arguments, and sequence), the system showed high fidelity. For the "Daily Life" category, it achieved a Tool/Node Name F1 of 0.9064 and Sequence Similarity of 0.8956.
• Impact of Critique: Integrating the critique mechanism with iterative correction further improved metrics. For "Daily Life," the Tool/Node Name F1 increased from 0.9064 to 0.9161, and Sequence Similarity improved from 0.8956 to 0.9034, demonstrating that the critique effectively corrects ordering and structural errors.

5. Significance and Claims

The paper claims that AutoMAS represents a significant step toward scalable, automated multi-agent system design.

• Scalability: Unlike approaches that feed entire agent registries into an LLM's context (which is limited by token counts and cost), the proposed two-stage retrieval architecture scales effectively to registries with thousands of agents.
• Robustness: The inclusion of the Variable Call Graph and the Critique Mechanism ensures that the system is not only accurate in selecting individual agents but also robust in constructing a globally optimal workflow that adheres to constraints.
• Modest Limitations: The authors acknowledge that LLM-based re-rankers may exhibit inherent biases (e.g., favoring lower-cost agents even when instructed otherwise) and that the mechanism for how critique improves retrieval queries requires further investigation. They also note that while their approach scales better, it currently cannot dynamically alter the selection of agents beyond the retrieved set, unlike frameworks where the LLM has full context of all tools to choose from.

In conclusion, the work demonstrates that a multi-stage approach combining fast retrieval, semantic re-ranking, description enrichment, and holistic critique is essential for building reliable, end-to-end multi-agent systems capable of handling complex user intents in large-scale agent ecosystems.