Technical Summary: GENESISFUNC

Problem Statement

The advancement of Large Language Models (LLMs) into proactive agents capable of interacting with dynamic environments relies heavily on function-calling (FC). However, the field faces a critical bottleneck: the scarcity of high-quality, diverse, and broadly representative training data.

• Real-world data limitations: Collecting and annotating real function-calling data is costly, labor-intensive, and often fails to capture the complexity of ambiguous user intents, multi-turn interactions, and rapidly changing environments.
• Synthetic data limitations: Existing automated pipelines for generating synthetic FC data suffer from unreliable APIs, limited tool scalability, insufficient diversity (often focusing on single-turn or isolated calls), and weak quality control. These limitations constrain the generalizability and robustness of tool-augmented LLMs.

Methodology: GENESISFUNC Pipeline

To address these challenges, the authors propose GENESISFUNC, an automated, three-stage pipeline designed to generate accurate, diverse, and scenario-rich FC training data. The pipeline is built upon reliable tools from public benchmarks and employs a multi-agent framework to ensure quality and diversity.

1. Tool Pool Construction

Instead of relying on manually constructed or unreliable synthetic APIs, GENESISFUNC curates a Tool Pool of 1,000 tools sourced directly from the BFCL (Berkeley Function-Calling Leaderboard) benchmark.

• Process: A two-stage filtering process is used. First, GPT-4o performs semantic clustering to remove redundant or highly similar tools. Second, light human verification ensures correctness and usability.
• Goal: This ensures the underlying APIs are reliable, cover multiple real-world domains, and are practically accessible, forming a robust foundation for data generation.

2. Multi-Agent Dialogue Generation

This module leverages a coordinated multi-agent framework to synthesize dialogues that span diverse scenarios (single-turn, multi-turn, and special cases). The system utilizes four distinct agents (backed by LLMs, specifically Gemini-2.5 Pro in the primary implementation):

• Sample Agent: Selects a subset of tools from the pool, grouping semantically similar tools as targets and selecting distractors with varying relevance to force the model to distinguish between related and unrelated tools.
• Memory Agent: Records past dialogue instances and their semantic context types. It summarizes history to guide the generator toward producing dialogues with unseen semantic types, thereby enhancing semantic diversity.
• Function Agent: Selects specific tools to address user queries and extracts parameters. Crucially, it employs a slot-selection strategy, randomly choosing a subset of optional parameters to instantiate. This creates varied and realistic tool calls rather than rigid, full-parameter invocations.
• Judge Agent: Evaluates multiple candidate dialogues generated in a round, selecting the best one based on problem significance and tool appropriateness, while controlling for positional bias.

The Dialogue Generation System orchestrates a User Agent and an Assistant Agent to produce:

• Single-turn and Multi-turn dialogues: Covering both single-task and multi-task settings.
• Special cases: Scenarios involving relevance checking (mismatched tools) and error detection (invalid parameters).

3. Multi-Stage Evaluation

To guarantee the correctness of the synthesized data, a rigorous three-step evaluation process is applied:

1. Rule Checker: Performs static, rule-based compliance checks on tool definitions, call formats, parameter types, and dialogue structure without executing tools.
2. Model Checker: Uses GPT-4o to evaluate semantic quality, task completion, and faithfulness beyond static rules. Only samples with a confidence score above a threshold (0.75) are retained.
3. Human Validation: A targeted human review of the remaining ~5% of samples that fail automated checks or have low confidence. This stage focuses on errors preventing correct execution or those in high-impact functions.

Key Contributions

1. GENESISFUNC Pipeline: The introduction of an automated, end-to-end pipeline for generating high-quality FC training data. It uniquely combines reliable, benchmark-sourced tools with a multi-agent dialogue generation system and a multi-stage evaluation module.
2. High-Quality Dataset: The construction of a dataset that is accurate, diverse, and covers real-world scenarios (including multi-turn and multi-task interactions). The authors demonstrate strong scalability to downstream tools, proving the pipeline's practical applicability.
3. State-of-the-Art Performance: The fine-tuning of Qwen3-8B (resulting in GENESISFUNC-8B) using this dataset. The model achieves superior performance compared to similarly sized open-source models and remains highly competitive with proprietary API-based models.

Experimental Results

The authors evaluated GENESISFUNC-8B on three widely used benchmarks: BFCL (in-domain), API-Bank, and ACEBench (out-of-domain).

• In-Domain (BFCL): GENESISFUNC-8B achieved an overall accuracy of 93.31% on the Non-Live split and 83.78% on the Live split. It outperformed open-source baselines of comparable scale (e.g., ToolACE-8B, LLaMA-3.1-8B) and surpassed or matched larger models (e.g., Qwen3-32B). Notably, it exceeded the performance of API-based models like GPT-4o-mini on the Non-Live split.
• Out-of-Domain (API-Bank & ACEBench): The model demonstrated strong generalization, achieving 64.79% on API-Bank and 83.67% on ACEBench. These results represent significant relative gains over prior open-source SOTA and performance on par with GPT-4o-mini and Gemini-1.5-Pro.
• Ablation Studies:
  • Removing the Judge Agent or Memory Agent resulted in clear performance drops, validating the necessity of accuracy control and diversity enhancement.
  • Slot Selection: Dynamic slot selection (randomly choosing optional parameters) outperformed fixed strategies (filling all or none).
  • Evaluation Module: Models trained on data passing the multi-stage evaluation significantly outperformed those trained on non-evaluated data.
• Scalability & RL:
  • The pipeline scales effectively to new tools (e.g., adding ACEBench tools improved performance on ACEBench without degrading BFCL results).
  • Applying Reinforcement Learning (RL) via GRPO on multi-turn data further enhanced performance in complex interaction scenarios, improving multi-turn accuracy from 65.00 to 70.00 on ACEBench.

Significance and Claims

The paper claims that GENESISFUNC represents a concrete advance in tool-augmented AI agents by solving the data bottleneck through a systematic, automated approach.

• Bridging the Gap: The results suggest that strong, systematic alignment between training data distribution and real tool semantics can significantly narrow the performance gap between smaller open-source models and much larger or proprietary models.
• Practical Applicability: The methodology is not limited to specific model architectures; it is model-agnostic and scalable to broader tool inventories.
• Future Potential: While the current work focuses on function-calling, the authors posit that the methodology can be extended to more agentic and complex function-calling tasks.

The authors maintain a modest tone regarding limitations, acknowledging that while GENESISFUNC-8B is competitive, it still falls short of top-tier API-based models (like GPT-4) in broader reasoning and comprehension, and that highly complex, tightly coupled multi-turn scenarios remain an area for future work.