Enhancing Tool Calling in LLMs with the International Tool Calling Dataset

This paper introduces International Tool Calling (ITC), a large-scale, multilingual benchmark comprising over 3,500 real APIs and 17,000 tasks across 40 countries, designed to address the limitations of existing datasets by improving LLM robustness, cross-lingual generalization, and performance in realistic global tool-calling scenarios.

The Gist

Imagine you have a super-smart robot assistant (a Large Language Model, or LLM) that can write poems, answer trivia, and chat about anything. But right now, this robot is stuck in a library. It knows about the world, but it can't actually do anything in the real world. It can't check the weather, book a flight, or look up a stock price because it doesn't have a key to the outside world.

Tool Calling is giving that robot a set of keys (APIs) so it can open doors, turn on lights, and interact with the real world.

This paper introduces a new, massive "training gym" called International Tool Calling (ITC) to teach these robots how to use those keys better, especially when the world gets complicated and multilingual.

Here is the breakdown of why this paper matters, using some everyday analogies:

1. The Problem: The Robot's "Fake" World

Before this paper, researchers trained robots to use tools using simulated environments.

• The Analogy: Imagine teaching a pilot to fly by having them sit in a flight simulator where the clouds are just paintings on a wall and the wind is a fan. It looks like flying, but if you take that pilot out to a real storm, they might crash.
• The Reality: Many existing datasets used fake APIs or real ones that were locked behind paywalls or restricted to English speakers. They didn't teach the robots how to handle the messy, diverse, and often broken reality of the actual internet. Also, most training was only in English, leaving the robots confused when asked to book a train in Tokyo or check a recipe in Spanish.

2. The Solution: The "International Tool Calling" (ITC) Dataset

The authors built a massive, realistic training ground.

• The Scale: They collected 3,571 real-world tools (like weather apps, translation services, and banking APIs) from 40 different countries.
• The Diversity: They created 17,540 tasks (questions) in 29 different languages.
• The Analogy: Instead of a flight simulator with painted clouds, they dropped the pilot into a real airport with real turbulence, different air traffic controllers speaking different languages, and airports in 40 different countries. They even included tricky scenarios where the robot has to use multiple tools in a row (like checking the weather, then booking a flight, then reserving a hotel) to solve one problem.

3. How They Built It: The "Quality Control" Factory

Building this dataset wasn't just about downloading files; it was a rigorous process.

• Step 1: The Hunt: They scoured the internet for real APIs.
• Step 2: The Filter: They tested every single one. If an API was broken, didn't work, or gave weird errors, they threw it out. (Like a chef tasting every ingredient before cooking).
• Step 3: The Human Touch: They hired 100 human experts to check the questions. They made sure the questions were clear, culturally appropriate, and actually solvable.
• Step 4: The "Three-Way" Check: To make sure the answers were perfect, they used three different super-intelligent AIs to generate answers, then had humans pick the best one. This prevented the "hallucinations" (making things up) that AIs are famous for.

4. The Results: The "Before and After"

They tested 24 different robots (LLMs) on this new gym.

• The Gap: The "closed-source" robots (like GPT-4o, which are expensive and proprietary) were already pretty good, but the "open-source" ones (free to use) struggled. They often picked the wrong tool or forgot to fill in the necessary details.
• The Magic of Fine-Tuning: When they took the open-source robots and gave them a crash course using this new ITC dataset, they got significantly smarter.
  • The Analogy: It's like taking a student who knows the theory of driving and putting them behind the wheel for 100 hours of real-world practice in different countries. Suddenly, they aren't just guessing; they are confident drivers.
• The Multilingual Win: The biggest surprise? The robots got much better at handling non-English questions. By training on data from 40 countries, the robots learned to think and reason in the language the user asked, rather than just translating everything to English in their heads first.

5. Why This Matters for You

This isn't just about better robots; it's about fairness and reliability.

• Global Access: Right now, AI tools work great for English speakers in the US but often fail for someone in Brazil, India, or China. This dataset helps fix that gap.
• Real-World Reliability: As we start using AI to book flights, manage finances, or diagnose health issues, we can't afford for the AI to "hallucinate" or pick the wrong tool. This dataset teaches them to be precise and robust.

Summary

Think of this paper as the International Driving License for AI. Before, AI drivers only knew how to drive on quiet, English-only test tracks. Now, thanks to the International Tool Calling (ITC) dataset, they are learning to navigate the chaotic, multilingual, and diverse highways of the real world, making them safer and more useful for everyone, everywhere.

Technical Summary

Here is a detailed technical summary of the paper "Enhancing Tool Calling in LLMs with the International Tool Calling Dataset".

1. Problem Statement

While Large Language Models (LLMs) have made significant strides in tool calling (interacting with APIs), existing benchmarks suffer from critical limitations that hinder the development of robust, real-world applications:

• Reliance on Simulated Data: Many datasets use synthetic or simulated APIs that fail to capture the complexity, variability, and error states of real-world services.
• Reproducibility Issues: Benchmarks using real APIs often rely on paid access keys, strict rate limits, or proprietary restrictions, making them difficult to reproduce or scale.
• Lack of Diversity: Current benchmarks are predominantly English-centric and geographically narrow. They overlook cultural, regulatory, and linguistic nuances, failing to test models on region-specific tools (e.g., local weather, regional news, or specific financial services) or non-English queries.
• Limited Generalization: Models trained on narrow datasets struggle with out-of-domain tools and cross-lingual generalization.

2. Methodology: The International Tool Calling (ITC) Dataset

The authors introduce ITC, a large-scale, multilingual benchmark designed to address these gaps through a rigorous four-stage curation pipeline (illustrated in Figure 1 of the paper):

A. API Collection and Construction

• Source: Aggregated 49,937 real-world REST APIs from five global sources (primarily RapidAPI, Juhe Data, and GitHub repositories).
• Standardization: API documentation was normalized into a unified schema (name, description, endpoints, parameters, authentication).
• Verification: A response-driven manual completion strategy was employed. For APIs with missing metadata, the authors executed live tests (via Python/curl) to verify endpoints and supplement documentation with actual runtime responses.
• Filtering: Automated weekly monitoring removed non-responsive, error-prone, or unstable APIs. This reduced the pool to 3,571 high-quality, callable APIs.

B. Query Generation

• Task Types: The dataset covers Single Tool Calling and Multiple Tool Calling (Repeated, Parallel, and Nested/Chained calls).
• Seed-Based Generation: A manually curated seed pool of 36 high-quality instances was used to generate diverse queries.
• Selection Principles: APIs were selected based on geographic diversity (covering 40 countries), functional variety, and disambiguation challenges (similar API names).
• Multilingual Support: Queries were generated in 29 languages, ensuring representation beyond English.

C. Query Scoring and Filtering

• Automated Scoring: Two LLMs (Claude-3.5-Sonnet and Gemini-1.5-Pro) scored 44,198 generated queries across five dimensions: Relevance, Practicality, Linguistic Applicability, Clarity, and Specificity.
• Human Verification: 100 qualified annotators verified the top-scoring queries using a "Gold Standard" injection method (10% control questions) to ensure quality.
• Final Dataset: 17,540 high-quality QA pairs were retained (15,790 for training, 1,750 for testing).

D. QA Pair Generation

• Tri-Model Strategy: To mitigate bias, three models (GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet) independently generated candidate answers.
• Adversarial Evaluation: Other models evaluated these candidates for reasoning consistency and validity. Human experts performed final selection, particularly for complex tasks.

3. Key Contributions

1. ITC Dataset: The first large-scale, globally distributed tool-calling benchmark featuring 3,571 real APIs and 17,540 tasks across 20 categories and 40 countries.
2. Multilingual & Cultural Diversity: Unlike previous English-only benchmarks, ITC includes 29 languages, with a significant portion of tasks requiring cross-lingual reasoning (e.g., a Japanese user querying a Chinese API).
3. Real-World Validity: All APIs are publicly accessible without authentication keys, ensuring reproducibility and focusing on actual runtime behaviors rather than simulated endpoints.
4. Comprehensive Evaluation Metrics: The authors introduced Language Matching Accuracy (LM) as a new metric to evaluate whether the model's internal reasoning matches the user's input language, alongside standard metrics (Tool Selection, Tool Invocation, Format Matching).

4. Experimental Results

The authors benchmarked 16 open-source and 8 closed-source LLMs on the ITC test set.

• Zero-Shot Performance:

  • Closed-Source Dominance: Models like GPT-4o and Claude-3.5-Sonnet significantly outperformed open-source models, achieving high scores in Language Matching (LM > 95%) and Format Matching (FM > 95%).
  • Open-Source Gaps: While models like DeepSeek-V3 and Qwen2.5-Coder-32B performed well, many smaller open-source models struggled with parameter generation and schema adherence.
  • Error Analysis: The most common errors were missing tools (omission) and incorrect parameter generation. Reasoning-focused models (e.g., o3-mini) sometimes failed format adherence due to excessive chain-of-thought generation.

• Fine-Tuning Impact:

  • Fine-tuning on the ITC training set yielded substantial improvements. For example, Qwen2.5-3B saw a ~50% increase in Tool Invocation F1 scores.
  • Generalization: Fine-tuned models showed marked improvements on external benchmarks (e.g., ToolBench, Seal-Tools), proving that ITC enhances out-of-domain robustness.

• Ablation Study (Language Impact):

  • Models trained only on English data performed significantly worse on non-English queries compared to models trained on the full multilingual dataset.
  • Full multilingual training improved Tool Selection Recall by 42.0% for Qwen2.5-7B on non-English tasks, compared to only 22.6% for English-only training. This highlights the critical role of diverse data for cross-lingual generalization.

5. Significance and Future Directions

• Advancing Robustness: ITC provides a rigorous standard for evaluating LLMs in complex, multi-tool, and international scenarios, moving beyond "toy" examples to real-world utility.
• Bridging the Gap: The dataset demonstrates that fine-tuning on diverse, real-world data significantly closes the performance gap between open-source and closed-source models, particularly in non-English contexts.
• Limitations & Future Work: The authors acknowledge that certain regions (Africa, parts of Asia) remain underrepresented and that the dataset currently focuses on REST APIs (excluding SOAP or database connectors). Future work aims to expand geographic coverage and incorporate more heterogeneous tool types to build truly universal tool-calling systems.

Conclusion: The International Tool Calling (ITC) dataset represents a pivotal step toward making LLMs more capable, culturally aware, and robust in interacting with the global digital ecosystem. It serves as a vital resource for researchers aiming to develop next-generation AI agents capable of operating effectively across linguistic and geographic boundaries.