Technical Summary: BioTool

Problem Statement

While Large Language Models (LLMs) have achieved significant success in general-purpose tasks, their performance in specialized biomedical domains remains unsatisfactory. A primary limitation is the inability of LLMs to effectively leverage the external biomedical tools and databases (e.g., NCBI, Ensembl, UniProt) that experts rely on for daily workflows. Unlike general queries, biomedical problems often require consulting external data to draw reliable conclusions. Without tool integration, LLMs are prone to hallucinations and imprecise generalizations when attempting to infer biological functions or retrieve specific sequences.

Existing attempts to integrate tools into LLMs largely rely on in-context learning (ICL) with a limited set of tools. These approaches are constrained by context length limits and fail to capture the diverse, complex usage scenarios and rigorous schema constraints of scientific databases. Consequently, models struggle to map natural language questions to the specialized identifiers, parameters, and API conventions required for accurate data retrieval.

Methodology

To address these gaps, the authors introduce BioTool, a comprehensive biomedical tool-calling dataset designed for instruction fine-tuning. The dataset construction follows a multi-stage pipeline:

1. Tool Selection: The authors curated 34 frequently used tools from three authoritative databases: NCBI, Ensembl, and UniProt. These tools span 124 unique API endpoints covering variation, genomics, proteomics, evolution, and general biology.
2. API Call Synthesis: Official documentation was used to generate diverse combinations of API parameters. These synthesized calls were executed and filtered to remove cases with unavailable or uninformative responses, resulting in 3,829 unique, valid API calls.
3. Query Generation: Cutting-edge reasoning models were prompted with the API calls and their corresponding observations to generate realistic user queries.
4. Filtering and Verification:
   • Necessity Check: Chain-of-Thought prompting was used to ensure the generated query strictly required the API observation to be answered, avoiding explicit tool references.
   • Sufficiency Check: An LLM-based judge evaluated whether the observation contained relevant facts to support the query.
   • Human Refinement: Bioinformatics experts reviewed the dataset to remove low-quality queries, refine phrasing, and ensure biological terminology accuracy.
5. Final Dataset: The process yielded 7,040 high-quality, human-verified query–API call pairs.

Key Contributions

• BioTool Dataset: The creation of a large-scale, diverse dataset specifically for biomedical tool calling, bridging the gap between general tool-calling datasets and the rigorous requirements of biomedical research.
• Fine-Tuning Paradigm: Demonstrating that instruction fine-tuning on BioTool allows smaller open-source models to outperform massive proprietary models in tool-calling tasks.
• Evaluation Framework: The introduction of the BioTool Score, a metric based on semantic embedding similarity between ground-truth observations and model predictions, alongside Exact Match (EM) and API Success (AS) rates.

Experimental Results

The authors evaluated the dataset by fine-tuning open-source LLMs (ranging from 4B to 8B parameters) and comparing them against cutting-edge commercial models (GPT-5.1, Gemini 3 Pro, Claude 4.5 Sonnet) using in-context learning.

• Tool-Calling Performance: A 4-billion-parameter model (Qwen3-4B) fine-tuned on BioTool achieved an overall BioTool score of 93.6, significantly outperforming the best commercial model (Claude 4.5 Sonnet, 81.4) and GPT-5.1 (55.4). This represents a 15.0% improvement over the strongest proprietary model.
• Precision and Robustness: Fine-tuned models showed drastically higher Exact Match (EM) scores (e.g., 42.4% for Qwen3-4B vs. 6.5% for Claude 4.5 Sonnet), indicating superior ability to adhere to strict API parameter schemas.
• Answer Quality: Human evaluations on 1,048 test queries revealed that augmenting a base LLM (GPT-5.1) with BioTool-fine-tuned tool callers improved normalized answer quality by 69% compared to the base model without tools. When augmented with oracle (ground-truth) tool calls, the improvement was 88.4%.
• Error Analysis: Proprietary models exhibited pervasive failures in semantic parameter mapping (e.g., using "goat" instead of "capra_hircus"). BioTool fine-tuning substantially mitigated these errors, reducing parameter-level failure rates across all trained variants.

Significance and Claims

The paper claims that BioTool effectively enhances the biomedical capabilities of LLMs by providing high-density training signals that general-purpose pre-training and in-context learning cannot replicate. The results suggest that:

1. Domain-Specific Fine-Tuning is Critical: Specialized training on high-quality tool-calling data is necessary to navigate the technical constraints and precise parameter mappings of biological repositories.
2. Efficiency: Smaller models fine-tuned on BioTool can surpass much larger proprietary models in specialized tool-calling tasks, offering a more efficient path to reliable biomedical agents.
3. Reliability: Integrating a BioTool-fine-tuned tool caller significantly reduces hallucinations and imprecise generalizations, leading to verifiable, scientifically accurate answers.

The authors conclude that while the current framework focuses on single-hop tool calling, BioTool lays the groundwork for developing reliable, tool-augmented biomedical agents capable of supporting complex scientific discovery.