💬 NLP

BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases

The paper introduces BiomedSQL, the first benchmark designed to evaluate scientific reasoning in text-to-SQL generation over a real-world biomedical knowledge base, revealing significant performance gaps between current large language models and expert baselines in handling complex domain-specific criteria.

Original authors: Mathew J. Koretsky, Maya Willey, Owen Bianchi, Chelsea X. Alvarado, Tanay Nayak, Nicole Kuznetsov, Sungwon Kim, Mike A. Nalls, Daniel Khashabi, Faraz Faghri

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mathew J. Koretsky, Maya Willey, Owen Bianchi, Chelsea X. Alvarado, Tanay Nayak, Nicole Kuznetsov, Sungwon Kim, Mike A. Nalls, Daniel Khashabi, Faraz Faghri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Asking a Librarian a Very Specific Question

Imagine you walk into the world's largest library. This library doesn't just have books; it has billions of pages of medical data: gene maps, drug trial results, and patient records.

You want to find a specific answer, like: "Which genetic mutations make Parkinson's disease worse?"

In the old days, you'd have to hire a team of expert librarians (biomedical scientists) to manually search through the stacks, cross-reference tables, and do the math for you. It's slow, expensive, and hard to scale.

Now, we have AI (Large Language Models). We want to teach the AI to be that librarian. You type your question in plain English, and the AI instantly writes a computer command (called SQL) to fetch the answer from the database.

The Problem: Current AI is great at writing simple commands like "Show me all books by Shakespeare." But when you ask it complex medical questions, it often fails. It doesn't understand the hidden rules of science. It might forget to check if a drug is actually approved, or it might miss the strict statistical rules scientists use to decide if a result is real or just a fluke.

Enter BiomedSQL: The "Final Exam" for Medical AI

The authors of this paper created BiomedSQL, which is essentially a massive, super-hard final exam designed specifically to test how well AI can do medical research.

Here is what makes it special:

It's Not Just Translation: Most AI tests just check if the AI can translate English into computer code. BiomedSQL checks if the AI can think like a scientist.
- Analogy: Imagine a test where you ask a student, "Who is the tallest person in the room?" A basic AI just looks for the word "tallest." A smart AI knows it needs to measure everyone, ignore people standing on chairs, and only count adults. BiomedSQL forces the AI to know that "significant" in science means a specific math number (like a p-value less than 0.00000005), not just "important."
The Dataset: They built a giant, real-world database using Google's BigQuery. It contains:
- Gene-Disease links: How genes connect to diseases like Alzheimer's.
- Drug info: Which drugs target which genes and if they are approved by the FDA.
- Clinical Trials: The status of drug testing.
- They generated 68,000 questions based on real scientific queries.
The "Gold Standard" Baseline: Before testing the AI, they had real human experts answer these questions. The experts got it right 90% of the time. This sets the bar high.

The Results: The AI is Trying Hard, But Still Stumbling

The researchers tested many of the smartest AI models available (like GPT-4, Gemini, and Claude) on this exam. Here is what happened:

The Score: The best AI model (Gemini-3-Pro) only got 58% of the answers right.
The Gap: There is a huge gap between the AI (58%) and the human experts (90%).
The Custom Agent: The authors built their own specialized AI assistant called BMSQL. It acts like a detective that doesn't just guess once; it checks its work, fixes errors, and asks for clarification. This boosted the score to 62.6%, but it still couldn't catch up to the human experts.

Why did the AI struggle?
The paper found that the AI made three main types of mistakes:

Wrong Bookshelf: It looked in the wrong database table (e.g., looking for drug info in a gene table).
Missing the Fine Print: It forgot the strict scientific rules (like "only show results with a p-value under 5e-8").
Bad Math: It messed up the complex filtering logic needed to combine different pieces of data.

Why This Matters

Think of BiomedSQL as a stress test for the future of medical AI.

Current State: If we let AI run medical research today, it might give us answers that look right but are scientifically wrong. It's like a GPS that takes you to the right city but drops you off in the middle of a lake because it didn't understand the road rules.
The Goal: This benchmark helps researchers see exactly where AI is failing so they can build better tools.
The Future: The ultimate goal is to create an AI that acts as a "super-assistant" for scientists. Imagine a researcher asking, "Find me a drug that targets this specific gene and has passed Phase 3 trials," and getting a perfect, verified answer in seconds. That would accelerate cures for diseases like Alzheimer's and Parkinson's.

The Takeaway

The paper concludes that while AI is amazing at writing code, it still lacks the deep scientific intuition required to navigate the complex, rule-heavy world of biomedical research. BiomedSQL is the tool we need to bridge that gap, ensuring that when AI helps us discover new cures, it does so with the same rigor and safety as a human expert.

1. Problem Statement

While text-to-SQL systems have advanced significantly in general domains (e.g., SPIDER, BIRD) and clinical EHR retrieval (e.g., EHRSQL), they fail to address the specific requirements of biomedical scientific discovery.

The Gap: Current systems treat query generation as a syntactic translation task. However, biomedical questions often require implicit domain reasoning that is not explicitly encoded in the database schema.
Specific Challenges:
- Implicit Thresholds: Questions like "What SNPs are significantly associated with Alzheimer's?" require the model to infer statistical thresholds (e.g., $p < 5 \times 10^{-8}$ for GWAS) without them being stated in the prompt.
- Multi-hop Reasoning: Queries often require chaining logic across multiple tables (e.g., linking drugs to genes, then genes to diseases, then filtering by trial phase).
- Contextual Knowledge: Models must understand concepts like "drug approval status" or "causal inference" which rely on external scientific conventions rather than just column names.
Current Limitations: Existing benchmarks do not evaluate the ability to operationalize these scientific conventions, leading to a performance gap where models generate syntactically correct but scientifically invalid SQL.

2. Methodology

A. Dataset Construction: BiomedSQL

The authors introduce BiomedSQL, the first benchmark designed specifically to evaluate scientific reasoning in text-to-SQL.

Scale: 68,227 question/SQL/answer triples.
Source Data: A harmonized Google BigQuery database integrating:
- Gene-Disease Associations: OpenTargets Platform and ChEMBL.
- GWAS Summary Statistics: Alzheimer's (21M rows) and Parkinson's disease (7.8M rows) data.
- Causal Inference: SMR (Summary-data-based Mendelian Randomization) data from omicsSynth.
- Drug Records: FDA approval status, trial phases, and licensing data.
Generation Process:
1. Gold Standard Creation: Domain experts authored 40 unique "seed" questions and verified SQL queries.
2. Template Expansion: These seeds were programmatically expanded using entity substitution (diseases, genes, SNPs) to create the full dataset.
3. Ground Truth: Every query was executed on the BigQuery database to generate ground-truth execution results for evaluation.

B. Evaluation Framework

The authors evaluated a wide range of models (Open-source: LLaMA-3.1, Qwen-2.5-Coder; Closed-source: GPT-4o/o3-mini/5.2, Gemini-2.0/3-Pro, Claude-3.7/4.5) across different paradigms:

Isolated Generation: Single-turn prompts with varying few-shot examples and domain-specific instructions (e.g., statistical thresholds).
Interaction Paradigms:
- ReAct: Iterative reasoning with tool use.
- Schema Indexing: Dynamic retrieval of relevant schema parts.
- DAIL-SQL: Retrieval of similar example queries.
- BMSQL (Custom Agent): A multi-step agent that performs:
  1. Table/Column selection.
  2. Initial SQL generation.
  3. Syntax error correction.
  4. Threshold refinement (applying scientific rules like p-value cutoffs).
  5. Natural language response generation.

C. Metrics

Execution Accuracy (EX): Measures if the LLM's query returns the exact same set of rows/UUIDs as the ground truth.
Jaccard Index (JAC): Measures set overlap (more lenient than EX).
Syntax Error Rate (SER): Percentage of non-executable queries.
BioScore (RQR & SR): An LLM-as-a-judge metric (validated against domain experts) to evaluate the quality and safety of the final natural language answer.

3. Key Contributions

BiomedSQL Benchmark: A large-scale (68k+), domain-specific benchmark that shifts focus from schema translation to scientific reasoning (thresholds, multi-hop logic, ontology resolution).
Infrastructure: Release of a harmonized BigQuery database, gold-standard SQL annotations, and an open-source evaluation toolkit.
Comprehensive Evaluation: A rigorous assessment of state-of-the-art LLMs, revealing that even top models struggle with implicit domain constraints.
BMSQL Agent: A custom multi-step agent that outperforms single-turn baselines by explicitly handling error correction and domain rule application.

4. Key Results

Performance Gap: There is a substantial gap between current LLMs and human experts.
- Expert Baseline: 90.0% Execution Accuracy.
- Best LLM (Gemini-3-Pro): 58.1% Execution Accuracy.
- Best Custom Agent (BMSQL + GPT-o3-mini): 62.6% Execution Accuracy.
- Conclusion: Even the best systems lag experts by ~27-30%.
Model Performance:
- Qwen-2.5-Coder-32B showed strong performance among open-weight models (40.8% EX), outperforming larger parameter models like Llama-3.1-405B.
- Gemini-3-Pro and Claude-4.5-Opus were the top proprietary performers.
Impact of Techniques:
- Few-Shot Prompting: 10-shot examples provided the best gains; adding more (up to 40) yielded diminishing returns.
- Interaction Paradigms:
  - Schema Indexing improved Safety Rate (abstention) but hurt accuracy.
  - ReAct offered marginal improvements.
  - BMSQL was the most effective, particularly in applying statistical thresholds and refining queries based on execution feedback.
- Inference-Time Compute: Increasing reasoning steps (1-pass to 3-pass) in BMSQL yielded only marginal gains in accuracy (62.6% $\to$ 61.7%), suggesting the bottleneck is reasoning logic, not just search depth.
Error Analysis:
- The most common errors were Incorrect Table Selection and Missing/Incorrect Statistical Thresholds (e.g., failing to apply $p < 5e-8$ ).
- Models struggled significantly with Join, Multi-Filter, and Similarity Search operations.
Scalability: When the schema size was doubled (10 to 20 tables), performance dropped by 7-10% for baseline models, but only 2-4% for BMSQL, highlighting the robustness of the multi-step agent approach.

5. Significance and Future Directions

Scientific Impact: BiomedSQL demonstrates that current text-to-SQL systems are not yet ready for autonomous scientific discovery in biomedicine. They lack the ability to internalize domain-specific conventions (like statistical significance) without explicit prompting.
Research Direction: The paper argues for moving beyond syntactic translation toward reasoning-aware agents that can infer implicit constraints and iteratively refine queries.
Accessibility: By releasing the dataset and code, the authors aim to democratize access to biomedical knowledge bases and provide a rigorous testbed for developing trustworthy AI for scientific research.
Limitations: The dataset relies on templated questions (though the authors argue this does not trivialize the task due to the complexity of the reasoning required) and uses BigQuery (a cloud-specific dialect), which may limit direct comparison with SQLite-based benchmarks.

In summary, BiomedSQL establishes that while LLMs are powerful, they currently lack the "common sense" of biomedical domain experts required to translate qualitative scientific questions into executable, statistically valid database queries.