SANE Schema-aware Natural-language Evaluation of Biological Data

Technical Summary: SANE – Schema-aware Natural-language Evaluation of Biological Data

Problem Statement
High-throughput microscopy generates vast, structured datasets capturing cellular responses to pharmacological perturbations. While platforms like Cell-Profiler facilitate data processing and storage, effective data access remains a bottleneck. Traditional querying requires SQL expertise and familiarity with complex, hierarchical schemas (e.g., cell lines, drugs, concentrations, and image replicates). Although Large Language Models (LLMs) offer a natural-language alternative via Text-to-SQL, their tendency to hallucinate and generate incorrect queries raises concerns regarding reliability in scientific contexts. Furthermore, existing benchmarks (e.g., Spider, BIRD) focus on general, human-authored queries and lack the specific experimental structures found in domain-specific biomedical databases.

Methodology: The SANE Framework
The authors present SANE (Schema-Aware Natural-language Evaluation), a novel paradigm for evaluating Text-to-SQL systems specifically tailored to biological data. Unlike static benchmarks, SANE automatically generates evaluation queries directly from the underlying database schema and real experimental designs.

• System Architecture: The evaluation utilizes a few-shot LLM (4-bit quantized LlaMA 3.1) integrated into a web application to query a PostgreSQL database. The system operates without any model training or fine-tuning.
• Three-Stage Pipeline:
  1. Context Assessment: The LLM first determines if the natural language query contains sufficient context to be answered, outputting a binary label (sufficient/missing). If context is missing, the system triggers a clarification request.
  2. SQL Generation: If context is sufficient, the LLM generates a SQL statement using schema-aware prompting. This prompt injects database structure, domain terminology, filtering rules (e.g., excluding border artifacts), and specific examples.
  3. Result Interpretation: The database executes the SQL, and a second LLM pass interprets the structured result into a natural language response (text, bullet points, or tables).
• Benchmark Generation: SANE systematically derives 572 non-trivial test cases from real high-throughput drug response experiments. These queries are categorized into six groups: Simple, Complex, Errors (typos/abbreviations), Contextless (underspecified), Multi-step, and Schema queries. The framework introduces controlled perturbations (e.g., typos, missing context) to simulate realistic user interactions.
• Evaluation Metrics: Accuracy is determined by comparing the LLM's predicted context label against ground truth and, for sufficient queries, by verifying that the generated SQL execution results match the reference results (result set equivalence).

Key Contributions

1. Scalable Domain-Specific Benchmarking: A framework for generating realistic, schema-grounded test queries from high-throughput biological data, moving beyond general-purpose benchmarks.
2. Large-Scale Evaluation: A comprehensive evaluation comprising 572 automatically generated queries derived from actual experimental structures.
3. Robustness Analysis: A detailed failure mode analysis identifying ambiguity in natural language as the primary source of errors, rather than inherent SQL generation failures.

Results
The few-shot LLM system achieved an overall accuracy of 97.2% across the 572 queries without any training or fine-tuning.

• High-Performance Categories: Simple (100%), Complex (98.4%), and Schema (96.9%) queries demonstrated high reliability.
• Lower-Performance Categories: Contextless (81.8%), Errors (86.1%), and Multi-step (85.7%) queries showed slightly lower accuracy.
• Failure Analysis: Of the 16 failures, 10 were due to incorrect prediction of the "missing context" label (often due to unknown synonyms or misinterpreted underspecified queries). Only 5 failures involved slightly incorrect SQL generation (e.g., omitting identifiers), and one was a generic dismissal due to misclassification.
• Zero-Shot Baseline: Without few-shot examples, relying solely on schema injection, accuracy dropped significantly to 29.9%, highlighting the critical role of domain-specific prompting.

Significance and Claims
The paper claims that SANE demonstrates that few-shot LLMs can provide reliable database access in well-defined, constrained domains when combined with schema-aware prompting and guardrails. The primary significance lies in:

• Eliminating Barriers: The approach removes the need for SQL expertise and model training, allowing researchers to directly interrogate experimental results.
• Reliability in Constrained Domains: The results suggest that for specific scientific applications, the risk of hallucination can be mitigated through structured prompting and context validation, making LLMs viable as research assistants.
• Future Direction: The authors argue that future improvements should prioritize interactive disambiguation mechanisms to handle ambiguous user inputs, rather than focusing on model fine-tuning, as the model's core SQL generation capability is already robust when the input is clear.