📄 other

A Biomedical Benchmark and Modular RAG Framework for Text2Cypher

This paper introduces bio2C, a FAIR-compliant benchmark of 2,500 curated natural language-Cypher pairs, and a modular Retrieval-Augmented Generation framework that significantly improves Text2Cypher accuracy for biomedical knowledge graphs by leveraging retrieved examples over schema-based prompting.

Original authors: Emanuele Cavalleri, Nada Bou Kanaan, Marco Mesiti

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emanuele Cavalleri, Nada Bou Kanaan, Marco Mesiti

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, bustling library where every book, author, and character is connected by invisible threads. This isn't just any library; it's the library of life itself, holding the secrets of our genes, diseases, and medicines. Scientists call this a "Biomedical Knowledge Graph." It's like a giant, living map where a "Gene" node might be tied to a "Disease" node, which is tied to a "Drug" node. The problem is, this library doesn't have a friendly librarian who speaks your language. To ask a question like, "Which drugs might help this specific disease?" you have to speak a secret, code-like language called "Cypher." It's like trying to order a pizza by writing a computer program instead of just saying, "I want a pepperoni." Most doctors and researchers don't speak this code, so they can't easily explore the library's treasures.

Enter the heroes of this story: Large Language Models (LLMs). You might know them as the super-smart AI chatbots that can write stories or answer trivia. Scientists have been trying to teach these AIs to act as translators, turning our plain English questions into the secret Cypher code the library understands. This is called "Text2Cypher." But here's the catch: teaching an AI to do this in the medical world has been like trying to teach a dog to play chess using a broken instruction manual. We didn't have enough good examples, and the tests we used to check the AI's work were often too simple or made up by computers, missing the messy, complex reality of real medical questions. Without a good map and a fair test, we couldn't be sure if the AI was actually learning or just guessing.

This paper introduces a brand new, high-quality map and a rigorous testing ground to fix that. The researchers created bio2C, a benchmark consisting of 2,500 carefully hand-crafted pairs of natural language questions and their correct Cypher answers. Unlike previous attempts that relied on computer-generated templates, these questions were written to reflect how real scientists actually think, covering everything from simple lookups to complex, multi-step investigations involving genes, diseases, and biological processes. They organized these questions into five levels of difficulty, ranging from finding a single item to navigating a three-step path through the graph and performing advanced calculations.

To see how well different AI strategies work, the team built a modular framework to test various "prompting" techniques. They compared three main approaches:

Schema-only: Giving the AI a blueprint of the library's structure but no examples.
Retrieval-Augmented Generation (RAG): Giving the AI a few similar examples of questions and answers from the bio2C collection to learn from.
Hybrid: Combining the blueprint with the examples, and sometimes even showing the AI the results of those example queries.

They tested these methods using both powerful, proprietary AI models (like GPT-4) and open-source ones (like LLaMA), with and without extra training (fine-tuning) on their new benchmark.

The results were clear and surprising. The paper found that simply showing the AI a blueprint of the database structure wasn't enough. Instead, the most effective strategy was retrieving representative examples. When the AI was shown similar, real-world examples of questions and answers (RAG), it performed significantly better than when it only had the structural blueprint. In fact, for the best-performing configuration, this approach improved the accuracy of the generated queries by 31.6 percentage points compared to the standard schema-based methods.

Interestingly, the paper suggests that for models that had been fine-tuned (trained specifically on the bio2C data), seeing similar examples was so helpful that they didn't even need the structural blueprint to perform at their peak. However, for models that hadn't been fine-tuned, combining the blueprint with the examples (S+RAG) worked best. The study also revealed that this method works particularly well for complex, multi-step questions (like 3-hop queries), where the examples help the AI understand the "shape" of the problem better than a static description of the database could.

The researchers also tested their system on a different, smaller dataset and with questions written by an independent biologist who had never seen their training data. The system held up well, suggesting that these improvements aren't just a trick to memorize the test questions but actually help the AI understand how to navigate complex biomedical data in the real world. The paper concludes that having a high-quality, diverse, and human-curated benchmark is more important than just having a huge number of examples, and that showing an AI the right kind of examples is the key to unlocking its ability to translate human curiosity into machine-readable answers.

Technical Summary: A Biomedical Benchmark and Modular RAG Framework for Text2Cypher

Problem Statement

Biomedical Knowledge Graphs (bioKGs) are essential for integrating heterogeneous data such as genomic interactions and drug-target relationships. While property graph databases like Neo4j offer efficient storage and querying via the Cypher language, the requirement for technical expertise in writing Cypher queries limits accessibility for domain experts. Although Large Language Models (LLMs) show promise in translating Natural Language (NL) questions into Cypher (Text2Cypher), progress in the biomedical domain is hindered by a lack of representative benchmarks and reproducible evaluation frameworks. Existing benchmarks are often template-generated, lack diversity in query complexity, or contain insufficient curated data to support robust training and evaluation. Furthermore, current Text2Cypher approaches struggle with the specific challenges of property graphs, which require inferring graph traversals, relationship directions, and complex constructs (e.g., variable-length paths, aggregation) that differ significantly from Text2SQL or Text2SPARQL tasks.

Methodology

The bio2C Benchmark

The authors introduce bio2C, a FAIR-compliant biomedical benchmark comprising 2,500 manually curated NL-Cypher pairs. Unlike previous resources that prioritize quantity, bio2C focuses on representativeness and annotation quality.

Source Data: The benchmark is grounded in a view of RNA-KG, a large bioKG containing approximately 99,000 nodes and 1.6 million relationships involving miRNAs, genes, diseases, phenotypes, and Gene Ontology annotations.
Structure: The 2,500 pairs are organized into five complexity levels (500 pairs each):
1. Node: Single-node retrieval with logical conditions.
2. 1-hop: Queries on relationships between node pairs.
3. 2-hop: Traversals involving three nodes and two relationships.
4. 3-hop: Deeper traversals involving four nodes and three relationships.
5. Advanced: Queries utilizing complex Cypher constructs (WITH, REDUCE, CASE, aggregation, ranking).
Curation Process: Each instance was developed by identifying graph patterns, implementing syntactically correct and biologically relevant Cypher queries, and formulating NL questions that reflect realistic biomedical information needs rather than template-generated expressions. All queries were validated against the RNA-KG view to ensure semantic correctness.

Modular RAG Framework

To systematically evaluate retrieval-augmented strategies, the authors designed a modular framework comprising six prompt configurations. These pipelines incrementally combine three sources of contextual information:

Graph Schema ( $G_S$ ): Property graph schema descriptions (using the Enhanced Schema paradigm).
Retrieved Examples: Semantically similar NL-Cypher pairs retrieved from the bio2C vector database (RAG).
Execution Outputs: The results of executing the retrieved Cypher queries ( $R(Q_C)$ ).

The six evaluated pipelines are:

Vanilla: NL question only.
Schema: NL question + Schema.
RAG: NL question + Retrieved examples.
RAG+O: NL question + Retrieved examples + Execution outputs.
S+RAG: Schema + Retrieved examples.
S+RAG+O: Schema + Retrieved examples + Execution outputs.

Experimental Setup

The framework was evaluated using proprietary models (GPT-4o, GPT-4o-mini) and an open-source model (LLaMA-3.1-8B), alongside their fine-tuned (FT) counterparts trained on bio2C. Performance was measured using:

Jaro-Winkler (JW): Syntactic similarity.
Jaccard (Jac): Semantic similarity of execution outputs.
Coverage (Cov): The fraction of ground truth results returned by the generated query (Execution Accuracy).
Pass: The indicator function where Coverage equals 1 (complete semantic coverage).

Key Results

Performance of RAG Pipelines

Retrieval vs. Schema: Retrieving representative NL-Cypher examples (RAG) consistently outperformed schema-only prompting across all models and metrics.
Fine-Tuning vs. Prompt Engineering: While fine-tuned models generally achieved higher accuracy, rich prompt engineering configurations (specifically S+RAG+O) significantly narrowed the performance gap with fine-tuned models for base models. For instance, GPT-4o with S+RAG+O achieved a Coverage of 76.9% and Pass of 83.1%, outperforming the fine-tuned GPT-4oFT under vanilla prompting (76.1% and 80.0%).
Best Configuration: The best-performing configuration overall was GPT-4oFT with RAG, which achieved a Pass score of 89.3%. Notably, for fine-tuned models, the RAG pipeline alone often outperformed the combination of schema and examples, suggesting that fine-tuned models can infer schema constraints directly from examples.
Impact of Execution Outputs: Including execution outputs (RAG+O, S+RAG+O) provided additional gains, particularly for base models, by offering semantic guidance on expected results.

Generalization and Robustness

Blind Validation: When tested on 50 independently formulated questions by a molecular biologist, the GPT-4oFT with RAG pipeline achieved a Coverage of 73.4% and Pass of 70%, demonstrating generalization to real-world user queries.
Cross-Dataset Evaluation: The framework was tested on MedT2C (a Hetionet-based dataset). Despite MedT2C having a less complex schema and fewer examples, the trend held: pipelines with retrieved examples outperformed vanilla and schema-only approaches. The best configuration (GPT-4oFT with S+RAG) achieved 88.8% Coverage.
Query Category Impact: Performance varied by query complexity. Node and 3-hop queries yielded the highest Pass scores (92%), while 1-hop and Advanced queries were lower (78%). Analysis revealed that RAG is particularly effective for structurally rich multi-hop queries because retrieved examples often share substructures with the target query, whereas 1-hop queries frequently retrieve non-conforming examples, leading to performance drops.

Error Analysis

The best-performing setting (GPT-4oFT with RAG) eliminated all Compile-time Errors (CE) and reduced Logic Errors (LE) by 91.4% compared to vanilla prompting. Remaining errors were primarily concentrated in the Advanced and 1-hop categories, often due to hallucinations or misinterpretation of complex aggregations.

Significance and Contributions

The paper claims two primary contributions to the field of biomedical Text2Cypher:

bio2C Benchmark: The first FAIR biomedical benchmark specifically designed for Text2Cypher, offering systematic coverage of real bioKG schemas and progressively complex Cypher constructs. The authors argue that the quality and diversity of the benchmark are more critical than sheer volume, as manual curation significantly improved coverage and model performance compared to automatically generated datasets.
Modular RAG Framework: A reproducible evaluation framework that isolates the contributions of schema information, retrieved examples, and execution outputs. This framework establishes strong baselines for future research and demonstrates that carefully curated retrieval examples substantially improve Cypher generation over bioKGs, often surpassing schema-based prompting strategies.

The work concludes that high-quality, curated benchmarks and representative retrieval examples are essential for developing reliable Text2Cypher systems, enabling domain experts to query complex biomedical knowledge graphs without deep technical expertise. The benchmark, source code, and evaluated pipelines are publicly available.