Technical Summary: Text2GraphQuery-Bench

Problem Statement

Graph Database Management Systems (GDBMSs) are critical for domains characterized by complex relationships, such as fraud detection, social network analysis, and supply chain optimization. However, accessing these systems is difficult for non-experts due to the diversity of graph query languages (e.g., Cypher, GQL, SQL/PGQ) and their departure from the unified standards of relational SQL. While Large Language Models (LLMs) have successfully lowered barriers for Text-to-SQL tasks, the field of Text-to-Graph-Query lacks a standardized, comprehensive benchmark. Existing resources suffer from three primary limitations:

1. Limited Language Coverage: Most benchmarks focus on a single language (typically Cypher), neglecting emerging international standards like GQL and SQL/PGQ.
2. Rigid Synthesis Pipelines: Current data generation relies on fixed templates and narrow domain coverage, failing to produce structurally complex queries with genuine business semantics or to scale to diverse industrial scenarios.
3. Inadequate Evaluation Protocols: Existing metrics often rely solely on execution accuracy, ignoring grammar validity and semantic alignment. Furthermore, they fail to account for graph-native complexities (e.g., path traversal, reachability) and practical variations in user input, such as different abstraction levels and schema aliasing.

Methodology

The authors present Text2GraphQuery-Bench, a unified benchmark designed to address these gaps through a flexible, four-stage construction framework and a comprehensive evaluation protocol.

1. Benchmark Construction Framework

The framework standardizes raw resources into executable (Question, Graph Query) pairs across three languages: Cypher, ISO GQL, and ISO SQL/PGQ.

• Data Sources: The dataset combines existing resources (e.g., Neo4j Text2Cypher, FinBench, BIRD) with newly synthesized industrial-domain datasets.
• Schema & Data Generation: Heterogeneous sources are unified via a canonical intermediate representation, SchemaGraph. For synthetic domains, an LLM generates schema specifications and executable Python scripts to create realistic graph data with controlled referential integrity and distributions.
• Query Translation & Generation:
  • Translation: Existing SQL/Cypher queries are migrated to GQL and SQL/PGQ using a Graph-IR (Intermediate Representation) to preserve semantic structure.
  • Evolutionary Generation: For synthetic data, an iterative mechanism executes seed queries on a GDBMS, samples results, and prompts the LLM to combine them into more complex, multi-hop queries with genuine business semantics.
• Question Abstraction & Aliasing: Questions are annotated at three hierarchical levels: Syntactic (explicit primitives), Logical (entity-relation logic), and Business (high-level intent). Additionally, schema element names are replaced with semantically similar aliases to test robustness.

2. Evaluation Protocol

The benchmark employs a multi-metric protocol across three dimensions:

• Metrics: Execution Accuracy (EX), Grammar validity, GLEU (semantic alignment), and Jaro-Winkler Similarity (stylistic alignment).
• Difficulty Taxonomy: Queries are categorized into four graph-native levels: Easy (single node/edge), Medium (one-hop, simple aggregation), Hard (multi-hop, non-nested aggregation), and Extra Hard (complex paths, nested aggregation, multi-step MATCH).
• Robustness Testing: Evaluation includes variations in question abstraction levels and schema aliasing.

Key Contributions

1. First Multi-Language Benchmark: Text2GraphQuery-Bench is the first benchmark covering all mainstream declarative property graph languages (Cypher, GQL, SQL/PGQ), containing 267,276 (Question, Query) pairs across 34 databases and 13 domains.
2. Flexible Construction Pipeline: The framework overcomes the rigidity of prior work by supporting both adaptation from heterogeneous resources and domain-aware synthesis with evolutionary query generation, enabling rapid extension to new languages via Graph-IR.
3. Comprehensive Evaluation: The protocol jointly reports grammar, semantic alignment, and execution accuracy under diverse difficulty, abstraction, and vocabulary settings, moving beyond single-metric evaluation.
4. Empirical Insights: Experiments on 8 LLMs (ranging from 8B to frontier models) reveal critical insights into the capabilities and limitations of current models in this domain.

Experimental Results

The evaluation of 8 LLMs (including Claude Opus 4.8, Qwen3.7-Max, GPT-5.5, and fine-tuned open-weight models) yields the following findings:

• Significant Language Gap: Zero-shot performance on GQL and SQL/PGQ is substantially lower than on Cypher. For instance, zero-shot GQL Grammar for Qwen3.7-Max is 0.672 compared to 0.948 for Cypher. However, few-shot prompting largely recovers this gap, bringing Grammar scores above 0.90 for most models.
• Supervision vs. Capacity: Fine-tuning an 8B model (Qwen3-8B) on GQL and SQL/PGQ allows it to reach or exceed the zero-shot performance of much larger proprietary models on both Grammar and Execution Accuracy. Specifically, the fine-tuned model achieves 90.8% GQL Grammar (vs. 70.0% for Claude Opus 4.8) and 45.1% GQL EX (approaching the 48.4% of Gemini-3.5-Flash). On SQL/PGQ, the fine-tuned model reaches 92.1% Grammar (vs. 76.1% for Claude Opus 4.8) and 40.1% EX, which surpasses the zero-shot performance of several few-shot large models (e.g., 20.9% for Qwen3.7-Max and 2.9% for Claude Opus 4.8). This indicates that unfamiliarity with the language, rather than model capacity, is the primary barrier.
• Shifting Bottlenecks: As supervision increases, error profiles shift:
  • Zero-shot: Dominated by syntax errors.
  • Few-shot: Syntax errors recede; the bottleneck shifts to schema linking (mapping natural language to schema elements).
  • Fine-tuning: Syntax errors are largely suppressed. The remaining bottlenecks become language-specific: aggregation logic for GQL and schema linking/filtering for SQL/PGQ.
• Abstraction and Difficulty:
  • Higher question abstraction levels degrade execution accuracy, shifting the bottleneck from syntax to intent-to-schema grounding. Providing external knowledge significantly mitigates this drop.
  • Schema aliasing causes only a minimal average execution accuracy drop (−1.26%), suggesting strong model ability to resolve vocabulary variations.
  • Performance consistently degrades from Easy to Extra Hard. Even with few-shot prompting, Extra Hard queries remain a persistent bottleneck, requiring structural reasoning beyond in-context examples.

Significance

The paper positions Text2GraphQuery-Bench as both a rigorous benchmark and an extensible foundation for systematic Text-to-Graph-Query research. By covering multiple language standards and introducing a flexible, domain-aware synthesis pipeline, it addresses the critical lack of standardized evaluation in the field. The findings suggest that while current models struggle with the syntax of emerging standards like GQL and SQL/PGQ, these challenges are surmountable through targeted supervision (few-shot or fine-tuning). However, the persistent difficulty with complex, nested structural reasoning highlights that future progress requires advancements in handling graph-native logical depth, not just language adaptation.