PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems
The paper introduces PIPE-Cypher, an automated pipeline that transforms live enterprise property graphs and real-world user queries into executable, diverse, and balanced Text-to-Cypher benchmarks to enable repeatable and deployment-relevant evaluation of graph query systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you work for a massive bank or a complex logistics company. Your company has a giant, living map of data (a "property graph") showing how money moves, who knows whom, and where goods travel. This map is unique to your company; it uses its own special names, rules, and secret codes that no one else uses.
Now, imagine you want to build a smart assistant (an AI) that lets your employees ask questions in plain English, like "Show me all accounts that transferred money to a blocked user last week," and have the AI automatically write the complex computer code (called Cypher) needed to get that answer.
The problem? You can't just test this AI on public, generic maps. It needs to be tested on your specific map, with your specific rules. But building a test for this is a nightmare:
- The map changes every day.
- The AI might write code that looks right but asks the wrong question.
- You can't send your secret data to a public AI service to generate the test.
PIPE-Cypher is the solution the authors built. Think of it as an automated "Test Kitchen" that lives entirely inside your company's secure computer.
How the "Test Kitchen" Works
Instead of a human writing thousands of test questions, PIPE-Cypher is a pipeline (a factory line) that does the heavy lifting:
The Map Inspector (Schema Profiling):
First, the system quietly looks at your company's data map. It learns the names of the nodes (like "Account" or "Person"), the roads between them (like "Transfers To"), and the specific values used (like "Credit Card" or "Blocked"). It creates a "menu" of what is possible.The Reverse Engineer (Reverse Grounding):
This is the clever part. Instead of guessing a question and hoping the answer exists, the system starts with the answer. It runs safe, read-only queries to find real data points that actually exist.- Analogy: Imagine you want to test a chef. Instead of asking them to "make a soup" and hoping they have ingredients, you first check the pantry to see you have carrots and onions. Then you ask, "Make a soup with carrots and onions." This guarantees the question is answerable.
The Strict Editor (Constrained Generation):
A local AI (running on your own servers, not the cloud) writes the English question and the Cypher code based on those real ingredients. But it's not allowed to be creative in dangerous ways. It must follow strict rules: "Only read data, never delete it," "Use exact names," and "Don't invent roads that don't exist."The Safety Gatekeeper (Deterministic Validation):
Before the test is accepted, a non-AI computer program checks the code. It acts like a bouncer at a club:- Is the code safe? (No deleting data).
- Does it use real names? (No hallucinated terms).
- Does it actually run? (If the code returns no results, it's rejected).
- Does it go the right way? (In graphs, "A to B" is different from "B to A").
The Judge (LLM Reviewer):
Finally, a second local AI acts as a judge. It looks at the question, the code, and the actual results. It asks: "Does this code actually answer the question? Is it clear?" If the code runs but gives the wrong answer, the judge rejects it.
The Results: A "Living" Benchmark
The authors used this pipeline to create a massive test set of 3,000 questions using real-world financial and social network data.
- It's Discriminating: When they tested standard AI models on this new benchmark, most failed miserably (getting less than 4% of answers right). This proves that just because an AI can write code that looks correct, doesn't mean it understands your specific data map.
- It's Refreshable: Because the pipeline is automated, if your company adds a new type of data (like "Crypto Wallets") next month, you can just run the pipeline again to generate new tests instantly. You don't have to wait for a public dataset to be updated.
- It's Private: Everything happens on your own computers. No sensitive data ever leaves your building.
The Big Takeaway
The paper argues that for enterprise AI, static test sets are dead. You can't evaluate a system designed for your unique, changing business on a generic, frozen dataset.
PIPE-Cypher changes the game by turning benchmark creation into a repeatable, automated factory process. It ensures that the tests are:
- Real: Based on actual data that exists.
- Safe: Guaranteed not to break your database.
- Honest: Strictly checking if the AI actually solves the problem, not just if it writes pretty code.
In short, PIPE-Cypher is a tool that lets companies build their own, high-quality "driving tests" for their AI assistants, ensuring they can actually navigate the complex, unique roads of the company's data without crashing.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.