NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation
This paper introduces NL2SHACL-Bench, a novel benchmark suite designed to evaluate the translation of natural language requirements into SHACL shapes, revealing that while current large language models can generate syntactically valid SHACL, they still struggle with producing semantically equivalent constraints for complex logical and structural patterns.
Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine the internet as a giant, chaotic library where every book, photo, and fact is stored as a tiny, interconnected card. This is called a "Knowledge Graph." But libraries need rules to keep things organized; otherwise, you might find a recipe for soup listed under "Car Engines" or a date for a meeting that hasn't happened yet. To keep these digital libraries tidy, computer scientists use a special set of rules called SHACL. Think of SHACL as the strict librarian who checks every card to make sure it fits the right category and follows the correct format.
However, writing these rules is incredibly hard. It requires knowing both the messy, real-world topic (like chemistry or healthcare) and the complex, technical language of the computer rules. Most experts know their field but don't speak "computer librarian." This creates a bottleneck: we have all this data, but we can't easily tell the computer what to check. The big question is: Can we just tell a computer, in plain English, what the rules should be, and have it write the code for us? This is the challenge of turning "Natural Language" into "SHACL."
Enter NL2SHACL-Bench, a new toolkit created by researchers at the Technical University of Munich to test if modern AI can solve this problem. Think of this paper as the "driver's license test" for Artificial Intelligence in the world of data rules. The researchers built a massive practice exam containing 240 different scenarios, ranging from chemical data to government invoices. They asked four of the smartest AI models available today to read a plain English description of a rule and write the corresponding SHACL code.
The results were a mix of "wow" and "not quite yet." The paper finds that these AI models are surprisingly good at the basics. They can almost always write code that looks correct and follows the grammar rules of the language (like a student who knows how to spell and use punctuation). In fact, one model got a perfect score on these basic checks. However, when the rules got tricky—like dealing with complex "if this, then that" logic or specific paths through the data—the AI started to stumble. It often wrote code that looked right but actually meant something different, or it invented new, fake rules that didn't exist in the official language.
The researchers conclude that while AI is ready to be a helpful assistant for simple tasks, it isn't quite ready to replace human experts for complex data validation yet. The paper doesn't claim the problem is solved; instead, it provides the first standardized way to measure exactly where AI is failing, so developers can build better tools in the future. It's a crucial step toward a future where anyone can describe their data rules in English, and a computer will handle the rest, but for now, we still need to double-check the AI's homework.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.