From Law to Gherkin: A Human-Centred Quasi-Experiment on the Quality of LLM-Generated Behavioural Specifications from Food-Safety Regulations

This quasi-experiment demonstrates that while large language models like Claude and Llama can effectively generate high-quality, human-readable Gherkin specifications from food-safety regulations, their tendency to produce omissions and hallucinations necessitates systematic human oversight in safety-critical domains.

The Gist

Imagine you are a chef trying to build a robot that cooks food. But before you can program the robot, you have to read a 50-page government rulebook about food safety. These rules are written in "legalese"—dry, complex, and designed to apply to any technology, not just your specific robot.

Translating those rules into instructions the robot can understand is like trying to turn a thick, dusty encyclopedia into a simple, step-by-step recipe card. Doing this by hand is slow, boring, and easy to mess up. If you miss a step, the robot might serve you a poisonous meal, and you could get in huge trouble.

This paper is about testing a new "smart assistant" (AI) to see if it can do this translation job for us.

Here is the story of their experiment, broken down simply:

1. The Goal: The "Translator" Test

The researchers wanted to see if Large Language Models (LLMs)—the same kind of AI that powers chatbots like Claude and Llama—could read food safety laws and automatically write "Gherkin" specifications.

• What is Gherkin? Think of Gherkin as a special "recipe card" language. Instead of writing code, you write simple sentences like: "Given the egg is frozen, When I check the bacteria, Then it must be under 50,000."
• Why Gherkin? It's a bridge. It's written in plain English so humans can understand it, but it's structured enough that computers can read it to run automatic tests.

2. The Experiment: The "Taste Test"

The researchers didn't just ask the AI to do it; they set up a rigorous "taste test" with human experts.

• The Ingredients: They took 30 real food safety laws (like rules about how much water is allowed in dried eggs).
• The Chefs: They hired two AI chefs: Claude and Llama. Each AI tried to turn the 30 laws into 60 different Gherkin "recipe cards."
• The Judges: They recruited 10 human judges (mostly computer science students who know how to write these recipe cards).
• The Task: Each judge looked at 12 of the AI-generated cards and rated them on five things:
  1. Relevance: Did it actually talk about the law?
  2. Clarity: Was it easy to understand?
  3. Completeness: Did it miss any steps?
  4. Singularity: Did it focus on just one thing per card, or did it mix everything up?
  5. Time Savings: Would this save me time if I used it?

3. The Results: The AI is a Great "Draftsman"

The results were surprisingly good, but with a catch.

• The Good News: The AI did an amazing job. The human judges gave the AI-generated cards very high scores.

  • 95% were relevant.
  • 100% were clear.
  • 94% were complete.
  • The judges felt it would save them a massive amount of time.
  • The Verdict: Both AI chefs (Claude and Llama) performed almost exactly the same. Neither was clearly "better" than the other.

• The Bad News (The "Hallucinations"): Even though the scores were high, the human judges found some dangerous errors when they looked closely.

  • The "Ghost" Ingredients (Hallucinations): Sometimes the AI invented rules that didn't exist. For example, one AI wrote a rule saying the robot should "display a warning light" if the temperature was wrong. But the original law never mentioned a warning light! The AI just made it up because it thought that's what a robot should do.
  • The "Missing" Ingredients (Omissions): Sometimes the AI forgot a crucial step. It might forget to mention that a label must be in both English and French, even though the law required it.
  • The "Mixed-Up" Recipe (Singularity): Sometimes the AI tried to do too much in one step. Instead of one card for "checking weight" and another for "checking bacteria," it mashed them together into one confusing card.

4. The Conclusion: AI is a Co-Pilot, Not the Captain

The paper concludes that AI is fantastic at writing the first draft, but it cannot be trusted to finish the job alone.

• The Analogy: Think of the AI as a very fast, very creative intern. The intern can write a 10-page report in 10 seconds that looks perfect. But if you don't read it carefully, the intern might have invented a fake statistic or forgotten a key paragraph.
• The Takeaway: In safety-critical fields (like food safety, medicine, or aviation), you cannot just let the AI run the show. You must have a human expert review every single line to catch the "ghost ingredients" and "missing steps."

In short: The AI can do the heavy lifting and save us hours of work, but we still need to be the editors to make sure the final product is safe and accurate.

Technical Summary

Here is a detailed technical summary of the paper "From Law to Gherkin: A Human-Centred Quasi-Experiment on the Quality of LLM-Generated Behavioural Specifications from Food-Safety Regulations."

1. Problem Statement

In regulated domains like food safety, software systems must comply with complex legal provisions. However, translating technology-neutral legal texts into concrete, machine-readable software artifacts (such as behavioral specifications) is currently a manual, labor-intensive, and error-prone process.

• The Gap: Legal texts are often ambiguous and broad, making it difficult for software engineers to derive precise acceptance criteria or test cases.
• The Opportunity: Large Language Models (LLMs) offer the potential to automate the derivation of structured specifications from legal texts.
• The Challenge: It is unknown whether current LLMs can accurately translate legal provisions into Gherkin (a domain-specific language for Behavior-Driven Development) without introducing critical errors like hallucinations, omissions, or mixed-intent scenarios, which could lead to non-compliance in safety-critical systems.

2. Methodology

The authors conducted a quasi-experimental human-subject study to evaluate the quality of Gherkin specifications generated by two state-of-the-art LLMs: Claude 3.7 Sonnet and Llama 3.3 70B Instruct.

• Data Source: 30 legal provisions from the Safe Food for Canadians Regulations (SFCR) and related food safety instruments.
• Participants: 10 participants with software engineering backgrounds (ranging from senior undergraduates to PhD students) proficient in BDD and Gherkin.
• Procedure:
  1. Generation: The LLMs were prompted using a structured template (including a domain glossary) to convert the 30 legal provisions into Gherkin specifications (Given-When-Then format). This resulted in 60 total specifications (30 per model).
  2. Evaluation: Each participant independently assessed 12 specifications (6 from each model) based on five quality criteria:
     • Relevance: Alignment with the legal text.
     • Clarity: Unambiguity of the specification.
     • Completeness: Inclusion of all implied functions/characteristics.
     • Singularity: Whether a scenario focuses on a single atomic objective (avoiding mixed purposes).
     • Time Savings: Perceived reduction in manual authoring effort.
  3. Design: Each specification was rated by two participants (120 total assessments). The study used non-parametric statistical tests (Mann-Whitney U, Wilcoxon signed-rank) and Benjamini-Hochberg correction for multiple comparisons.

3. Key Contributions

• First Systematic Human Evaluation: This is the first study to systematically evaluate LLMs' ability to derive Gherkin behavioral specifications directly from legal texts.
• Domain-Specific Application: It addresses the high-stakes domain of food safety, bridging the gap between technology-neutral regulations and executable software requirements.
• Quality Framework: The study operationalizes a specific set of quality metrics (Relevance, Clarity, Completeness, Singularity, Time Savings) tailored for legal-to-code translation.
• Empirical Evidence on Failure Modes: It provides concrete qualitative data on specific failure patterns (hallucinations, omissions, mixed intents) that persist even in high-performing models.

4. Key Results

The quantitative and qualitative results yielded the following insights:

Quantitative Findings:

• High Overall Quality: Ratings were uniformly high. Across 120 assessments:
  • Relevance: 95% rated as "Mostly" or "Fully" relevant.
  • Clarity: 100% rated as "Mostly" or "Fully" clear.
  • Completeness: 94.2% rated as "Mostly" or "Fully" complete.
  • Singularity: 93.4% rated as "Mostly" or "Fully" singular.
  • Time Savings: 91.7% reported significant or maximum time savings.
• Model Comparison: Statistical analysis (Mann-Whitney U) found no statistically significant differences between Claude and Llama across any of the five criteria.
  • Descriptive nuance: Llama slightly outperformed in clarity, completeness, and time savings, while Claude showed slightly better consistency in maintaining scenario singularity (fewer mixed-purpose scenarios).
• Plausibility: Only 3 unique specifications were deemed "implausible" (violating physical laws or common sense) by participants.

Qualitative Findings (Failure Modes):
Despite high scores, qualitative feedback identified critical issues:

• Omissions: The LLMs occasionally missed specific clauses, conditions, or mandatory terms (e.g., missing bilingual labeling requirements).
• Hallucinations: Models sometimes generated content not present in the source text (e.g., assuming a system must display a warning message when the law only required a measurement).
• Mixed Intents: In complex provisions, models sometimes combined multiple distinct obligations into a single scenario, violating the "singularity" principle of BDD.

5. Significance and Implications

• Human-in-the-Loop is Essential: The study concludes that while LLMs are effective "first-draft" co-authors that can significantly reduce manual effort, they cannot be used for hands-off automation in safety-critical domains. Systematic human review is mandatory to catch omissions and hallucinations that could lead to regulatory non-compliance or safety hazards.
• Workflow Shift: The role of practitioners shifts from authoring specifications from scratch to reviewing and refining LLM-generated drafts.
• Future Research: The authors suggest future work should focus on:
  • Developing prompt engineering strategies and glossaries to reduce omissions.
  • Creating "critique-refine" loops where humans and AI iteratively improve specifications.
  • Extending the study to other legal domains (e.g., privacy, data protection) and more complex, abstract legal texts.

Conclusion: LLMs can successfully translate food safety regulations into high-quality Gherkin specifications, but their output must be treated as a draft requiring rigorous human validation, particularly regarding completeness and factual accuracy.