💻 computer science

PICKLES: a Natural Language Framework for Requirement Specification and Model-Based Testing

This paper introduces PICKLES, a natural language framework that bridges Behaviour-Driven Development and Model-Based Testing by translating human-readable scenarios into formal models to enable automatic test generation with significantly higher coverage than standard BDD.

Original authors: María Belén Rodríguez, Petra van den Bos

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: María Belén Rodríguez, Petra van den Bos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. You have two main ways to do this:

The "Strict Engineer" Way (Model-Based Testing): You write a complex mathematical blueprint of every possible road, weather condition, and traffic light. It's incredibly precise and covers every scenario, but only a mathematician can read it. If you want to show the boss what the robot will do, you have to translate the math back into English, which is hard and prone to errors.
The "Storyteller" Way (Behavior-Driven Development): You write a story: "Given the car is on the road, When the light turns red, Then the car stops." Everyone understands this. It's easy to write and read, but it's like writing a recipe for just one specific day. If you want to test every possible traffic light timing, you have to write thousands of separate stories, which becomes a messy, unmanageable pile of paper.

The Problem:
In the real world, companies want the precision of the "Strict Engineer" but the readability of the "Storyteller." Usually, they have to pick one or the other. If they pick the story, they miss hidden bugs. If they pick the math, nobody understands the tests.

The Solution: PICKLES
The authors of this paper created a new framework called PICKLES (Precise Input and Control-flow Keyword-based Language for tEst Scenarios). Think of PICKLES as a "Universal Translator" that bridges the gap between stories and math.

Here is how it works, using a simple analogy:

1. The "Smart Recipe" (The Language)

Instead of writing a story about one specific car stopping at one specific red light, PICKLES lets you write a "Smart Recipe."

Old Way: "The car stops at the red light on Main Street." (Too specific).
PICKLES Way: "The car stops at any red light, provided the light is red and the car is moving."
The Magic: You can define rules like "The light can be any color from Red to Green" or "The car can be any speed between 0 and 100." This is called parametrization. It turns a single story into a rule that covers thousands of possibilities at once.

2. The "Automatic Architect" (The Translation)

Once you write these "Smart Recipes" in plain English (using a format similar to the popular Gherkin language), the PICKLES tool acts as an Automatic Architect.

It reads your English rules.
It instantly builds a Formal Model (a complex mathematical map) behind the scenes.
This map is so detailed that it knows every possible path the system could take, even paths you didn't explicitly write down.

3. The "Master Map" (Combining Stories)

In traditional testing, you test Story A, then Story B, then Story C, separately.

The Flaw: Story A might work, and Story B might work, but if you do A then B, the system might crash. Traditional stories often miss these "handshake" errors.
The PICKLES Fix: The tool takes all your separate stories and stitches them together into one giant "Master Map." It figures out how Story A connects to Story B, and how Story B connects to Story C. It creates a complete picture of how the system behaves over time, not just in isolated moments.

4. The "Back-Translation" (Readable Results)

This is the most clever part. The tool uses the Master Map to generate hundreds of specific test cases automatically.

Usually, these tests would come out as confusing math code.
PICKLES takes those math results and translates them back into English stories.
So, the tester gets a report that says: "Here is a specific test where the car was going 50mph and the light turned red at 2.5 seconds. The car stopped correctly."
The human can read it, understand it, and verify it, even though the heavy lifting was done by the math engine.

The Real-World Test (The Traffic System)

The authors tested this on a real traffic management system used by a company called Technolution.

The Setup: They took existing stories about traffic sensors detecting broken detectors.
The Result:
- Using the old "Storyteller" method (running stories one by one), they only tested about 30% of the possible system behaviors.
- Using PICKLES (combining the stories and using the math engine), they achieved 100% coverage of the system's logic.
- They did this without writing hundreds of new stories. They just made the original stories "smarter" by adding variables (like "any lane" or "any position") and let the tool do the rest.

Why This Matters

PICKLES allows non-technical people (like project managers or business analysts) to write requirements in plain English that are unambiguous and mathematically rigorous. It ensures that:

No one is confused: The requirements are clear to everyone.
No bugs are hidden: The math engine finds edge cases humans might miss.
Less work: You write fewer stories to test more scenarios.

In short, PICKLES is like giving a robot a storybook that is smart enough to understand every possible variation of the story, solve the math problems inside it, and then write a clear report back to you in plain English.

1. Problem Statement

The paper addresses a critical tension in industrial software testing, particularly for safety-critical systems:

The Limitation of Model-Based Testing (MBT): While MBT offers rigorous, systematic, and high-coverage testing via formal models, its adoption is hindered by the difficulty of defining and maintaining these models. They often require specialized expertise, lack readability for non-technical stakeholders, and create communication barriers.
The Limitation of Behavior-Driven Development (BDD): BDD uses human-readable, natural language scenarios (Given-When-Then) to facilitate collaboration. However, BDD relies on concrete examples, which often leads to:
- Ambiguity and Incompleteness: Natural language can be vague.
- Low Coverage: Testing isolated examples often misses edge cases and complex interaction sequences.
- Maintenance Costs: Large suites of specific test cases are expensive to maintain and update.

The Core Challenge: How to achieve the precision and high coverage of MBT while retaining the accessibility, readability, and stakeholder collaboration benefits of BDD, without forcing practitioners to choose between the two.

2. Methodology: The PICKLES Framework

The authors propose PICKLES (Precise Input and Control-flow Keyword-based Language for tEst Scenarios), a framework that bridges BDD and MBT through a bi-directional translation pipeline.

A. The Domain Specific Language (PicklesDSL)

Pickles extends the standard Gherkin syntax (Given-When-Then) with formal constructs to eliminate ambiguity:

Variable Settings: A prelude block where variables are explicitly typed (e.g., integer, boolean, array, structure) and assigned domains/ranges (e.g., [1, 3], {AV, PART_AV}).
Parametrization: Steps can reference variables and include Guard Blocks (conditions) using logical operators (AND, OR) and quantifiers (at least, at most, exactly).
Structure:
- Given: Defines preconditions and initial state guards.
- When: Defines input interactions with parameters and guards.
- Then: Defines expected outputs and validation guards.

B. Formal Semantics: Symbolic Transition Systems (STS)

The framework translates PicklesDSL specifications into Symbolic Transition Systems (STS), a formalism extending Labelled Transition Systems with data constructs.

Translation (Spec $\to$ STS):
- Each Pickles scenario maps to a partial STS.
- Given clauses become initial guards.
- When clauses become input switches (transitions).
- Then clauses become output switches.
- Variables and guards are mapped to STS parameters, location variables, and logical predicates.
Model Composition:
- Individual scenario STSs are composed into a Master Model.
- Choice Composition ( $\bowtie$ ): Merges initial states of multiple scenarios, allowing any scenario to start.
- Sequential Composition ( $\triangleright$ ): Connects the sink (end) states of one STS to the initial states of others, enabling the generation of long execution traces that combine multiple scenarios.

C. Test Generation and Reverse Translation

Test Generation: Standard MBT algorithms (specifically aiming for 100% switch coverage) are applied to the Master Model to generate formal test cases (sequences of switches with specific parameter values).
Reverse Translation (STS $\to$ Pickles): The formal test cases are translated back into PicklesDSL syntax.
- Input values are explicitly defined in the When steps.
- Output values are replaced by the original Then guards to validate the System Under Test (SUT) behavior.
- The result is an executable, human-readable test case compatible with standard BDD tools (e.g., Cucumber).

3. Key Contributions

PicklesDSL: A new Domain Specific Language that preserves the Gherkin structure but adds explicit variable typing, domain definitions, and control-flow constraints, ensuring unambiguous specifications.
Bi-directional Translation: A formal mapping between PicklesDSL and Symbolic Transition Systems (STS), allowing seamless movement between human-readable requirements and formal models.
Master Model Composition: A method to automatically compose partial STSs from individual scenarios into a unified Master Model, enabling the discovery of complex interaction bugs that isolated scenarios miss.
Prototype Implementation: A Python-based tool (using the Lark parser) that automates the translation, composition, and reverse translation processes.
Industrial Validation: Application of the framework to a real-world traffic management component from the company Technolution.

4. Results

The framework was evaluated using a case study involving a traffic management system component. The results demonstrated significant improvements over traditional BDD approaches using the same set of manually defined scenarios:

Transition Coverage:
- Traditional BDD: Executing scenarios in isolation covered only ~30% of the possible system transitions.
- Pickles: By composing scenarios and generating tests from the Master Model, coverage increased to 100%. This represents a 70 percentage point improvement.
Input Coverage:
- Traditional BDD with hardcoded values exercises a tiny fraction of possible inputs.
- Pickles, by utilizing variable domains and boundary analysis, represented 112 distinct input possibilities for a single scenario (Scenario 02) that would otherwise require hundreds of manual test cases. The study noted a 98.5% improvement in input coverage compared to concrete BDD examples.
Efficiency: Achieving 100% transition coverage manually would have required at least seven distinct test traces. Pickles achieved this with only four input scenarios, significantly reducing manual effort and maintenance costs.

5. Significance

The PICKLES framework represents a significant step forward in making Model-Based Testing accessible to industrial practitioners:

Bridging the Gap: It successfully eliminates the trade-off between readability and precision. Stakeholders can collaborate on requirements in natural language, while the system automatically ensures formal rigor.
Shift-Left Testing: By enabling the automatic composition of scenarios, it allows for integration and system-level testing to be performed earlier in the development lifecycle, detecting interaction bugs that isolated unit tests miss.
Maintainability: Changes to requirements (e.g., variable ranges) automatically propagate to all generated test cases, reducing the maintenance burden associated with large BDD suites.
Explainability: Unlike black-box formal models, the generated test cases remain in human-readable Pickles format, making it easier to explain test results to non-technical managers and auditors.

In conclusion, PICKLES demonstrates that by grounding natural language specifications in formal semantics and leveraging automated model composition, organizations can achieve the high reliability of MBT without sacrificing the collaborative benefits of BDD.