💬 NLP

Mining Subscenario Refactoring Opportunities in Behaviour-Driven Software Test Suites: ML Classifiers and LLM-Judge Baselines

This paper presents an automated pipeline that uses paraphrase-robust clustering and an XGBoost classifier to identify, rank, and categorize refactoring opportunities in Behaviour-Driven Development test suites, demonstrating that the classifier significantly outperforms both rule-based and Large Language Model baselines in detecting extractable step subsequences across a large corpus of Gherkin files.

Original authors: Ali Hassaan Mughal, Noor Fatima, Muhammad Bilal

Published 2026-05-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Hassaan Mughal, Noor Fatima, Muhammad Bilal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to organize a massive, chaotic library where every book is a software test written in a special language called Gherkin. These tests tell a story: "Given this, When that, Then this."

Over time, the librarians (developers) have written thousands of these stories. But they've made a huge mistake: they keep copying and pasting the same paragraphs over and over again. Sometimes they change a word here or there ("click the button" vs. "press the button"), but the meaning is exactly the same. This makes the library bloated, hard to read, and a nightmare to update. If you need to change how a button works, you have to find and fix it in hundreds of different places.

This paper is about building a smart robot librarian that can find these repetitive paragraphs, figure out which ones are worth cleaning up, and tell you exactly how to fix them.

Here is how the paper breaks down, using simple analogies:

1. The Problem: The "Copy-Paste" Mess

In the past, researchers could only spot if a whole story (a full test scenario) was a duplicate. But the real mess happens in the middle of the stories.

The Analogy: Imagine two novels where the first three chapters are identical, but the rest is different. A simple scanner might miss this because the whole book isn't a copy.
The Paper's Solution: They looked at small chunks of text called "slices" (groups of 2 to 18 steps in a row). They found over 5 million of these slices in 339 different software projects.

2. The "Paraphrase" Challenge

The robot librarian can't just look for exact text matches. Developers often rephrase things.

The Analogy: If one person writes "The user logs in" and another writes "The customer signs in," a simple search engine sees two different things. But a human knows they mean the same thing.
The Paper's Solution: They used a special AI (called SBERT) that understands the meaning behind the words, not just the spelling. It groups together slices that mean the same thing, even if the words are different. This is like grouping all synonyms together so the librarian sees the pattern clearly.

3. The Three Ways to Fix the Mess

Once the robot finds a repetitive slice, it has to decide how to fix it. The paper identifies three specific "tools" for the job, depending on where the repetition happens:

Tool A: The "Background" (Within a single file)
- Analogy: If every story in a specific book starts with the same three sentences, you don't rewrite them in every chapter. You write them once at the very top of the book as a "Background" note.
- When to use: When the same steps repeat inside one single file.
Tool B: The "Reusable Chapter" (Within one project)
- Analogy: If a specific sequence of events happens in 50 different books in the same library, you write that sequence once in a "Reusable Chapter" book. Then, in the other 50 books, you just write "See Reusable Chapter 4."
- When to use: When the same steps repeat across different files in the same software project.
Tool C: The "Universal Command" (Across different companies)
- Analogy: If you find that almost every library in the world uses the exact same phrase for "Log in," you create a universal dictionary entry for it. Any library can just say "Use Universal Log In."
- When to use: When the same steps repeat across totally different software projects owned by different people or companies.

4. The "Smart Brain" (Machine Learning vs. LLMs)

The authors had to teach the robot which repetitive slices are actually worth fixing. Not every repetition is useful; some are just boring, trivial things (like "status 200" which just means "it worked").

The Training: They hired three human experts to look at 200 random slices and decide: "Is this worth fixing?" and "Which tool (A, B, or C) should we use?"
The Contest: They trained a smart computer model (called XGBoost) to learn from these humans. Then, they challenged it against two other "AI judges" (Large Language Models, or LLMs).
The Result: The XGBoost model was the clear winner.
- The Human Expert (XGBoost): Got it right 89% of the time.
- The AI Judges (LLMs): Got it right only 73% and 59% of the time.
- Why? The LLMs were too cautious. They often said "No, don't fix this" even when it was a good idea, whereas the specialized model learned exactly what the humans were looking for.

5. The Big Findings

After running this robot on the massive library of 1.1 million test steps, here is what they found:

Repetition is everywhere: About 75% of the test files have repetitive chunks that could be cleaned up using Tool A (Background).
Cross-file repetition is common: About 60% of projects have chunks that could be cleaned up using Tool B (Reusable Chapter).
Cross-company repetition is rare but real: Only about 12% of projects have chunks that are so universal they could be shared across different companies (Tool C).
The "Same Owner" Trap: They noticed that many "cross-company" repetitions were actually just one company (like DataDog) publishing many different versions of their software in different languages. The robot learned to ignore these, because they aren't truly "shared" across different organizations.

6. The Takeaway

This paper provides a blueprint and a tool to automatically find the "copy-paste" mess in software tests.

It doesn't just say "Hey, there's a duplicate."
It says: "Here is the duplicate, here is why it's worth fixing, and here is the exact code change you need to make to fix it."

The authors released all their code, data, and the "rulebook" they used for the humans, so anyone else can use this robot to clean up their own software libraries. They proved that a specialized, trained model is better at this specific job than the general-purpose AI chatbots we often hear about today.

Technical Summary: Mining Subscenario Refactoring Opportunities in Behaviour-Driven Software Test Suites

Problem Statement
Behaviour-Driven Development (BDD) test suites, written in Gherkin, accumulate maintenance hazards similar to unit-test corpora, including copy-paste setup, parameter-varying near-duplicates, and silent drift. While prior work has identified duplication at the single-step level or catalogued "smells" at the whole-scenario level, a critical gap remains: the practical unit of refactoring is rarely an isolated step but a contiguous run of two or more steps (a "slice"). Existing sequence mining algorithms (e.g., PrefixSpan, SPADE) operate on exact symbol sequences and fail to handle the surface-paraphrase variations common in BDD steps. Furthermore, while three concrete refactoring mechanisms exist (within-file Background, within-repo reusable scenarios, and cross-organisational shared higher-level steps), no prior work automates the identification of which recurring subsequences are worth extracting or maps them to the appropriate mechanism.

Methodology
The authors present a static, paraphrase-robust subsequence miner applied to a corpus of 339 GitHub repositories (276 distinct upstream owners) containing 1.1 million Gherkin steps. The pipeline is organized into three main stages:

Mining and Embedding (Phases 0–4):
- Slice Extraction: The system extracts all contiguous $L$ -step windows ( $L \in [2, 18]$ ) from scenarios, generating 5,382,249 slices.
- Paraphrase Robustness: Instead of raw text matching, slices are keyed by "hybrid cluster identifiers" derived from the cukereuse pipeline. To recover paraphrase-equivalent slices missed by exact cluster-ID matching, the authors employ a Sentence-BERT (SBERT) embedding pass, reduced via UMAP and clustered using HDBSCAN. This yields 33,121 paraphrase-equivalence clusters.
- Scope Definition: Recurrence is measured across three nested scopes:
  - Within-file (RQ1): Candidates for a Background block.
  - Within-repo cross-file (RQ2): Candidates for a reusable scenario invoked via I call feature file.
  - Cross-organisational (RQ3): Candidates for a custom higher-level step, defined by recurrence across distinct upstream owners (not just repositories, to avoid inflating counts from single owners with multi-language SDKs).
Human Labelling (Phase 5):
- A stratified pool of 200 slices was labelled by three authors against a written rubric.
- The rubric defines a four-category "extraction-worthy" label (yes, no, uncertain, flagged-spec) and a mechanism label.
- Inter-rater agreement (Fleiss' $\kappa$ ) was 0.56 for extraction-worthiness and 0.79 for mechanism selection.
Modelling and Baselines (Phases 6–8):
- Classifier: An XGBoost binary classifier was trained to predict "extraction-worthiness" using features such as slice length ( $L$ ), support counts, outlier fractions, and scope signals.
- Mechanism Predictor: A secondary head predicts the specific refactoring mechanism.
- Baselines: The classifier was compared against a tuned rule-based baseline and two open-weight Large Language Model (LLM) judges (gpt-oss-120b and ling-2.6-1t).

Key Results

Corpus Scale: The miner identified 692,020 distinct recurring patterns. Of these, 30,955 recur across $\ge 2$ distinct upstream owners.
Classifier Performance: The XGBoost classifier achieved an out-of-fold F1 score of 0.891 (95% CI [0.852, 0.927]). This significantly outperformed the rule-based baseline (F1 = 0.836, $p=0.017$ ) and both LLM judges (best LLM F1 = 0.728, $p < 10^{-4}$ ).
Prevalence: After applying the classifier gate:
- 75.0% of scenarios contain a within-file Background candidate.
- 59.5% contain a within-repo reusable-scenario candidate.
- 11.7% contain a cross-organisational shared-step candidate.
LLM Limitations: While LLMs were highly precise on the "yes" class, they were conservative in recall, under-calling extraction-worthiness. Their performance on the mechanism selection task (conditional on "yes") was high ( $\ge 0.95$ accuracy), but they failed to match the classifier on the primary extraction decision.
Filtering: A six-rule verification filter chain (R1–R6) reduced the candidate set from 464,073 to 84,564 patterns, primarily by collapsing nested representations and removing trivial content.

Significance and Contributions
The paper claims to provide the first paraphrase-robust subsequence miner for BDD specifications that operates at corpus scale. Its primary contributions are:

A Slice Inventory: A ranked list of 5.4 million contiguous step subsequences collapsed into 692,020 recurring patterns, mapped to three specific reuse mechanisms with published Cucumber-Java implementations.
Automated Discovery: A pipeline that automates the identification of which slices are worth extracting, addressing the "limiting cost" of manual target identification in large-scale BDD adoption.
Empirical Evidence: A demonstration that a specialized ML classifier significantly outperforms both rule-based heuristics and off-the-shelf LLM judges in determining refactoring suitability for BDD test suites.
Reproducibility: All artifacts, including the mining scripts, the labelled pool, the rubric, and the trained classifiers, are released under the Apache-2.0 license.

The authors position this work as the "which" component in a three-paper arc: Mughal (2024) provided the "how" (mechanisms), Mughal et al. (2026) provided the "how much" (step-level duplication scale), and this paper provides the "which" (per-slice extraction decisions). The paper concludes that while recurring structure is pervasive, the cross-organisational signal is dominated by infrastructural duplication (e.g., HTTP request/response idioms) rather than complex business logic, suggesting the most immediate value lies in short, frequently repeated macros.