💻 computer science

STMutants: A Mutation Testing Dataset for Structured Text Programs in Industrial Automation

This paper introduces STMutants, the first publicly available mutation testing dataset for IEC 61131-3 Structured Text programs in industrial automation, which contains 108 screened mutants to enable reproducible research on test-suite effectiveness and AI-assisted quality assurance for safety-critical PLC software.

Original authors: Md Humaun Kabir, Md Rakibul Islam, Helen H. Lou

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md Humaun Kabir, Md Rakibul Islam, Helen H. Lou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a quality inspector at a factory. The machines on the assembly line are controlled by special computers called PLCs (Programmable Logic Controllers). These machines run on a specific coding language called Structured Text (ST). If the code has a tiny mistake, the whole factory could stop, or worse, a machine could break and hurt someone.

For a long time, researchers had a great way to test if their safety checks were good enough for regular computer programs (like apps on your phone), but they didn't have a standard "practice test" for these industrial factory machines. It was like trying to teach a student to drive a semi-truck without ever letting them sit in one.

This paper introduces STMutants, a new "practice test" designed specifically for these factory machines. Here is how it works, explained simply:

1. The "Fake Mistake" Game (Mutation Testing)

To see if a safety test is good, you don't just look at the code; you try to break it.

The Analogy: Imagine a teacher gives a student a math test. To see if the student really knows the answers, the teacher secretly changes one number in the problem (e.g., changing a "5" to a "6") and sees if the student still gets the answer right.
In the Paper: The researchers took 11 real-world factory programs and created 110 "fake mistakes" (called mutants). They changed tiny things, like flipping a switch from "On" to "Off," changing a "greater than" sign to a "less than" sign, or swapping a plus sign for a minus sign.
The Result: They cleaned up the list to 108 real, tricky mistakes that a computer could actually run. This is the STMutants dataset. It's the first time anyone has made a public, standardized list of these specific "fake mistakes" for factory code.

2. The "AI Student" Exam

Once they had the list of fake mistakes, the researchers wanted to see if modern Artificial Intelligence (AI) could act as a safety inspector. They asked three different AI models (think of them as three very smart students: GPT-5.2, Gemini 2.5, and Claude Sonnet 4.5) to do two things:

Write a Test: Create a checklist of inputs to see if the code works.
Find the Mistakes: Use that checklist to see if they could spot the 108 fake mistakes they planted earlier.

3. The Results: Who Passed?

The AI models did surprisingly well, but they weren't perfect.

The Winner: Gemini 2.5 was the best student, catching 94.4% of the fake mistakes.
The Runners-Up: GPT-5.2 and Claude Sonnet 4.5 tied, catching 86.1% of the mistakes.
The Easy vs. Hard Stuff:
- Easy: The AI was great at spotting simple math errors or wrong numbers (like changing a "5" to a "6").
- Hard: The AI struggled with time-based puzzles. One specific program, called SEQUENCE 8, was like a complex dance routine where the steps depend on what happened in the previous step. The AI got confused because it couldn't "remember" the sequence of events over time. It missed a lot of mistakes in this specific program.

4. Why This Matters

Before this paper, researchers had no standard way to compare different tools for testing factory code. It was like everyone using their own different ruler to measure a table.

STMutants is now the standard ruler.
It proves that AI can help write safety checks for factory machines, but it also shows that AI still needs help with complex, time-sensitive tasks.
The paper doesn't claim AI is ready to replace human engineers yet, but it provides the first solid "scorecard" to help researchers build better, safer tools for the future.

In a nutshell: The authors built a "practice exam" of fake errors for industrial machine code and used it to test three AIs. The AIs were good at finding simple errors but got tripped up by complex, time-based logic. This dataset is now available for anyone to use to build better safety tools for factories.

Technical Summary: STMutants

Problem Statement
Mutation testing is a established technique for evaluating test suite effectiveness, grounded in the Competent Programmer Hypothesis and the Coupling Effect. While widely applied to mainstream languages like C, Java, and Python, there is a critical gap in the domain of Industrial Automation. Specifically, IEC 61131-3 Structured Text (ST) programs, which control safety-critical Programmable Logic Controllers (PLCs), lack a publicly available, reproducible benchmark dataset for mutation-based research. The absence of such a dataset hinders the systematic assessment of automated testing tools and emerging Large Language Model (LLM) capabilities for PLC software, where faults can lead to equipment damage, production loss, or physical harm.

Methodology
The authors present STMutants, a curated dataset constructed through a rigorous four-phase methodology:

Fault-Type Profiling and Operator Selection: Eleven ST programs were selected from the OSCAT basic library and academic sources (Koziolek et al.), covering diverse functional categories (Logic, Control, Signal, etc.) and ranging from 38 to 211 lines of code. Seven mutation operator categories were defined and adapted from classical taxonomies to fit the ST domain:
- Value Replacement (VRO): Modifying constants (numeric, boolean, time).
- Comparison/Relational Operator (CRO): Altering boundary conditions (e.g., < to ≤).
- Arithmetic Replacement (ARO): Changing arithmetic operators (e.g., + to -).
- Negation Insertion/Omission (NIO): Inverting boolean logic.
- Logical Operator (LRO): Swapping boolean connectives (e.g., AND to OR).
- Arithmetic Operation Insertion/Omission (AOIO): Adding or removing arithmetic steps.
- Initialization (IO): Changing variable default values.
Syntactic Transformation: A total of 110 first-order mutants (10 per program) were generated by applying single atomic token substitutions.
Compilability Verification: All mutants were compiled using MATIEC, a standard IEC 61131-3 compliant environment, to ensure syntactic and type-level validity.
Equivalence Screening: A two-step process eliminated equivalent mutants (those that do not alter program behavior). This included static pre-screening for dead code and redundant assignments, followed by an independent dual-review by two experts. Inter-rater agreement was strong ( $\kappa = 0.87$ ). Two mutants from the FLOW METER program were excluded due to being test-unobservable, resulting in a final dataset of 108 retained mutants.

Key Contributions

First Public Benchmark: STMutants is the first publicly available mutation testing dataset specifically for IEC 61131-3 Structured Text programs.
Structured Taxonomy: It introduces a domain-specific set of seven mutation operators tailored to the syntactic and semantic characteristics of PLC software.
Reproducible Baseline: The dataset includes the original programs, the generated mutants, and the mutation descriptions, enabling reproducible research in automated test generation and fault analysis.
Baseline Evaluation: The authors provide the first empirical baseline for LLM-based mutation analysis in this domain, evaluating three models (GPT-5.2, Gemini 2.5, and Claude Sonnet 4.5) on a two-phase task: generating unit test suites and predicting mutation kill/survive outcomes.

Experimental Results
The study evaluated the ability of LLM-generated test suites to detect injected faults:

Overall Performance: Across the 108 retained mutants, the models achieved mutation detection accuracies of 86.1% (GPT-5.2), 94.4% (Gemini 2.5), and 86.1% (Claude Sonnet 4.5).
Statistical Significance: Cochran's Q test confirmed significant performance differences ( $p=0.017$ ). Post-hoc analysis revealed that Gemini 2.5 significantly outperformed the other two models, while GPT-5.2 and Claude Sonnet 4.5 performed equivalently.
Operator-Specific Findings:
- Models achieved perfect or near-perfect kill rates on Arithmetic (ARO, AOIO) and Initialization (IO) faults.
- Value Replacement (VRO) and Comparison (CRO) were the most prevalent fault types (70% of the dataset) and were generally well-handled, though VRO remained the most challenging category overall.
- Temporal Reasoning Limitations: Performance degraded significantly on the SEQUENCE 8 program (a complex 8-bit sequencer with 211 LOC). While GPT-5.2 achieved 50% kill rate, Gemini 2.5 and Claude Sonnet 4.5 dropped to 10%. The authors attribute this to the models' difficulty in simulating multi-cycle state transitions required to propagate faults in deeply nested temporal logic.

Significance and Claims
The paper claims that STMutants fills a critical void in industrial automation research by providing a standardized, reproducible benchmark for evaluating test adequacy in PLC software. The results demonstrate that while current LLMs are effective at generating test suites for value-based and relational faults in ST programs, they currently lack the reliability required for complex, state-dependent temporal reasoning. The dataset serves as a foundation for future research in automated test generation, mutation analysis, fault localization, and the development of AI-assisted quality assurance systems for safety-critical industrial control software. The authors emphasize that the dataset is designed to support rigorous, comparable evaluation rather than to claim that current AI solutions are fully ready for deployment in all safety-critical scenarios.

1. The "Fake Mistake" Game (Mutation Testing)

2. The "AI Student" Exam

3. The Results: Who Passed?

4. Why This Matters

Technical Summary: STMutants

More like this