Intelligent Test Case Scheduling Using Multi-Criteria Decision Making for Human–Machine Interface Testing​

In the factories that build the machines powering modern industry, a critical moment arrives before any product leaves the assembly line: the final inspection. This is not a casual check of the paint job, but a rigorous, high-stakes verification of the software that allows a human to talk to a machine. These systems, known as human-machine interfaces, must work perfectly every time, yet they are tested under a relentless constraint: time. The production line cannot stop for hours to run every possible check, and the budget for testing is fixed and non-negotiable. If a critical failure slips through, the consequences can be severe, yet if the testing takes too long, the factory loses money. For decades, engineers have solved this dilemma by relying on their own experience, picking a subset of tests to run based on gut feeling and old checklists. This approach is inconsistent; one engineer might catch a dangerous flaw that another misses, and the process changes every time a new product version arrives. The core challenge is to decide which tests are absolutely necessary and which can be skipped without risking safety, turning a subjective guess into a reliable, data-driven decision.

A team of researchers at Sichuan Technology and Business University has tackled this problem by developing a new way to schedule these tests, moving away from human intuition toward a mathematical scoring system. They call their method the Time-Weighted Utility Scoring Model. Instead of asking an engineer to decide whether to run a test or skip it, the system automatically calculates a score for every single test case in the library. This score is not a simple pass or fail; it is a complex value derived from three specific factors. First, the system looks at the risk: how likely is this specific function to fail based on its history? Second, it weighs the importance: how critical is this function to the safety and operation of the machine? Third, it measures the cost: how much time and computing power does it take to run this specific test? By combining these three elements, the model generates a single number that represents the overall value of running that test right now.

The researchers applied this model to an industrial human-machine interface system and found that it could dramatically shrink the time required for testing without sacrificing safety. In their experiments, running the full set of tests took 425.9 seconds. By using their new scoring system to select only the most valuable tests, they reduced the total time to 174.9 seconds, a reduction of nearly 59 percent. Crucially, this speed did not come at the cost of missing errors. The system detected 100 percent of the critical failures that the full test suite would have found. It achieved this by identifying a small group of high-priority tests that were mandatory for safety and then using a smart sampling method for the rest, ensuring that the most likely failures were caught early in the process.

The heart of the system is its ability to learn and adapt. The scores are not static; they are updated after every round of testing. If a test case has passed successfully many times in the past, its risk score drops, and the system may admit it for execution with a lower probability. Conversely, if a test case has a history of failing, its risk score rises, and the system prioritizes it to ensure the problem is caught. For cases scoring above a predefined threshold, the system deterministically includes them as mandatory tests, while the remainder are admitted by probabilistic sampling proportional to their scores. This creates a feedback loop where the testing strategy evolves with the product. The researchers validated this approach by having five different senior engineers independently rate the importance and risk of the tests. They found that while the engineers' individual opinions varied slightly, the system's final results remained stable and consistent across different groups of data. This proves that the method does not rely on the specific judgment of one person but produces a reliable outcome regardless of who is doing the scoring.

One of the most significant findings is that the system does not just pick the cheapest tests or the ones with the most historical errors. It balances all three factors simultaneously. When the researchers tested the system by looking at only one factor at a time—such as picking tests only based on cost or only based on risk—the results were far worse. The system that considered all three factors together was able to find failures much faster and with fewer tests than any single-factor approach. In their trials, the new method required running only about 23 specific test cases to reach a 100 percent detection rate of failures within the specific experimental group of 1,200 executions, whereas other methods required significantly more tests to achieve the same level of confidence. This efficiency means that factories can verify the safety of their products in less than half the time, freeing up resources for production while maintaining a zero-tolerance standard for critical defects.

The study demonstrates that replacing ad-hoc, experience-based decisions with a documented, score-based procedure is not just possible but superior for safety-critical industrial software. The researchers showed that by treating test selection as a quality-risk decision rather than a simple efficiency concern, they could guarantee the verification of critical functions while maximizing the number of faults found per unit of time. The results were consistent across multiple independent groups of data, suggesting that the method is robust and reproducible. By turning the chaotic process of test selection into a systematic, resource-aware plan, this approach offers a new standard for factory acceptance testing, ensuring that the machines powering our world are safe, reliable, and ready for the market without wasting precious time.