📊 statistics

Benchmarking machine learning against a regulatory rule-based baseline for drinking-water compliance classification

This study demonstrates that for drinking-water compliance classification, machine learning models cannot outperform a regulatory rule-based baseline because domain-defined thresholds already capture nearly all available signal, highlighting that refining regulatory rules and implementing consensus strategies are more effective than model selection.

Original authors: Aljaž Sovič, Barbara Hribar Lee, Zoran Bosnić

Published 2026-10-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aljaž Sovič, Barbara Hribar Lee, Zoran Bosnić

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every drop of water that flows from a tap is the result of a constant, invisible vigilance. Before it reaches a glass, it has been tested against a strict set of rules designed to keep it safe for human consumption. These rules are not suggestions; they are legal limits. If a sample of water contains even a single chemical or biological marker above a specific threshold, the entire sample is considered unsafe. For decades, checking these samples has been a straightforward, manual process: scientists measure the water, compare the numbers to the rulebook, and declare the water either compliant or non-compliant. This method is reliable because it is based on clear, written laws, but it leaves open a question that has fascinated scientists in recent years: could a computer learn to do this job better?

Machine learning, the field of computer science where programs learn from data rather than following explicit instructions, has made huge strides in many areas. It is often proposed as a way to find hidden patterns that human eyes might miss. In the world of environmental safety, the idea is that a computer could look at thousands of water measurements and predict which ones are dangerous, perhaps spotting subtle connections between different chemicals that a simple rulebook cannot see. The hope is that these smart algorithms could catch dangers earlier or more accurately than the current system. But for this promise to be real, the new system must be tested against the old one. It is not enough to say a computer is smart; it must be shown to be smarter than the established rules it is meant to replace.

A team of researchers in Slovenia decided to put this idea to the test using a massive archive of real-world data. They gathered 22,152 water samples collected over seven years from across the country. Each sample had been measured for hundreds of different things, from bacteria and metals to pesticides and physical properties like temperature and clarity. The researchers set up a challenge: could a machine learning model, trained on this data, predict whether a sample was safe or unsafe better than the official rulebook? They built four different types of learning models, ranging from simple decision trees to complex neural networks, and compared them against the standard method used by regulators. To make the test fair, they used a rigorous method where the models were trained on data from some years and tested on data from a year they had never seen before, mimicking how these systems would work in the real world.

The results were surprising and clear. The machine learning models performed very well, but they did not beat the rulebook. In fact, they matched the rulebook's performance but never surpassed it. The researchers found that the existing rules already captured almost all the information needed to make a correct decision. The most important factors for determining if water was unsafe were things like the presence of certain bacteria, which the rulebook already flagged immediately. The computer models learned these same patterns but could not find any extra "secret" signals in the data that the simple rules missed. The study showed that for this specific task, the complex algorithms offered no advantage over the straightforward, deterministic check of the regulations.

However, the story does not end with a simple "no" to machine learning. The researchers discovered that the biggest improvements came not from changing the computer model, but from fixing a flaw in the data itself. They found a specific inconsistency in how the data was labeled regarding a pesticide called metolachlor. For several years, the rules for this chemical were different than they were in the final year of the study, but the historical records had not been updated to reflect this change. This created confusion where some water samples were labeled as safe when, by the new standards, they should have been flagged as unsafe. When the researchers corrected this inconsistency in the data, the performance of the rule-based system improved dramatically. The number of errors dropped by more than half, from 19 mistakes down to just 9 across the entire dataset.

This correction was more valuable than any change to the computer models. The researchers also tried a hybrid approach, a two-step system where a sample would only be declared unsafe if both the rulebook and the computer agreed. They hoped this would combine the strengths of both methods, but it did not work as hoped. Because the computer and the rulebook were making mistakes on the exact same samples, asking for agreement only made the system more cautious without making it more accurate. The study concluded that in a field governed by strict thresholds, the best approach is to ensure the data is clean and the rules are applied correctly. While machine learning is a powerful tool, in this case, it was the careful, human understanding of the regulations and the data that made the real difference. The lesson is that before we rush to replace established systems with artificial intelligence, we must first ensure that the foundation we are building on is solid.

Technical Summary: Benchmarking Machine Learning Against a Regulatory Rule-Based Baseline for Drinking-Water Compliance Classification

Problem Formulation
The study addresses the prediction of drinking-water sample compliance with prescribed quality indicators, formulated as a binary classification problem. The objective is to classify samples as either compliant ( $y=0$ ) or non-compliant ( $y=1$ ), where non-compliance is defined by at least one measured parameter exceeding its regulatory limit. The dataset comprises 22,152 national monitoring samples from Slovenia spanning 2019–2025. The problem is characterized by strong class imbalance, with only approximately 9.4% of samples being non-compliant. The authors posit that while machine learning (ML) is frequently proposed for environmental compliance, it is rarely benchmarked against the specific regulatory rule sets it aims to replace, leaving the marginal predictive value of learned models unclear.

Methodology
The research employs a systematic comparison between a rule-based (RB) regulatory baseline and four machine-learning models: Gradient Boosting (GB), Random Forest (RF), LightGBM, and a Multi-Layer Perceptron (MLP).

Data and Features: The raw dataset included 231 parameters, reduced to 68 attributes (69 with a derived feature) after filtering for completeness and defined limit values. Features were categorized into microbiological, physical, chemical, metals, and pesticides. Missing values in non-microbiological parameters were imputed with medians, and outliers were capped at the 99th percentile.
Feature Selection: SHAP (SHapley Additive exPlanations) values were used to rank attribute importance and determine the optimal number of features ( $K$ ) for model training.
Validation Protocol: To avoid data leakage inherent in random splits for time-series data, the study utilized Leave-One-Year-Out (LOYO) cross-validation. In each of seven iterations, one year served as the test set while the remaining six years formed the training set.
Performance Metrics: Given the class imbalance, the primary evaluation metric was the macro-averaged F1 measure, supplemented by precision and recall. Statistical significance was assessed using McNemar's test, bootstrap confidence intervals, and paired t-tests.
Proposed Interventions:
1. Metolachlor Regulatory Correction: A preprocessing step addressing a labeling inconsistency arising from the 2025 tightening of regulatory criteria for metolachlor metabolites (ESA and OXA). Values for these metabolites prior to 2025 were set to missing to prevent the model from learning spurious correlations between pre-2025 measurements and compliant labels.
2. Two-Step Consensus Model: A hybrid approach where a sample is labeled non-compliant only if both the rule-based approach and a machine-learning model independently predict non-compliance (logical AND).
3. Derived Feature: The inclusion of the total pesticide concentration sum as an additional input variable.

Key Results

Model Performance: In the base scenario (without corrections), the rule-based approach achieved the highest performance ( $F1 = 0.998$ at $K=30$ ). Machine learning models (specifically GB and LightGBM) matched the rule-based baseline ( $F1 \approx 0.997$ ) but did not statistically exceed it. The MLP performed significantly worse on the full attribute set ( $F1 = 0.966$ ).
Feature Importance: SHAP analysis revealed that the signal for non-compliance is heavily concentrated in microbiological parameters (e.g., coliform bacteria, E. coli), which are the top-ranked attributes. The difference in importance between the top microbiological attribute and the first non-microbiological attribute was approximately 20-fold.
Impact of Interventions:
- Derived Features: Adding the total pesticide sum did not yield a statistically significant improvement in F1 scores for any model.
- Metolachlor Correction: This intervention proved critical. By removing the labeling inconsistency, the rule-based approach's error count dropped from 19 to 9 errors across the full dataset (a reduction of more than 50%). The correction improved the rule-based approach's F1 score to 0.9988 (at $K=40$ ), while having negligible impact on the ML models' performance.
- Two-Step Model: The consensus model did not outperform the rule-based baseline. Error analysis showed that the ML models and the rule-based approach made errors on nearly identical samples (high overlap), meaning the consensus strategy failed to exploit complementary errors. The two-step model achieved higher precision but lower recall compared to the standalone ML models.

Significance and Claims
The paper concludes that for regulatory compliance tasks where the domain is strictly defined by thresholds and the signal is concentrated in a few key indicators, machine learning models do not surpass a faithfully reconstructed rule-based baseline. The study argues that the "success" of high F1 scores (e.g., >0.99) in ML literature often merely reflects the reproduction of existing deterministic thresholds rather than the discovery of new predictive patterns.

The authors claim that the most valuable interventions were not algorithmic but domain-driven:

Data Quality over Model Complexity: Correcting regulatory labeling inconsistencies (the metolachlor correction) had a substantially greater impact on performance than the choice of model family or hyperparameter tuning.
Necessity of Baselines: A rule-based baseline is an indispensable control in regulatory monitoring. Without it, high ML performance metrics can be misleading, as they may simply replicate what the legal thresholds already achieve.
Limiting Factors: At the achieved performance levels (9 errors in 22,152 samples), the ceiling for improvement is determined by data quality and labeling inconsistencies rather than model architecture.

The protocol is presented as transferable to other threshold-governed environmental domains (e.g., air quality, wastewater limits), emphasizing that the comparison must always be anchored in the specific regulatory rules rather than generic parameter sets.

More like this