💻 computer science

Distinguishing Order Effects from Random Variation in Cleaning Pipelines for Tabular Data Classification

This paper introduces the Order Sensitivity Index and a permutation test to rigorously distinguish meaningful pipeline order effects from random variation in tabular data classification, revealing that while order impacts accuracy beyond chance in many cases, the specific sequence of balancing classes and correcting labels is critical to preventing minority-class label degradation.

Original authors: Muhammad Idrees, Zaheer Ullah, Thanh Thi Nguyen, Muhammad Taimour Hamid

Published 2026-11-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammad Idrees, Zaheer Ullah, Thanh Thi Nguyen, Muhammad Taimour Hamid

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Before a computer can learn to recognize patterns in a spreadsheet, the data it receives must be clean. Real-world records are rarely perfect; they contain gaps where information is missing, strange entries that look like errors, and labels that might be wrong. To fix these problems, data scientists use a series of steps. They might fill in the missing numbers, remove the strange entries, correct the wrong labels, or adjust the mix of categories so that rare events are not ignored. These steps are standard practice, but for a long time, the order in which they are performed was treated as a matter of habit rather than a scientific choice. It seemed logical to fix missing numbers before removing strange entries, or to correct labels before balancing the groups, but no one had rigorously tested whether this sequence actually changed the final result.

The question of order matters because each step changes the data that the next step receives. If you create new examples to balance a group before fixing wrong labels, you might accidentally copy those errors into the new examples. If you fix labels while the groups are still unbalanced, the system might misunderstand what a correct label looks like for the rare group. While some previous studies had hinted that the sequence could matter, they often relied on a few random tests that could easily produce lucky or unlucky results by chance. Without a way to distinguish a real pattern from random noise, it was impossible to know if the order truly mattered or if the best sequence was just a fluke.

A team of researchers set out to solve this problem by treating the order of cleaning steps as a variable to be tested rather than a rule to be followed. They took ten different real-world datasets, ranging from medical records to credit applications, and deliberately introduced specific errors into them, such as missing values, outliers, and flipped labels. Because they knew exactly where the errors were, they could measure the true impact of every cleaning decision. They then ran every possible arrangement of four key cleaning steps—filling gaps, removing outliers, correcting labels, and balancing groups—across three different types of learning algorithms. In total, they tested twenty-four different orders on each dataset, repeating the process many times to see how much the results varied simply due to random chance.

To separate a real effect from random luck, the researchers developed a new way of measuring the difference between the best and worst orders. They compared the actual gap in performance against a baseline gap that would be expected if the order didn't matter at all. This baseline was calculated by shuffling the results as if the steps were interchangeable, effectively asking, "How big would the difference be if we just got lucky?" They found that in many cases, the apparent advantage of one order over another was largely explained by this random variation. In fact, for more than half of the dataset and algorithm combinations, the observed difference was smaller than what chance alone would produce. This means that for many situations, the specific order of steps might not matter as much as previously thought, and the "best" order found in a single test could simply be a statistical accident.

However, the study did find that order does matter in specific, predictable circumstances. The effect became much stronger when the data contained a high amount of label noise, particularly when the errors were concentrated in the rare or minority group. The most consistent and significant finding concerned the relationship between correcting labels and balancing the groups. When the researchers corrected labels while the data was still unbalanced, the system often made mistakes, changing correct labels for the rare group into the majority label because it had seen so few examples of the rare group to begin with. By balancing the groups first, or by ensuring the correction process saw a balanced set of examples, the system preserved the correct labels for the rare group much better. This specific change in order improved the accuracy of the final model by between 0.6 and 2.8 percentage points in the cases where order truly mattered.

The researchers also tested whether a sequence that worked well on one set of data would work on another. They found that an order chosen based on performance across several datasets was significantly better than a random guess when applied to a new, unseen dataset. This suggests that while the perfect order might vary, there are general principles that hold true. The study concludes that data scientists should not rely on a single default sequence for all problems. Instead, they should test different orders against the possibility of random chance, paying special attention to the sequence of label correction and group balancing, especially when dealing with rare events or noisy data. The work provides a clear method for determining when the sequence of cleaning steps is a critical design choice and when it is merely a matter of convention.

Technical Summary: Distinguishing Order Effects from Random Variation in Cleaning Pipelines for Tabular Data Classification

Problem Statement
In tabular data classification, practitioners routinely apply a sequence of preprocessing steps—such as imputation, outlier removal, label noise correction, and class balancing—before training a model. While each step has a substantial literature, the order in which these steps are applied is often chosen by convention rather than empirical evidence. Because each step alters the data distribution received by subsequent steps, the order can significantly impact model accuracy. However, distinguishing genuine order effects from random variation (seed-to-seed variance) is difficult. Previous studies often compare orders based on average accuracy over a few train-test splits; with few splits, the observed gap between the "best" and "worst" orders may simply result from chance rather than a true structural dependency. Furthermore, existing automated pipeline search tools (e.g., TPOT, Learn2Clean) or recommendation frameworks (e.g., DIANA) do not explicitly test whether an observed order difference exceeds the gap expected when order has no effect.

Methodology
The authors propose a rigorous framework to isolate and quantify order effects, centered on the Order Sensitivity Index (OSI) and a permutation test.

Experimental Design: The study evaluates all 24 permutations of four cleaning steps: Missing Value Imputation (IMP), Outlier Removal (OUT), Label Noise Correction (LNR), and Class Balancing (BAL). These are tested across 12 binary classification datasets (10 in the main experiment) using three classifiers: Random Forest, Logistic Regression, and XGBoost.
Fault Injection: To ensure ground truth, the authors inject specific errors into the training data: 15% label flipping (concentrated on the minority class in the main condition), missing values, and outliers. The test set remains clean.
Order Sensitivity Index (OSI): Defined as the relative gap between the best and worst average pipeline performance:
$OSI = \frac{\max_{\pi} \bar{P}(\pi) - \min_{\pi} \bar{P}(\pi)}{\max_{\pi} \bar{P}(\pi)}$
where $\bar{P}(\pi)$ is the mean balanced accuracy over $S$ random seeds.
Permutation Test (Baseline): To determine if an observed OSI is statistically significant, the authors establish a baseline ( $OSI_0$ ) representing the gap expected if order had no effect. They perform a within-replicate permutation test (10,000 iterations) where pipeline labels are randomly reassigned to scores within each seed. This preserves the difficulty of the dataset and the variance between seeds while breaking the link between order and performance.
Metrics: The primary metric is the excess sensitivity ( $\Delta OSI = OSI - OSI_0$ ) and the permutation $p$ -value, corrected for false discovery rate. The study also analyzes specific interactions, such as the order of class balancing versus label correction, and uses "fold-safe" variants (SMOTE applied inside cross-validation folds) to isolate mechanisms.

Key Contributions

A Chance-Calibrated Metric: The paper introduces a procedure to judge order effects against a baseline of random variation. By using a permutation test to estimate the expected gap under the null hypothesis (no order effect), the authors avoid the pitfall of attributing random variance to pipeline structure.
Exhaustive Evaluation: The study provides a comprehensive evaluation of all 24 orders of four standard cleaning steps across 10 datasets and three classifiers, totaling over 41,000 experimental runs. It also evaluates 41 shorter subsequences to determine the necessity of specific steps.
Mechanistic Insight into Label Correction: The authors identify that the interaction between class balancing and label correction is critical. They demonstrate that training a label-correction model on imbalanced data leads to the "correction" of many valid minority-class labels. Performing class balancing before label correction (or using fold-safe SMOTE within the correction model) prevents this error.
Transferability Analysis: A leave-one-dataset-out analysis shows that an order optimized on a set of datasets transfers better to a held-out dataset than a random order, though the gains are modest.

Results

Magnitude of Effects: In the main experiment, the baseline ( $OSI_0$ ) exceeded half of the observed OSI in every dataset-classifier pair. This indicates that a significant portion of the raw gap between best and worst orders is due to chance.
Significant Order Effects: After correcting for false discovery, 15 of 30 dataset-classifier pairs showed order effects beyond chance. These effects were small but meaningful, changing balanced accuracy by 0.6 to 2.8 percentage points—sufficient to alter which pipeline a practitioner would select.
Conditions for Sensitivity: Order effects were most pronounced under conditions of high label noise and severe class imbalance (specifically with minority-anchored noise). Conversely, when label errors were distributed evenly across classes (symmetric noise), no significant order effects were detected.
The Balancing vs. Correction Interaction: The most consistent finding was the order of balancing and label correction. When label errors were concentrated in the minority class, balancing before correction (or using fold-safe SMOTE) improved balanced accuracy by 1.4 to 1.9 points compared to correcting on imbalanced data. This improvement stems from preventing the correction model from misidentifying valid minority examples as noise.
Method Robustness: The order effects were not specific to a single set of cleaning methods; similar results were observed with a second method set (using median imputation, LOF, and Edited Nearest Neighbors).

Significance and Claims
The paper modestly claims to provide a necessary statistical baseline for evaluating preprocessing pipelines. It argues that without accounting for seed-to-seed variation, researchers risk mistaking random noise for meaningful order effects. The study demonstrates that while order effects exist, they are often smaller than raw comparisons suggest and are highly dependent on data conditions (specifically label noise distribution and class imbalance).

The authors conclude that:

Order comparisons must be tested against chance to be valid.
Label correction strategies must account for class imbalance; specifically, balancing the data before or within the label correction process prevents the degradation of minority class performance.
While a single "default" order is not universally optimal, the interaction between balancing and correction is a critical design decision that practitioners should address based on the nature of their label noise.

Technical Summary: Distinguishing Order Effects from Random Variation in Cleaning Pipelines for Tabular Data Classification

More like this