🔢 mathematics

Addressing Methodological Sensitivity in MCDM with a Systematic Pipeline Approach to Data Transformation Sensitivity Analysis

This paper introduces a systematic, automated pipeline leveraging Scikit-Criteria to evaluate and quantify the sensitivity of multicriteria decision-making rankings to different normalization techniques, addressing the current lack of robustness in method selection.

Original authors: Juan B. Cabral, Alvaro Roy Schachner

Published 2026-05-07

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan B. Cabral, Alvaro Roy Schachner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to decide which of your nine friends is the "best" at a specific skill, like cooking. You have a list of criteria: speed, taste, presentation, and cost. To figure out the winner, you need a scoring system.

In the world of complex decision-making (called MCDM or Multi-Criteria Decision Making), this is exactly what happens, but with things like cryptocurrencies, business strategies, or public policies instead of friends.

Here is the problem the paper addresses, explained simply:

The "Recipe" Problem

When you calculate a score, you have to "normalize" the data. Think of this as converting all your ingredients into the same unit. Do you measure flour in cups or grams? Do you measure time in minutes or seconds?

The paper points out that the choice of "unit" (normalization method) changes the winner.

If you use "Recipe A," your friend Bob might come in 1st place.
If you use "Recipe B," Bob might drop to 5th place.

The authors found that in real-world scenarios, changing the recipe can flip the ranking of 20% to 40% of the competitors. The scary part is that most people just pick a recipe because it's their favorite, what their software has by default, or what they've seen before, without checking if it's the right one.

The Solution: The "All-Recipes" Kitchen

The authors built a new tool called SKCCombinatorialPipeline.

Imagine a super-efficient kitchen robot. Instead of you picking one recipe and cooking one dish, this robot:

Takes every possible way to measure your ingredients (normalization).
Takes every possible way to combine those measurements (aggregation).
Automatically cooks every single combination at the same time.

If you have 3 ways to measure and 2 ways to combine, the robot instantly cooks 6 different dishes and serves them all to you.

How It Works (The Pipeline)

The tool uses a "pipeline" approach, which is like an assembly line:

The Filter: It cleans the data (e.g., removing friends who didn't show up).
The Scale: It converts the data (e.g., turning "minutes" into "scores").
The Judge: It calculates the final ranking.

The magic is that the robot tries every possible combination of these steps. It doesn't just guess; it explores the entire "menu" of possibilities.

The Test Drive: Cryptocurrencies

To prove it works, the authors tested this robot on a list of 9 cryptocurrencies (like Bitcoin, Ethereum, and Dogecoin). They asked: "Which is the best investment?"

They ran the robot with 6 different "recipes" (combinations of measurement and scoring methods). Here is what they found:

The Rock Stars: Bitcoin and Binance Coin were always in the top 2 spots, no matter which recipe was used. They are robust.
The Consistent Losers: Two other coins were always at the bottom. They are unstable in a bad way.
The Chameleons: Some coins, like Dogecoin, jumped around wildly. Depending on the recipe, Dogecoin could be 4th or 7th. This tells us that the ranking of Dogecoin is highly sensitive to how we do the math.

What This Tells Us

The tool doesn't just give you one answer; it gives you a confidence map.

If the robot says, "No matter how we slice it, Bitcoin is #1," you can be very confident.
If the robot says, "The winner changes depending on the math," you know you need to be very careful. You can't just pick one method and ignore the rest.

The "Speed Limit"

The paper notes that doing all these calculations at once can be heavy on the computer, like trying to bake 1,000 cakes at once. However, because the robot can use many processors (like having 1,000 bakers working in parallel), it can handle this quickly for most standard problems.

The Bottom Line

This paper introduces a way to stop guessing which "recipe" is best for your decisions. Instead of picking one method and hoping for the best, you can run a systematic test that shows you:

Which results are stable (safe bets).
Which results are shaky (danger zones).
Exactly how much your final decision depends on the math you chose.

It turns decision-making from a game of "trust me" into a transparent, data-driven process where you can see exactly how sensitive your results are to the rules you play by.

Technical Summary: Addressing Methodological Sensitivity in MCDM with a Systematic Pipeline Approach

Problem Statement
Multicriteria Decision-Making (MCDM) methods suffer from a fundamental limitation: the final rankings of alternatives are critically dependent on methodological choices, particularly the selection of data normalization (scaling) techniques. Empirical evidence indicates that varying normalization methods can alter 20–40% of final rankings in real-world applications. Current practices are characterized by ad-hoc selections based on precedent, preference, or software availability, lacking systematic robustness evaluation. Existing literature is fragmented, often limited to pairwise comparisons, and fails to provide unified frameworks for evaluating sensitivity across the broader methodological landscape. Furthermore, standard software tools typically require users to specify a single normalization procedure, preventing the systematic exploration of the methodological space and obscuring the inherent variability introduced by these choices.

Methodology
The authors propose a systematic framework for the automated exploration of the MCDM transformation space, built upon the existing Scikit-Criteria infrastructure. The core of this approach is the SKCCombinatorialPipeline, which implements a combinatorial strategy inspired by Grid Search techniques in machine learning (e.g., Scikit-Learn).

Combinatorial Generation: The framework generates the Cartesian product of all possible methodological combinations. If a pipeline consists of $k$ steps, where step $i$ has $|S_i|$ alternative methods, the system automatically constructs all $\prod |S_i|$ possible configurations.
Pipeline Architecture: The system utilizes modular components including:
- Transformers: Preprocessing steps such as negation (for minimization criteria), filtering (satisficing or non-dominated), and scaling (Sum, Vector, MinMax).
- Aggregators: Final evaluation methods such as the Weighted Sum Model (WSM) and TOPSIS.
Parallel Execution: To address computational complexity, the evaluation of independent pipelines is executed in parallel. The temporal complexity is modeled as $T_{parallel} = O(\frac{S \cdot k \cdot c}{P})$ , where $S$ is the total number of paths, $c$ is the cost per alternative, and $P$ is the number of processors. This allows for "embarrassingly parallel" execution with near-linear speedup.
Sensitivity Analysis: The framework leverages the RanksComparator module to analyze the generated rankings. It computes:
- Correlations: Spearman, Kendall, and Pearson correlations to measure monotonic agreement.
- Regression Metrics: Coefficient of determination ( $R^2$ ) to assess predictive capability between rankings.
- Distance Metrics: 20 pairwise distance measures (including Hamming, Euclidean, and Manhattan) to quantify geometric differences.
- Statistical Summaries: Median and Median Absolute Deviation (MAD) to categorize alternatives based on ranking stability.

Key Contributions
The paper presents three primary contributions:

Algorithmic Framework: The SKCCombinatorialPipeline enables the automated, exhaustive exploration of the MCDM transformation space through the Cartesian product of modular components.
Multi-metric Sensitivity Analysis: A comprehensive approach that provides correlations, covariances, $R^2$ scores, and distance-based measures between rankings generated by different methodological configurations, explicitly quantifying methodological sensitivity bounds.
Empirical Validation: Application of the framework to a real-world cryptocurrency dataset, demonstrating the ability to assess the robustness of conclusions and interpret sensitivity limits.

Results: Cryptocurrency Case Study
The framework was applied to a dataset of nine cryptocurrencies evaluated across market capitalization, volatility, and trading volume. The experiment combined three scaling methods (Sum, Vector, MinMax) with two aggregation methods (WSM, TOPSIS), generating six methodological scenarios.

Robust Alternatives: BTC and BNB consistently ranked in the top two positions across all configurations, while ADA and XRP remained fixed at the bottom (ranks 8 and 9), showing zero Median Absolute Deviation (MAD).
Sensitive Alternatives: DOGE exhibited the highest sensitivity, with rankings fluctuating between positions 4 and 7 (MAD = 1.0). XLM and LINK showed moderate variability (MAD = 0.5).
Methodological Agreement: Spearman correlation analysis revealed high agreement (0.983–1.000) between scaling methods when paired with the same aggregator (e.g., Sum-WSM vs. Vector-WSM). The lowest correlations (0.850) occurred between WSM and TOPSIS when paired with MinMax scaling, identifying this as the frontier of methodological sensitivity. However, all correlations exceeded 0.850, suggesting the fundamental ranking structure remained relatively stable despite methodological variations.

Significance and Claims
The authors claim that this work addresses the lack of unified frameworks for systematically evaluating sensitivity to scaling transformations. By materializing theoretical principles of sensitivity analysis into practical, automated tools, the framework facilitates the exhaustive exploration of the methodological space without manual intervention.

The significance of the proposed approach lies in:

Transparency: Providing explicit quantification of methodological sensitivity bounds, which is essential for contexts requiring transparent justification (e.g., strategic investments, public policy, audits).
Robustness Assessment: Enabling decision-makers to distinguish between alternatives that are robust to methodological choices (low MAD) and those that are highly sensitive, thereby preventing erroneous selections near decision boundaries.
Standardization: Potentially establishing new standards of methodological transparency for contemporary MCDM practice by moving away from ad-hoc selections toward systematic, data-driven evaluation.

The authors note that while the framework is powerful, its application is best suited for high-impact scenarios where decision accuracy is critical and computational resources allow for parallelization. They caution against its use for immediate operational decisions or when the methodological space becomes excessively large without intelligent sampling. The study acknowledges that its validation is currently limited to a single application domain (cryptocurrencies) and that generalization to other sectors requires further empirical validation.