🔢 mathematics

Closing a 17-Year Gap: Algorithmic Detection and Empirical Prevalence of Rank Reversal in Multi-Criteria Decision Analysis

This paper presents an open-source algorithmic framework within Scikit-Criteria that operationalizes Wang and Triantaphyllou's 17-year-old theoretical criteria for detecting rank reversal, revealing through a large-scale audit that while top-alternative stability is nearly universal, transitivity and recomposition consistency violations are pervasive in current Multi-Criteria Decision Analysis literature.

Original authors: Juan Bautista Cabral, Gonzalo Giarda, Diego Nicolás Gimenez Irusta, Paula Pacheco, Alvaro Roy Schachner, Agustín Borda

Published 2026-08-12

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan Bautista Cabral, Gonzalo Giarda, Diego Nicolás Gimenez Irusta, Paula Pacheco, Alvaro Roy Schachner, Agustín Borda

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a talent show, trying to pick the best act from a lineup of singers, dancers, and magicians. You have a scorecard with several categories: voice, stage presence, and originality. You add up the scores, and the magician wins. But then, a new act walks in—a terrible juggler who nobody cares about. Suddenly, the magician drops to second place, and the singer takes the top spot. Or maybe, you decide to judge the singers and dancers separately first, then combine the results, only to find that the combined winner is different from the one you picked when judging everyone together. This confusing flip-flopping is called "Rank Reversal." It happens in a field of science called Multi-Criteria Decision Analysis (MCDA), which is basically a fancy way of saying "using math to make tough choices when you have many different things to consider." Whether you are picking a new smartphone, a medical treatment, or a space mission, you want your math to be fair and consistent. If the winner changes just because you added a useless option or broke the problem into smaller pieces, the whole system feels broken, and people can't trust the results.

For over 17 years, scientists knew this problem existed and had written down three specific rules to test if a decision-making method was trustworthy. But here's the catch: they only wrote the rules; they never built the actual machine to run the tests. It was like having a recipe for a cake but no oven. Because the tests were so tricky to turn into computer code, no one had ever actually checked if the popular methods used in the real world were passing these rules. This left a huge gap between the theory (what we think should happen) and the practice (what actually happens in the software people use every day).

This paper is the story of someone finally building that oven. The authors, a team of researchers from Argentina, created a new, open-source computer toolkit (part of a library called Scikit-Criteria) that turns those 17-year-old rules into working code. They didn't just write the code; they used it to put 27 different decision-making methods through a rigorous stress test.

Here is what they found, and it's a bit of a plot twist. First, they tested if the "best" choice stayed the best when they made the other choices slightly worse (like giving the other acts a bad hair day). They found that almost everyone passed this test—96.3% of the time, the top choice stayed on top. It seemed like the methods were pretty solid.

But then, they ran the harder tests. They broke the problems down into tiny pairs (comparing just two acts at a time) to see if the logic held up. This is where things got messy. About 15% of the methods failed the "transitivity" test, meaning their logic got circular (like saying A is better than B, B is better than C, but C is better than A). Even more surprising, when they tried to rebuild the full ranking from those tiny pairs, nearly half (48.1%) of the methods failed the strictest test. The final ranking they got from the "small pieces" didn't match the ranking they got from the "whole picture."

The paper doesn't say these methods are useless, but it does prove that rank reversal isn't just a rare, weird glitch that happens in made-up examples. It's a common, measurable feature of the tools scientists and engineers are currently using. The authors suggest that we can't just assume these methods are consistent; we have to test them. By making their testing tools free and open for everyone to use, they've handed the world a new way to check if our decision-making math is actually doing what it's supposed to do, ensuring that when we pick a winner, it's because they truly deserve it, not because the math got confused by a bad juggler.

Technical Summary: Closing a 17-Year Gap in Rank Reversal Detection

Problem Statement
Rank Reversal (RR) remains a critical threat to the reliability of Multi-Criteria Decision Analysis (MCDA). It occurs when the relative order of alternatives changes in ways that violate axioms of rational decision-making, such as the Independence of Irrelevant Alternatives (IIA) or transitivity. While Wang and Triantaphyllou (2008) proposed three systematic test criteria to detect and quantify these phenomena over 17 years ago, a validated, open-source implementation translating these theoretical criteria into practical algorithms has been absent. This gap persists despite the criteria's high citation count, largely due to the non-trivial algorithmic challenges involved in operationalizing these tests for real-world decision pipelines, particularly regarding preprocessing filters, tie-breaking, and graph-theoretic reconstruction.

Methodology and Algorithmic Framework
The authors present an algorithmic framework implemented in the open-source Python library Scikit-Criteria. The framework operationalizes the three Wang and Triantaphyllou (2008) criteria through two primary testing modules: RankInvariantChecker (for Test 1) and RankTransitivityChecker (for Tests 2 and 3).

The technical foundation relies on two core data structures:

RankResult: Encapsulates the ordered ranking of alternatives, handling ordinal classifications and metadata.
RanksComparator: Facilitates the comparison of multiple rankings, supporting statistical calculations and the storage of experimental provenance.

The three tests are implemented as follows:

Rank Reversal Test 1 (RRT1): Systematic Degradation
- Objective: To determine if the optimal alternative remains stable when suboptimal alternatives are systematically degraded.
- Algorithm: The method employs a controlled degradation strategy where each suboptimal alternative is mutated to a "worse" state while preserving ordinal consistency. Noise is sampled from a uniform distribution bounded by the difference between the current and next-worse alternative.
- Pipeline Compatibility: To handle real-world pipelines that may filter out alternatives (e.g., via satisficing or dominance analysis), the authors implement graceful degradation. If an alternative is removed by a filter, it is assigned the worst possible rank (max_rank + 1) rather than causing the test to fail, ensuring the ranking structure remains complete.
- Metrics: The test outputs the Optimal Preservation Rate (OPR) and Rank Displacement (RD).
Rank Reversal Test 2 (RRT2): Transitivity Analysis
- Objective: To detect violations of transitivity (preference cycles) by decomposing the decision problem into pairwise comparisons.
- Algorithm: The decision matrix is decomposed into all $n(n-1)/2$ pairs. Each pair is evaluated independently to construct a dominance graph (an $n$ -tournament).
- Detection: The algorithm identifies directed 3-cycles within this graph. The Transitivity Violation Rate is calculated as the ratio of detected 3-cycles to the maximum possible 3-cycles for the given number of alternatives.
- Output: A boolean result indicating if the rate is zero, alongside the graph structure and specific cycle details.
Rank Reversal Test 3 (RRT3): Recomposition Consistency
- Objective: To verify if the original ranking can be uniquely reconstructed from the pairwise dominance graph.
- Algorithm:
  - Condensation: The dominance graph is condensed into a Directed Acyclic Graph (DAG) where cycles are collapsed into "supernodes" representing ties. This is an exact transformation that discards no information.
  - Transitive Reduction: Redundant edges are removed to preserve only minimal reachability.
  - Recomposition: Two strategies are offered:
    - Generations: Produces a single ranking with explicit ties for collapsed cycles.
    - Cycle Permutations: Enumerates all possible strict orderings within collapsed cycles to check for ambiguity.
- Criterion: The test passes only if the original ranking matches the reconstructed ranking (or the first of the enumerated permutations) exactly.

Key Results
The framework was validated through two case studies:

Cryptocurrency Evaluation (Van Heerden et al., 2021):
- Applied to a TOPSIS pipeline evaluating nine cryptocurrencies.
- RRT1: Passed perfectly (OPR = 1.0), showing the top alternative (Bitcoin) was stable under degradation.
- RRT2/RRT3: Failed. A single intransitive triple (BNB $\succ$ LTC $\succ$ ETH $\succ$ BNB) was detected, resulting in a violation rate of 0.033. This demonstrated that a method can be stable under perturbation (RRT1) yet internally inconsistent under decomposition (RRT2/3).
Large-Scale Audit of Published Literature:
- Reproduced 27 pipeline/dataset combinations from 20 published MCDM methods (including TOPSIS, VIKOR, PROMETHEE, etc.).
- RRT1: Passed for 96.3% (26/27) of pipelines, indicating top-alternative stability is nearly universal.
- RRT2: Passed for 85.2% (23/27), with failures concentrated in methods relying on reference regions computed from the full set.
- RRT3: Passed for only 51.9% (14/27). The authors note that many pipelines passing RRT2 failed RRT3 because scoring alternatives "all together" (using global normalization constants) disagreed with scoring them "one pair at a time," a violation of IIA that RRT2 cannot detect.

Significance and Claims
The paper claims to close a 17-year gap between the theoretical proposal of Wang and Triantaphyllou (2008) and practical application. By providing the first comprehensive, open-source implementation, the authors enable:

Empirical Prevalence Analysis: The audit reveals that rank reversal is not a marginal or adversarial concern but a pervasive feature of current MCDA literature, particularly regarding recomposition consistency (RRT3).
Methodological Rigor: The framework allows researchers to move beyond theoretical debates (positivist vs. constructivist) by providing tools to measure, quantify, and compare the empirical robustness of different methods.
Reproducibility: The tools facilitate routine, comparative analysis of method robustness across diverse domains, transforming what was previously a manual, bespoke exercise into a standardized pipeline.

The authors maintain a pragmatic stance, acknowledging limitations such as the $O(n^2)$ computational cost for large alternative sets and the specific nature of the degradation strategy, positioning the work as a foundational step toward broader adoption of systematic robustness analysis in MCDA.

Technical Summary: Closing a 17-Year Gap in Rank Reversal Detection

More like this