📊 statistics

Geometric mean-based pairwise comparison method with the reference values -- statistical approach

This paper presents a statistical approach to the pairwise comparison method using reference values and geometric means to simultaneously capture inconsistency and preference distances, while defining interpretable indicators to measure the quality of the resulting weight vector.

Original authors: Konrad Kułakowski, Kamil Pustelnik, Jacek Szybowski

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Konrad Kułakowski, Kamil Pustelnik, Jacek Szybowski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a school club, and you need to pick the best three activities for the upcoming year. You ask your friends to vote, but instead of just picking a favorite, you ask them to compare every single activity against every other one. "Is 'Hiking' twice as fun as 'Baking'?" "Is 'Coding' three times better than 'Art'?" This method of comparing things in pairs is a classic tool used by experts to make tough decisions, from choosing a new city mayor to picking the best medical treatment. It's like trying to figure out the exact height of a mountain by only measuring the slope between two trees at a time.

The problem is that humans aren't perfect calculators. One friend might say Hiking is better than Baking, but then say Baking is better than Coding, and Coding is better than Hiking. That's a logical loop, or "inconsistency." For decades, mathematicians have had ways to smooth out these loops and find a final ranking, but they mostly treated the errors as messy noise to be ignored. They could tell you what the ranking was, but they couldn't easily tell you how sure they were about it. It was like getting a weather forecast that says "It will rain" without saying if there's a 51% chance or a 99% chance. This paper steps into that gap, treating those messy human comparisons not just as errors, but as data points in a statistical experiment, allowing us to measure the confidence in our decisions just like a scientist measures the confidence in a lab result.

The Paper's Big Idea: Turning Guesswork into a Math Lab

The authors of this paper, Konrad Kułakowski, Kamil Pustelnik, and Jacek Szybowski, are looking at a specific decision-making method called "Heuristic Rating Estimation" (HRE). Imagine you have a list of unknown items (like new video games) and a few known items (like your favorite classic game, which you know is a perfect 10/10). You ask an expert to compare the new games against the classics and against each other. The goal is to figure out the "weight" or score of the new games.

Traditionally, this is done using a geometric mean (a specific type of average that handles ratios well). The authors realized something brilliant: this whole process is actually a linear regression problem. That's a fancy way of saying it's the same math used to draw a "line of best fit" through a scatter plot of dots.

Here is the analogy: Imagine you are trying to guess the weight of several mystery boxes. You have a few known weights (reference values) on a scale. You don't know the mystery weights, but you have a friend who gives you clues like, "Box A is about half as heavy as Box B," or "Box C is twice as heavy as the 5kg reference weight." Your friend isn't perfect; sometimes they guess "twice as heavy" when it's actually "1.9 times."

The paper suggests we shouldn't just average these guesses. Instead, we should treat every single comparison as a "measurement" in a science experiment. The "unknown weights" are the variables we are trying to solve for, and the "friend's guesses" are our data points. By using the tools of statistics (specifically, the least squares method), we can find the most likely weights for the mystery boxes.

What They Found: Confidence Intervals and "Tie" Clusters

The real magic of this paper isn't just finding the weights; it's about knowing how much to trust them. Because they treated the problem as a statistical experiment, they could calculate things that were previously impossible:

Confidence Intervals: Just like a poll might say "50% support, plus or minus 3%," this method can tell you that a game's score is likely between 0.15 and 0.25. If the range is huge, you know the data is shaky. If it's tiny, you can trust the ranking.
Probability of Rank Reversal: This is the most exciting part. The authors calculated the exact probability that the order of two items is correct. For example, they found that for two specific items in their example, there was a 99.46% chance that one was better than the other. But for another pair, the chance was only 58.18%. That's barely better than flipping a coin!
The "Tie" Solution: This is where the paper gets really practical. If the probability that Item A is better than Item B is low (say, below 75%), the authors suggest we shouldn't force a ranking. Instead, we should group them into a "tie cluster." Imagine a race where two runners are so close that the finish line camera can't tell who won. Instead of declaring a fake winner, you say, "They tied." The paper proposes an algorithm to automatically group these uncertain items together and give them an average score. This prevents the decision-maker from making a bold choice based on weak evidence.

What They Don't Claim (and What They Rule Out)

It is important to note what this paper does not do. It doesn't claim to fix the human brain or make experts stop making mistakes. The "noise" or inconsistency in the data is still there; the paper just measures it better.

The authors explicitly argue against the idea that we should just ignore the uncertainty or rely solely on traditional "inconsistency indices" (old math tools that give a single number to say how messy the data is). They show that a single number isn't enough because two different pairs of items can have the same "messiness" score, but one pair might be clearly separated (easy to rank) while the other is almost identical (hard to rank). Their new method captures both the "messiness" and the "distance" between the items.

They also don't claim this is a magic bullet for every situation. Their results are based on mathematical proofs and simulations (like the example with 7 alternatives they worked through). They suggest that in group settings, where multiple experts provide data, this method can be even more powerful because it can combine everyone's "measurements" to get a clearer picture, even if each expert missed some comparisons.

The Takeaway

In simple terms, this paper takes the messy, subjective world of "I think this is better than that" and turns it into a rigorous, data-driven science. It gives us a way to say, "We are 99% sure A is better than B, but we are only 58% sure about C and D, so let's call C and D a tie." It turns a rigid list of rankings into a flexible, honest representation of what we actually know, helping decision-makers avoid the trap of pretending to be more certain than they really are.

Technical Summary: Geometric Mean-Based Pairwise Comparison Method with Reference Values – A Statistical Approach

Problem Statement
Pairwise comparison (PC) methods, such as the Analytic Hierarchy Process (AHP) and the Best-Worst Method (BWM), are widely used for decision-making involving expert judgments. While traditional methods like the Eigenvector Method (EVM) and Geometric Mean Method (GMM) calculate alternative weights, they often struggle with incomplete data and lack a robust statistical framework to quantify the uncertainty of the resulting rankings. Existing statistical interpretations of PC methods have largely focused on complete matrices without reference values. Furthermore, classical inconsistency indices (e.g., Consistency Ratio) measure the dispersion of judgments but do not account for the "preference distance" between alternatives. Consequently, a highly inconsistent dataset might still yield a reliable ordering for clearly distinct alternatives, while a consistent dataset might fail to distinguish between alternatives with very similar weights. There is a need for a method that integrates reference values (alternatives with known weights) with a statistical approach to assess the reliability of the derived weight vector and the probability of rank reversals.

Methodology
The authors propose a statistical interpretation of the Geometric Heuristic Rating Estimation (HRE) method, which calculates weights for unknown alternatives based on comparisons with known reference alternatives.

Regression Formulation: The paper demonstrates that the incomplete geometric HRE is mathematically equivalent to a linear regression problem. By taking the logarithm of the pairwise comparison ratios ( $y_{ij} = \ln c_{ij}$ ) and the weights ( $\theta_i = \ln w_i$ ), the relationship is modeled as:
$y_{ij} = \theta_i - \theta_j + \epsilon_{ij}$
where $\epsilon_{ij}$ represents a random error term assumed to be normally distributed. The unknown logarithmic weights of non-reference alternatives are treated as model parameters to be estimated via the Ordinary Least Squares (OLS) method.
Matrix Construction: The problem is structured using design matrices ( $X_U$ for non-reference alternatives and $X_K$ for reference alternatives) and a response vector derived from the observed comparisons. The solution involves inverting a matrix $L_U$ (a sub-matrix of the graph Laplacian) to estimate the logarithmic weights.
Statistical Inference: Once the weights are estimated, the framework applies standard statistical tools:
- Variance Estimation: The residual sum of squares is used to estimate the error variance ( $\hat{\sigma}^2$ ).
- Covariance Matrix: The covariance matrix of the estimators is derived as $\hat{\sigma}^2 L_U^{-1}$ , allowing for the calculation of standard errors for individual weights.
- Confidence Intervals: Using the Student's $t$ -distribution, confidence intervals are constructed for the logarithmic weights and transformed back to the original scale via exponentiation.
- Rank Reversal Probability: The authors derive formulas to calculate the probability that the estimated order of two alternatives ( $P(a_i \prec a_j)$ ) is correct. This involves calculating the distribution of the difference between two estimators, accounting for their covariance. This extends to triplets and larger sets of alternatives using multivariate $t$ -distributions.
Quality Indicators and Tie Clustering:
- lcPOI (Least Certain Pairwise Order Index): The minimum probability of order preservation among all pairs.
- alPOI (Average Likelihood Pairwise Order Index): The average probability of order preservation.
- Tie Clustering: A mechanism is proposed where alternatives with low order-probability (below a threshold $\delta$ ) are grouped into "tie clusters." Their weights are averaged, providing a more conservative and statistically honest representation of the ranking when distinctions are uncertain.
Group Decision Making: The framework is extended to group settings by aggregating comparisons from multiple experts into a single set of observations. The authors suggest a weighted regression approach where experts with lower error variance (higher confidence) are assigned greater weights in the aggregation.

Key Results and Contributions

Statistical Foundation for Geometric HRE: The paper establishes that incomplete geometric HRE is a least-squares regression problem, providing a rigorous statistical basis for a method previously defined heuristically.
Probabilistic Ranking Measures: The introduction of $P(a_i \prec a_j)$ allows decision-makers to assess the reliability of the ranking order, not just the point estimates of weights. The paper shows that this probability depends on both the inconsistency of the data and the magnitude of the difference between alternative weights.
New Quality Indicators: The definitions of lcPOI and alPOI offer a more nuanced view of data quality than traditional inconsistency indices, specifically highlighting cases where similar weights lead to uncertain rankings even with low inconsistency.
Tie-Threshold Mechanism: The proposed algorithm for grouping uncertain alternatives into ties prevents the over-interpretation of statistically insignificant differences, leading to more robust decision outcomes.
Group Decision Extension: The methodology successfully adapts to multi-expert scenarios, allowing for the aggregation of incomplete matrices and the potential weighting of experts based on their individual consistency (variance).

Significance
The paper claims that its primary contribution is twofold: first, it provides an intuitive interpretation of inconsistency as statistical error within the least-squares regression framework; second, it introduces probabilistic measures to assess the reliability of rankings. By shifting the focus from deterministic inconsistency indices to probabilistic order preservation, the approach allows for a more realistic assessment of decision quality. It acknowledges that even with "noisy" data, clear distinctions between alternatives can be reliable, while "clean" data may fail to distinguish between similar options. The proposed tie-clustering mechanism offers a practical solution for representing these uncertainties, moving away from false precision toward a more interpretable and cautious decision result. The framework is presented as a coherent extension of existing HRE developments, specifically targeting the assessment of uncertainty and the quality of weight vectors derived from incomplete pairwise comparison data.

More like this