🤖 AI

Efficient Elicitation of Collective Disagreements

This paper proposes a stratified framework and the "plurality matrix" to identify the minimal aggregated preference information required to compute various disagreement measures, demonstrating that many such metrics necessitate data beyond simple pairwise comparisons and introducing elicitation protocols to balance participant load with estimation accuracy.

Original authors: Mohamed Ouaguenouni, Felipe Garrido-Lucero, Umberto Grandi, César Hidalgo, Magdalena Tydrichova

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed Ouaguenouni, Felipe Garrido-Lucero, Umberto Grandi, César Hidalgo, Magdalena Tydrichova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why "Who Wins?" Isn't Enough

Imagine you are organizing a party and need to pick a playlist. You ask your friends to vote.

The Old Way: You ask everyone, "Do you prefer Rock or Jazz?" Then, "Rock or Pop?" Then, "Jazz or Pop?" You tally up who wins the most head-to-head battles.
The Problem: The authors of this paper argue that this "head-to-head" method is like trying to understand a storm by only measuring the wind speed at two points. It tells you who is beating whom, but it misses the structure of the disagreement.

The Analogy:
Imagine two groups of people voting on three songs: A, B, and C.

Group 1 (The Chaos): Everyone has a totally different taste. Some love A, some B, some C. It's a free-for-all.
Group 2 (The Civil War): Half the group loves A and hates C. The other half loves C and hates A. They are deadlocked.

If you only ask "A vs. B" or "B vs. C," both groups look exactly the same! The math says they are identical. But the feeling of the room is totally different. Group 1 is just confused; Group 2 is deeply divided. The old method can't tell the difference.

The Solution: The "Plurality Matrix" (The Voting Pyramid)

To fix this, the authors introduce a new tool called the Plurality Matrix. Think of this as a pyramid of questions, getting slightly more complex as you go up.

Level 2 (The Base): This is the old "Head-to-Head" method. "Do you prefer A or B?"
Level 3 (The Middle): This asks about small groups. "If you had to pick a winner from A, B, and C, who would it be?"
Level 4+ (The Top): "Who wins if we pick from A, B, C, and D?"

The authors call this a "stratified framework." It's like building a house: you can't just look at the bricks (Level 2); you need to see how the walls are arranged (Level 3) to understand if the house is stable or about to collapse.

The Discovery: Some Disagreements Need "Level 3"

The paper proves a fascinating mathematical fact: You cannot measure deep disagreement using only Level 2 questions.

They looked at three famous ways to measure disagreement in society:

Agreement Index: How much do people agree? (This only needs Level 2).
Rank Variance: How much does a candidate's position jump around? (This needs Level 3).
Divisiveness: Is a candidate loved by one side and hated by the other? (This needs Level 3).

The Metaphor:
Imagine trying to describe a 3D object (like a cube) using only a 2D shadow.

If you only look at the "shadow" (Level 2/Pairwise comparisons), a flat square and a cube might look identical.
To see the depth (the true disagreement), you need to look at the object from a slightly different angle (Level 3/Triple comparisons).

The authors prove that for any level of complexity you choose (Level $k$ ), there is a specific type of disagreement that only reveals itself at the next level up (Level $k+1$ ). It's a strict hierarchy: you can't skip steps.

The "Special Cases" (When the Rules Change)

The paper also notes that in some very specific, orderly worlds, you don't need the complex levels.

Single-Peaked Preferences: Imagine voters are standing on a line. Everyone prefers the person closest to them. In this orderly world, the "shadow" (Level 2) is enough to reconstruct the whole 3D object.
Plackett-Luce Model: Imagine a game where every candidate has a hidden "strength score," and people pick based on that strength. Again, the simple head-to-head questions are enough to figure out the whole picture.

But in the messy, real world where people have complex, chaotic opinions, you need the higher levels.

The Practical Problem: Asking Too Much

Now, here is the catch. Asking people "Who wins among A, B, and C?" is harder than asking "A or B?"

Cognitive Load: This is the mental effort required to answer. Comparing two items is easy. Comparing three or four is harder and takes more brainpower.
The Trade-off:
- Option A (The Chain): Ask simple questions one by one. "A vs B? Winner vs C?" This is easy for the voter, but you need thousands of voters to get a clear picture.
- Option B (The Ranking): Ask the voter to rank all four items at once. This is hard for the voter (high cognitive load), but you need fewer voters because each one gives you a lot of data.

The authors designed two protocols to manage this trade-off:

The Chain Protocol: Low mental effort per person, but requires a huge crowd. Good for online platforms with millions of users.
The Ranking Protocol: High mental effort per person, but requires a smaller, dedicated group. Good for small committees or expert panels.

Summary

The paper argues that to truly understand how a group disagrees, we can't just look at who beats who in pairs. We need to look at how people vote in small groups (triples, quads, etc.).

The Tool: A "Plurality Matrix" that organizes votes by group size.
The Rule: Deep disagreement (like polarization) is invisible at the pair level; you need at least group-of-three data to see it.
The Cost: Getting this data is a balancing act between how hard you make the voters think and how many voters you need to ask.

The authors provide the mathematical map to navigate this trade-off, ensuring we can measure societal division accurately without burning out the voters.

Technical Summary: Efficient Elicitation of Collective Disagreements

Problem Statement
Social choice theory has traditionally focused on consensus and preference aggregation rules. However, collective decision-making often requires confronting and resolving disagreements before reaching agreement. Existing literature offers various measures of disagreement, polarization, and divisiveness, but these typically rely on full preference rankings or pairwise comparisons. The authors identify a critical limitation: pairwise comparisons (degree 2 information) are insufficient to distinguish between different structural forms of disagreement. For instance, a population with uniformly distributed rankings and a population split into two opposing factions may yield identical pairwise comparison data, yet possess fundamentally different disagreement structures. Furthermore, eliciting full rankings imposes a high cognitive load on voters, creating a trade-off between information richness and practical usability. The paper seeks to determine the minimal aggregated preference information required to compute specific disagreement measures and to design elicitation protocols that balance this information need with voter cognitive load.

Methodology
The authors introduce a probabilistic model of voter preferences and propose a new representation called the plurality matrix.

The Plurality Matrix: For a set of alternatives $A$ , the matrix records, for every subset $S \subseteq A$ and every alternative $a \in S$ , the probability $p_S(a)$ that $a$ is ranked first within $S$ .
Hierarchy of Degrees: The matrix is stratified by the size of the subset $S$ . Data of "degree $k$ " refers to entries where $|S|=k$ . Pairwise comparisons correspond to degree 2, while full plurality scores correspond to degree $m$ .
Level of a Measure: A disagreement measure is defined as being of "level $k$ " if it can be expressed using data of degree at most $k$ , but not degree $k-1$ .
Elicitation Protocols: To estimate the plurality matrix without requiring full rankings, the authors propose two protocols based on sequences of pairwise comparisons:
1. $k$ -chain: A voter performs $k-1$ sequential pairwise comparisons to determine a winner within a subset of size $k$ .
2. $k$ -ranking: A voter provides a full ranking of a subset of size $k$ via pairwise comparisons.
  The authors analyze the trade-off between the total number of voters (budget) and the maximum cognitive load (number of comparisons per voter) required to achieve a specific estimation accuracy ( $\epsilon$ ) with confidence ( $1-\delta$ ).

Key Contributions

Formalization of the Plurality Matrix: The paper defines the plurality matrix as a framework to organize aggregated preference data by subset size, enabling the analysis of disagreement measures based on the minimal information required.
Level Analysis of Disagreement Measures:
- The Agreement Index is shown to be of level 2, relying solely on pairwise comparisons.
- The Rank Variance and Divisiveness measures are proven to be of level 3. The authors demonstrate that these measures cannot be computed solely from pairwise data, as distinct preference profiles can share identical degree-2 data but differ at degree 3.
- The paper proves the strictness of the hierarchy: for any $k$ , there exist profiles that coincide up to degree $k$ but differ at degree $k+1$ .
- The authors construct specific measures for every level $k$ , such as the $k$ -central moment of the rank distribution (level $k+1$ ), which captures properties like skewness (level 4) and kurtosis (level 5).
Structural Assumptions: The hierarchy is shown to collapse to level 2 under specific structural assumptions. If voter preferences follow the Plackett-Luce model or are single-peaked, all higher-degree information can be recovered from pairwise comparisons alone.
Elicitation Protocols and Trade-offs: Two protocols ( $k$ -chain and $k$ -ranking) are designed to estimate the plurality matrix. Theoretical analysis using Hoeffding's inequality characterizes the trade-off between the number of required voters and the cognitive load per voter. The authors show that chains are optimal for low cognitive load constraints or large populations, while rankings are more efficient for smaller populations willing to answer complex queries.

Results

Theoretical: The paper establishes that many standard disagreement measures (Rank Variance, Divisiveness) inherently require information beyond pairwise comparisons (degree 3). It proves that no measure of level $k$ can distinguish between profiles that agree up to degree $k$ .
Experimental/Synthetic: Numerical experiments on synthetic data (using models like Mallows, Plackett-Luce, and Single-Peaked) illustrate that higher-degree measures (skewness and kurtosis) can distinguish between unimodal and bimodal rank distributions that are indistinguishable at level 3.
Real Data: Analysis of real election data (Glasgow STV, NSW Legislative Assembly, and French presidential survey) shows that real-world disagreements populate the "moment plane" (skewness vs. kurtosis) in diverse ways, including bimodal distributions that signify two-camp polarization.
Protocol Efficiency: The derived bounds show that for a fixed accuracy, increasing the subset size $k$ (and thus the cognitive load) reduces the number of voters required, but only up to the point where the cognitive load constraint is met.

Significance and Claims
The paper claims to provide a foundational framework for understanding the information-theoretic requirements of measuring collective disagreement. By introducing the concept of "level," it clarifies that pairwise comparisons are insufficient for capturing nuanced disagreement structures like divisiveness and rank variance. The work offers a practical path forward for elicitation: by identifying the specific degree of information needed for a given measure, practitioners can design protocols that minimize cognitive burden without sacrificing the ability to detect specific types of disagreement. The authors note that their framework is limited to measures expressible via the plurality matrix and assumes truthful, noiseless responses, suggesting future work could extend to partial orders, approval ballots, or noisy response models.