📈 economics

Desirable Rankings

This paper proposes and validates a method for generating "desirable" collective rankings of alternatives based on agents' preferences relative to their matches, demonstrating through theoretical analysis, simulations, and real-world data that this approach converges to true quality rankings and outperforms existing benchmarks like revealed preference and Borda counts.

Original authors: Gaurab Aryal, Thayer Morrill, Peter Troyan

Published 2026-09-07✓ Author reviewed ⓘ

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gaurab Aryal, Thayer Morrill, Peter Troyan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of higher education and professional training, people constantly look for guides to help them choose where to go. A student deciding on a medical residency, a researcher picking a journal, or a family choosing a college all rely on lists that promise to show which institutions are the best. Traditionally, these lists are built by looking at statistics: how many people get accepted, how many get in, or how often a school's work is cited by others. The idea is that these numbers act as a mirror, reflecting the true quality of the institution. However, there is a well-known problem with this approach. When a school knows that a specific number, like a low acceptance rate, makes it look better, it may start acting in ways to manipulate that number. It might reject good students to keep the rate low, or encourage applications from students it knows will be rejected just to make the pool look larger. These tactics can distort the very metrics meant to measure quality, turning the ranking into a game rather than a true reflection of excellence.

To solve this, a new method has been proposed that ignores the statistics and looks directly at what people actually want. Instead of counting how many applications a school receives, this approach asks a simpler question: if a student is already at one school, which other school would they rather be at? The core idea is that if a student prefers a different school over the one they are currently attending, that preference is a signal of higher quality. By gathering these desires from thousands of students and filtering out the personal reasons that might sway a single individual—like living near home or liking a specific city—the researchers can build a ranking that reveals the underlying quality of the institutions. This method does not require knowing how much tuition costs or what the specific test scores of every student are; it only needs to know who is matched with which school and what order the students listed their choices.

The researchers, working with data from Chilean medical programs, developed a specific process to turn these individual desires into a collective ranking. They call this process the "Iterated Removal of Underdemanded Schools." Imagine a group of students who have been assigned to various medical schools. The researchers first imagine a scenario where students could trade their spots with one another to make everyone happier, a step that helps remove the influence of personal quirks and leaves only the shared desire for better education. Once this "cleaned" list of assignments is established, they look for schools that no student wants to switch to. These are the least desired schools. The algorithm identifies these schools and places them at the very bottom of the ranking. Then, it removes them from the list and asks the same question again with the remaining schools: among the ones left, which are now the least desired? These are placed just above the first group. This process repeats, peeling away the least desired schools layer by layer, until every school has been placed in a specific tier. The result is a ranking that moves from the least desired to the most desired, built entirely on the logic of what students would choose if they could.

When the researchers tested this method on real data from Chile between 2006 and 2009, the results were striking. The system successfully separated the medical programs into distinct groups, with the top schools consistently identified as the ones most desired by students who were already in the system. The Catholic University and the University of Chile, which are widely recognized today as the top medical schools in the country, were ranked first and second in every single year the data was analyzed. This stability was impressive because the method used very little information; in fact, students in Chile typically rank only a few programs on their lists, often fewer than four out of the twenty available. Despite this limited data, the algorithm produced a clear and stable picture of quality that held up over time.

The researchers also compared their new method against two other common ways of ranking schools. The first was a method that tries to calculate quality by looking at student test scores, family income, and other detailed background information. While this approach used much more data, it produced rankings that jumped around wildly from year to year, placing the top schools in different positions depending on the year, which made little sense since the quality of a school does not change that quickly. The second comparison was a simple point-counting system, where a school gets points based on how high it appears on a student's list. This method tended to favor larger schools that appeared on many lists, even if they were not the best, and it failed to correctly identify the top institutions. In contrast, the new method remained steady and accurate, correctly identifying the elite schools without needing complex calculations or vast amounts of background data.

The power of this approach lies in its ability to ignore the noise. In any ranking, there are always personal reasons why a student might prefer a school that is not objectively the best, such as a desire to stay close to family. The new method filters these personal preferences out by looking at the pattern of desires across the whole group. It essentially asks: if we remove the personal reasons, what is left? The answer is a clear signal of quality. The researchers confirmed through computer simulations that as the number of students and schools grows larger, this method becomes even more accurate, converging on the true quality of the schools. This suggests that the method is not just a lucky guess for a small group, but a robust way to understand quality in large systems.

The implications of this work extend beyond medical schools. The same logic could be applied to ranking academic journals, where researchers submit their work, or to other markets where people are matched with institutions. The key insight is that when people are given a choice, their desire to move from one option to another reveals more about the true quality of those options than any static statistic can. By focusing on what people want rather than how they are counted, it is possible to create a ranking that is harder to game and more honest about the value of the institutions involved. The study shows that even with simple data and a straightforward process, it is possible to cut through the confusion and find the true order of things.

Technical Summary: Desirable Rankings

Problem Statement
The paper addresses the challenge of aggregating individual ordinal preferences into a collective social ranking of alternatives (e.g., colleges, medical residency programs, journals). Traditional approaches rely on "index" measures (e.g., acceptance rates, citation counts) or revealed preference models. The authors argue that index measures are susceptible to Goodhart's Law, where institutions manipulate metrics to improve their standing. Conversely, revealed preference approaches are often data-intensive, requiring detailed information on student characteristics, tuition, and admission cutoffs, and may yield volatile rankings due to model specification sensitivity. The paper seeks a method that relies on minimal data—specifically, observed rank-order lists (ROLs) and the resulting matching outcome—to filter out idiosyncratic preferences and isolate the common component of quality.

Methodology
The authors propose a framework based on the concept of desirability, formalized through two axioms and an algorithm.

Core Concept: Shadow Matching and Desire
- The approach assumes an initial matching $\mu$ (the outcome of an admissions process) which may not be Pareto efficient from the students' perspective.
- The authors define a shadow matching ( $\mu^*$ ) as a Pareto efficient, Pareto improvement of the initial matching $\mu$ . This step is crucial for "filtering out" idiosyncratic preferences. If a student $i$ prefers a college $c$ to their shadow match $\mu^*(i)$ , they are said to desire $c$ .
- The intuition is that Pareto inefficiencies in $\mu$ often arise from conflicting idiosyncratic preferences (e.g., location, fit) rather than quality differences. By moving to a Pareto efficient state ( $\mu^*$ ), the remaining "desire" reflects the common component of preferences (quality).
Axioms
- Axiom of Desire (AoD): If a student desires a college $c$ (i.e., $c \succ_i \mu^*(i)$ ), then $c$ must be ranked strictly higher than $\mu^*(i)$ in the aggregate ranking.
- Justification: A college $c$ can only be ranked in a high tier if it is desired by a student from a lower tier. Specifically, if a college is in tier $k$ , there must exist a chain of students in tiers $k+1, \dots, K$ where each student desires the college in the tier immediately above them. This prevents low-quality colleges from being ranked highly simply because no one desires them (which would satisfy AoD vacuously).
Algorithm: Iterated Removal of Underdemanded Schools (IRUS)
- The authors introduce the IRUS algorithm to construct a ranking satisfying the above axioms.
- Step 1: Identify the set of underdemanded schools relative to $\mu^*$ . A school is underdemanded if all students weakly prefer their shadow match to that school.
- Step 2: Rank these underdemanded schools in the lowest tier. Remove them and the students matched to them.
- Step 3: Repeat the process on the remaining sub-market to identify the next lowest tier.
- The process continues until all schools are ranked. The output is a partition of colleges into tiers, allowing for ties within tiers.

Key Contributions and Theoretical Results

Existence and Uniqueness (Theorem 1): The IRUS algorithm produces a ranking that satisfies both the Axiom of Desire and Justification. For a fixed shadow matching, the IRUS ranking is the unique desirable ranking.
Convergence to True Quality (Theorem 2): The paper models student utility as a convex combination of a common quality component ( $\theta_c$ ) and an idiosyncratic component ( $\eta_{i,c}$ ). The authors prove that as the market size grows large, any desirable ranking converges to the true underlying quality ordering. Specifically, a college with quality $\theta_c$ will be ranked at the $\theta_c$ -th percentile.
Tier Size (Theorem 3): The authors demonstrate that in large markets, the size of any individual ranking tier becomes negligible relative to the total market size. This ensures that while ties are allowed, the ranking meaningfully distinguishes between colleges in different tiers.
Robustness to Shadow Matching Choice: While multiple shadow matchings may exist for a non-Pareto efficient initial matching, the authors show that the differences between rankings generated by different shadow matchings vanish as the market grows large.

Empirical Application and Results
The authors apply the IRUS algorithm to rank medical programs in Chile using administrative data from 2006–2009.

Data Constraints: The method requires only the initial matching and students' ROLs. Notably, Chilean students submit incomplete lists (ranking an average of 3–4 programs out of ~20), yet the algorithm produces stable results.
Findings: The desirable rankings show remarkable stability across the four years. Top programs (Catholic University and University of Chile) consistently occupy the top two positions, aligning with contemporary external rankings (e.g., QS World University Rankings).
Comparative Performance:
- Vs. Revealed Preference: The authors construct a revealed preference ranking using a discrete choice model (following Avery et al., 2013) adapted to their data. This alternative method yields highly volatile rankings (e.g., the top program fluctuates between 4th and 20th place) and fails to identify the top programs consistently.
- Vs. Borda Count: The Borda ranking (aggregating points based on list position) is more stable but systematically biases toward programs with larger applicant pools, undervaluing elite institutions like the Catholic University (ranking it 3rd–5th instead of 1st).

Significance and Claims
The paper claims that desirable rankings offer a robust, parsimonious alternative to traditional ranking methods.

Data Efficiency: The method succeeds with minimal data (ROLs and matching outcomes), avoiding the need for detailed student/college characteristics or tuition data required by revealed preference models.
Resistance to Manipulation: Unlike index-based rankings (which can be gamed by manipulating acceptance rates) or Borda counts (which reward frequency), desirable rankings require a college to be desired by students at lower-ranked schools to move up. This creates incentives for colleges to improve actual quality rather than manipulate metrics.
Theoretical Validity: The convergence results provide a theoretical guarantee that the method recovers true quality in large markets, filtering out the noise of idiosyncratic preferences through the Pareto improvement step.

The authors conclude that their approach is particularly valuable in settings where data is limited or where traditional quality metrics are contested or manipulable, demonstrating practical viability through the Chilean medical school application.

Technical Summary: Desirable Rankings

More like this