One University Three Positions: A Reproducible Simulation Study of Database Dependent Institutional Rankings across OpenAlex Scopus and Web of Science
This study employs a reproducible Monte Carlo simulation of 700 synthetic universities to demonstrate that institutional rankings vary significantly across OpenAlex, Scopus, and Web of Science due to database-specific mechanisms like coverage and citation scaling, revealing that high system-level concordance masks consequential institution-level rank instability and regional biases.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every year, universities around the world are lined up in a single column, ranked from first to last, as if their standing were a fixed fact like a mountain's height. This practice relies on massive digital libraries that store millions of research papers, citations, and author names. These libraries, often called databases, are the tools researchers use to count how much a university produces and how often that work is read by others. The assumption behind these rankings is that if you count the same university in two different libraries, you will get roughly the same result. However, the reality is that no two libraries contain the exact same collection of work. One might include more papers written in languages other than English, while another might focus heavily on specific scientific fields or exclude certain types of documents like conference proceedings. Because the starting list of papers is different, the final count and the resulting rank can change dramatically, even though the university itself has not changed its research output at all.
A researcher named Mustafa Kayyali set out to understand just how much these differences matter. Instead of trying to fix the current rankings or compare real universities, he built a computer simulation to test the mechanics of the system. He created a fictional world containing seven hundred synthetic universities. Each of these made-up institutions was given a unique profile, including its size, its research quality, the languages it published in, and the mix of subjects it studied, such as medicine, engineering, or the arts. He then programmed three different digital libraries to "observe" this same group of universities. These libraries were designed to mimic the known behaviors of real-world systems: one was broad and inclusive, another was somewhere in the middle, and the third was more selective and stricter about what it included. The goal was not to see which library was the best, but to see how much the ranking of a single university would jump around depending on which library was used to measure it.
The results showed a surprising contradiction. When the researcher looked at the overall list, the rankings from the different libraries seemed to agree very well. The order of the universities moved together in a way that suggested a strong connection. However, when he looked at individual universities, the picture changed completely. A university that appeared in the top one hundred in one library might drop dozens of places in another. On average, a university moved about forty-one spots between the most inclusive library and the most selective one. This meant that roughly eighteen universities crossed the line into or out of the top one hundred simply because the observer changed. The high overall agreement was hiding a great deal of instability for specific institutions.
The study also revealed that these movements were not random. They followed a clear pattern based on where a university was located and what languages it used. Universities in the Global South, or those that published a significant portion of their work in languages other than English, tended to rank much higher in the broad, inclusive library. In the more selective library, these same institutions dropped significantly. Conversely, universities in other regions often saw their rankings fall when the library became more inclusive. This happened because the ranking system is a zero-sum game; if one group moves up because more of their work is counted, another group must move down. The simulation showed that the direction of the shift depended entirely on the rules built into the library, not on the actual quality of the research.
To understand why these gaps existed, the researcher broke down the differences into four main causes: how many papers were found, how well the university's name was linked to its papers, how citations were counted, and random errors in the data. The biggest factor was coverage, or simply how many papers the library managed to find. This single factor explained nearly two-thirds of the difference in rankings. The way universities were linked to their papers was a smaller factor, and in some cases, fixing the linking actually made the disagreement slightly worse before it got better. The study concluded that the variation was not a mistake to be fixed by better counting, but a fundamental feature of using different sources.
The paper argues that reporting a single number for a university's rank is misleading because it hides the uncertainty behind that number. A university might be ranked forty-seventh in one system and fifty-second in another, which is a small difference, but it could also be ranked sixty-second in one and one hundred and eighty-one in another. The researcher suggests that instead of a single rank, universities should be shown a range of possible positions, along with the probability of being in the top one hundred. This approach would make it clear that a university's position is not a fixed point on a map, but a location that shifts depending on the lens through which it is viewed. The study does not claim to have solved the problem of ranking, but it provides a clear method for measuring how much the choice of data source changes the answer, urging that this uncertainty be made visible to anyone who uses these lists.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.