Science-wide mapping and ranking of institutions based on affiliated authors' impact and research integrity proxies

This study generates a comprehensive dataset and proposes a penalized percentile ranking system for nearly 7,000 research institutions, balancing the volume of top-cited authors against proxies for research integrity issues like excessive self-citation, publication in discontinued journals, and retractions to provide a more nuanced assessment of institutional impact.

The Gist

Imagine the world of scientific research as a massive, bustling city with thousands of different neighborhoods (universities, research institutes, and tech companies). For years, people have tried to rank these neighborhoods based on how "famous" their residents are. Usually, they just count how many people in a neighborhood have won a "Best Citizen" award (highly cited papers).

But this paper argues that this method is flawed. It's like judging a neighborhood only by the number of celebrities living there, while ignoring whether those celebrities are actually good neighbors, or if some of them are just buying awards, shouting their own name to get attention, or living in houses that are falling apart.

Here is a simple breakdown of what the authors did, using some creative analogies:

1. The Problem: Counting Stars vs. Checking the Neighborhood

Most rankings are like a Gold Star Count. They look at a university and say, "Wow, you have 1,000 famous scientists! You are #1!"

• The Flaw: This ignores the size of the school. A giant university with 50,000 students might have 1,000 stars, but that's only 2% of its population. A tiny, elite research lab with only 20 people might have 10 stars (50% of its population). The giant school looks "bigger," but the tiny lab is actually more "star-studded."
• The Integrity Issue: Even worse, some of those "stars" might be fake. They might be famous because they only cite their own work (like shouting "I'm great!" over and over), they publish in sketchy magazines that were later banned, or they have papers that were taken back (retracted) because of mistakes or cheating.

2. The Solution: The "Smart Neighborhood Score"

The authors (led by Dr. John Ioannidis) decided to build a new scoreboard. Instead of just counting stars, they created a "Smart Neighborhood Score" that balances Fame with Good Behavior.

They looked at nearly 7,000 institutions and applied three specific "Integrity Checks":

• The "Echo Chamber" Check (Self-Citations):
  • Analogy: Imagine a town where the mayor only talks about himself at every town hall meeting.
  • The Rule: If a scientist cites their own work too much (more than 95% of their peers), it's a red flag. It suggests they are trying to game the system to look more popular than they are.
• The "Banned Magazine" Check (Discontinued Titles):
  • Analogy: Imagine a chef who only cooks in restaurants that have been shut down by health inspectors for serving rotten food.
  • The Rule: If a scientist publishes mostly in journals that Scopus (the giant library database) has kicked out for being low-quality, it's a penalty.
• The "Recalled Product" Check (Retractions):
  • Analogy: Imagine a car manufacturer that has to recall thousands of cars because the brakes don't work.
  • The Rule: If a scientist has papers that were officially taken back (retracted) because of errors or misconduct (not just a typo by the publisher), the institution gets a heavy penalty.

3. The Formula: The "Good Neighbor" Equation

The authors created a math formula that looks like this:

Your Score = (How many famous scientists you have) MINUS (How many bad behaviors they show).

• The Penalty: If an institution has a scientist who published a retracted paper, it's a huge hit. The authors say that two retracted papers are as bad as losing one famous scientist.
• The Bonus/Penalty: If an institution has too many scientists who shout at themselves (self-cite) or publish in banned magazines, their score goes down.

4. The Shocking Results: Who Won and Who Lost?

When they ran the numbers, the rankings changed dramatically.

• The Old Winners: Huge universities like Harvard and Stanford still did very well, but not always at the very top of the proportion list.
• The New Stars: Small, elite research institutes and tech companies (like Meta FAIR, Princeton, and Carnegie Mellon) shot to the top. Why? Because a huge chunk of their scientists are actually top-tier, and they have very few "bad apples."
• The Big Drop: Some countries and institutions that looked great on the old "Gold Star" lists fell hard.
  • Saudi Arabia, China, Malaysia, Iran, India, and Indonesia saw their rankings plummet.
  • Why? While they have many famous scientists, a surprisingly high number of them were flagged for the "bad behaviors" (retractions, self-citing, or publishing in banned journals).
  • The Analogy: It's like a city that claims to have the most "Best Citizen" awards, but when you check the records, you find that half the winners were caught cheating or living in condemned buildings. Once you subtract those, the city isn't looking so great anymore.

5. The Takeaway: Quality Over Quantity

The main message of this paper is: Don't just count the trophies; check if they were earned fairly.

The authors aren't saying these countries or institutions are "bad." They are saying that if you want to know who is truly excellent, you have to look at the whole picture:

1. How many great scientists do they have?
2. What percentage of their total staff are great?
3. Are those great scientists playing by the rules?

They provide a free, public database so that anyone can check these "Smart Neighborhood Scores." It's a tool to help universities, governments, and the public stop being fooled by big numbers and start looking for honest, high-quality science.

In short: It's a reminder that in science, as in life, it's better to be a small group of honest, hard-working people than a huge crowd of people trying to cheat their way to the top.

Technical Summary

1. Problem Statement

Current institutional research rankings and assessments suffer from three primary limitations:

• Lack of Integrity Metrics: Most rankings focus exclusively on impact (citations, volume) and ignore research integrity issues such as misconduct, retractions, or "gaming" of metrics (e.g., excessive self-citation, publishing in predatory journals).
• Size and Composition Bias: Rankings often fail to adjust for institution size or the specific composition of their workforce (e.g., tenured vs. adjunct, senior vs. early-career). Normalizing by total staff is difficult due to inconsistent global reporting standards for faculty and staff categories.
• Oversimplification: Existing metrics often rely on absolute numbers (total papers/citations) which favor large institutions, or simple proportions that do not account for the career stage or leadership roles of the authors.

The authors aim to create a standardized, institution-level dataset that balances citation impact against proxies for research integrity, while adjusting for institution size and scientific subfields.

2. Methodology

Data Source:

• Database: Scopus (data freeze: August 1, 2025).
• Author Pool: 10,933,183 authors with at least 5 full papers (articles, reviews, conference papers).
• Top-Cited Definition: Authors in the top 2% of their subfield (174 subfields) or top 100,000 globally, based on a composite citation indicator (combining total citations, h-index, hm-index, and position-weighted citations).

Institutional Aggregation:

• Authors' affiliations were mapped to "aggregated institutions" using Scopus hierarchies.
• Primary Analysis Filter: To ensure comparability across diverse institutions (universities, research institutes, tech companies), the study focused on a specific subset of authors:
  • Started publishing in 1980 or later (approx. 45-year career window).
  • Published 40+ total items.
  • Published 5+ items as single, first, or last author (indicating leadership/seniority).
• Thresholds: Institutions were included in the final percentile rankings if they had at least 100 authors meeting the primary analysis criteria.

Research Integrity Proxies:
Three indicators were used as proxies for potential integrity issues (not definitive proof of misconduct):

1. Retractions: Total authorships on papers retracted for reasons other than publisher/journal error ($\Sigma R$).
2. High Self-Citation: Proportion of top-cited authors with self-citation rates >95th percentile of their subfield peers.
3. Discontinued Titles: Proportion of top-cited authors with publication rates in Scopus-discontinued titles (titles removed for quality concerns) >95th percentile.

Scoring Mechanism:
A summary score was calculated for each institution to balance impact and integrity:
$$\text{Score} = \frac{N(t) - \frac{\Sigma R}{2} - \max(0, 0.05 - \frac{S(t)}{N(t)}) \cdot N(t) - \max(0, 0.05 - \frac{D(t)}{N(t)}) \cdot N(t)}{N(a)}$$
Where:

• $N(t)$: Number of top-cited authors (primary analysis).
• $N(a)$: Total number of eligible authors (primary analysis).
• $\Sigma R$: Sum of retraction authorships (penalized heavily; 2 retractions = loss of 1 top-cited author).
• $S(t)$: Top-cited authors with high self-citations.
• $D(t)$: Top-cited authors with high rates in discontinued titles.
• Note: If the proportion of problematic authors exceeds 5%, the institution is penalized; if below 5%, it receives a bonus.

Sensitivity Analysis:
The authors tested various thresholds (80th, 95th, 99th percentiles for self-citation/discontinuation) and retraction penalty factors (1x, 2x, 4x) to ensure ranking stability.

3. Key Contributions

1. Integrated Integrity-Impact Metric: The paper introduces a novel framework that does not just list top institutions by volume but provides a "risk-adjusted" ranking that penalizes institutions for integrity proxies.
2. Standardized Author Filtering: By filtering for authors with a 45-year career window and specific leadership publication counts, the study creates a more level playing field for comparing universities, research institutes, and corporate R&D labs.
3. Publicly Available Dataset: The authors released a comprehensive dataset covering 6,979 institutions (and detailed rankings for 2,380 institutions with sufficient sample sizes), allowing for granular exploration of institution-level performance.
4. Geographic Disparity Mapping: The study identifies specific geographic clusters where integrity proxies are disproportionately high, offering a new lens for understanding global research ecosystems.

4. Key Results

Volume vs. Proportion:

• Highest Volume: Institutions like Harvard, Stanford, and major Chinese universities (e.g., Shanghai Jiao Tong, Tsinghua) dominate in the absolute number of top-cited authors.
• Highest Proportion (Unadjusted): When looking at the percentage of top-cited authors, the list shifts dramatically. Small research institutes (e.g., Max Planck Institutes, Howard Hughes Medical Institute) and tech organizations (e.g., Meta FAIR, Alphabet Inc.) often rank higher than large universities.
• Chinese Institutions: While having the highest absolute numbers, Chinese institutions often show lower proportions of top-cited authors compared to US/European peers in the primary analysis, largely due to the massive recent growth in their author pools (diluting the percentage).

Integrity Penalties and Regional Trends:

• Retractions: 75 of the top 100 institutions ranked by retraction penalties are in China (18% of Chinese institutions in the dataset).
• Self-Citations: Russia dominates this category (26 of the top 100 institutions, representing 52% of Russian institutions).
• Discontinued Titles: India, China, Malaysia, Saudi Arabia, and Russia show high concentrations of institutions with high rates of publications in discontinued titles.
• Adjusted Rankings: When applying the integrity penalties:
  • High Performers: The US, UK, Switzerland, Netherlands, and Canada maintain high percentile rankings (median 82.6–85.5).
  • Low Performers: Saudi Arabia, China, Malaysia, Iran, India, and Indonesia see their median rankings drop significantly (3.9th to 21st percentile).
  • Penalty Magnitude: For countries like Saudi Arabia and China, the cumulative integrity penalty exceeds their number of top-cited authors by 1.29 to 2.75 times.

Specific Findings:

• There is minimal overlap between institutions with the highest volume of top-cited authors and those with the highest proportion of top-cited authors.
• Small institutions with very high proportions of top-cited authors often disappear from the top ranks once integrity penalties are applied, suggesting potential metric gaming in small entities.

5. Significance and Implications

• Reframing Research Assessment: The study challenges the "volume-first" approach of current rankings (e.g., Shanghai Ranking, QS) by demonstrating that high impact often co-exists with significant integrity risks in specific regions and institutions.
• Warning Signals: The integrity proxies serve as "warning signals" rather than definitive proof of misconduct. They highlight institutions and countries that require deeper, qualitative scrutiny regarding research culture and oversight.
• Policy Impact: The data supports initiatives like COARA (Coalition for Advancing Research Assessment) by providing a standardized, multi-dimensional metric that moves beyond simple citation counts.
• Limitations Acknowledged: The authors note limitations including potential Scopus author misclassification, the aggregation of affiliations, the fact that retractions are only a fraction of actual misconduct, and the arbitrary nature of the weighting in the summary score. However, sensitivity analyses show that the overall institutional ordering remains robust across different parameter settings.

In conclusion, this paper provides a critical tool for a more nuanced, honest, and integrity-aware evaluation of global research institutions, revealing that high citation impact does not always correlate with high research integrity.