📊 statistics

Assessing Methodological Distortions in the Clarivate Highly Cited Researchers List: International Comparisons and a Reform Framework

This paper critiques the methodological limitations of the Clarivate Highly Cited Researchers list by demonstrating how factors like whole counting and citation disparities distort international comparisons, and proposes a reform framework to address these biases through fractional counting, field normalization, and improved author disambiguation.

Original authors: F. PACHECO-TORGAL

Published 2026-07-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: F. PACHECO-TORGAL

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of science as a massive, global sports league. Every year, a famous organization called Clarivate releases a "Hall of Fame" list called the Highly Cited Researchers (HCR). This list is supposed to identify the top 1% of scientists whose work is read and referenced the most by their peers. Universities and governments use this list to decide who gets funding, promotions, and prestige.

However, this paper argues that the rules of the game are rigged. The author, F. Pacheco-Torgal, suggests that the way Clarivate counts points is flawed, leading to a distorted view of who is actually the "best" scientist.

Here is a breakdown of the paper's arguments using simple analogies:

1. The "Whole Counting" Problem: The Group Photo Flaw

The Issue: Currently, Clarivate uses a method called "whole counting." Imagine a team of 200 scientists writes one groundbreaking paper. If that paper gets 500 citations (mentions by other scientists), Clarivate gives 500 points to every single one of the 200 authors.
The Metaphor: It's like a group photo where everyone gets a gold medal just for being in the picture, regardless of who actually took the photo or who did the heavy lifting.
The Consequence: This massively inflates the scores of scientists who work in huge teams (common in fields like biology or physics). It rewards being part of a big club rather than individual brilliance. The paper suggests switching to "fractional counting," where the 500 points would be split 200 ways, giving each scientist only 2.5 points. This would be a fairer reflection of individual contribution.

2. The "Apples vs. Oranges" Problem: Different Sports, Same Trophy

The Issue: Different scientific fields have different "citation cultures." Some fields, like medicine, publish thousands of papers and cite each other constantly. Others, like mathematics or engineering, publish fewer papers and cite them less often.
The Metaphor: Imagine a race where sprinters (Medicine) run 100 meters in 10 seconds, and marathon runners (Math) run 42 kilometers in 3 hours. If the trophy is awarded to whoever runs the fastest distance in a single hour, the sprinters will always win, even though the marathon runners are incredibly skilled.
The Consequence: The current list doesn't adjust for these differences well. It favors fields that naturally generate more citations, making scientists in slower-paced fields look less impressive than they really are.

3. The "Self-Promotion" and "Name Confusion" Problem

The Issue: The current list doesn't strictly remove citations where a scientist cites their own work (self-citation), nor does it perfectly distinguish between two scientists with the same name (e.g., "J. Smith").
The Metaphor: It's like a popularity contest where you can boost your own score by shouting your own name in the crowd, and where two different people might accidentally get credit for the same award because they have the same name tag.
The Consequence: This can artificially inflate scores and misattribute credit to the wrong person.

The "Distortion Ratio": Comparing Two Scoreboards

To prove these flaws, the author compares Clarivate's list with another famous list: the Stanford Top 0.5% Scientists list. The Stanford list uses fairer rules (splitting points, removing self-citations, and better name-checking).

The author created a "Distortion Ratio" to see how much the two lists disagree.

If the ratio is 1.0: Both lists agree on how many scientists a country has.
If the ratio is high (like 2.0+): The Clarivate list is over-counting that country compared to the fairer Stanford list.
If the ratio is low (like 0.5): The Clarivate list is under-counting that country.

The Findings:

China: Had the highest distortion. The Clarivate list shows them as having twice as many top scientists as the Stanford list suggests. The author suggests this is because Chinese researchers are often in massive collaborative teams that benefit from the "whole counting" rule.
Norway, Japan, Italy, Canada: These countries were "under-represented" on the Clarivate list. Their scientists appeared less frequently there than they did on the Stanford list, likely because their research cultures involve smaller teams or different citation habits that the current rules penalize.
The Nobel Prize Test: The paper notes a shocking statistic: The Clarivate list includes only about 10% of Nobel Prize winners. In contrast, the Stanford list includes over 90%. This suggests the Clarivate list is missing many of the world's truly greatest scientific minds.

The Proposed Solution: A Three-Pillar Reform

The author doesn't say the list should be cancelled, but that it needs a major overhaul using three pillars:

Split the Points: Switch to "fractional counting" so credit is shared fairly among team members.
Level the Playing Field: Use better math to compare scientists within their specific, narrow fields (e.g., comparing a heart surgeon only to other heart surgeons, not to a physicist).
Clean Up the Data: Strictly remove self-citations and use unique ID numbers (like ORCID) to ensure the right "J. Smith" gets the credit.

The Bottom Line

The paper concludes that while the Clarivate Highly Cited Researchers list is popular and influential, its current rules act like a filter that distorts reality. It favors big teams and high-citation fields while potentially hiding the true impact of individual researchers in other areas. Until the rules are fixed, we should be very careful about using this list to make big decisions about funding or university rankings.

Technical Summary: Assessing Methodological Distortions in the Clarivate Highly Cited Researchers List

Problem Statement
The paper addresses significant methodological limitations in the Clarivate Highly Cited Researchers (HCR) list, a widely used indicator of scientific influence. The study argues that the current HCR methodology relies on "whole counting" (assigning full citation credit to every co-author regardless of team size) and broad disciplinary categorization. These practices are posited to inflate the apparent impact of researchers in large collaborative networks and fields with high citation densities (e.g., biomedical sciences), while systematically disadvantaging researchers in fields with lower citation rates or smaller team structures. Furthermore, the paper notes that the HCR list captures only approximately 10% of Nobel laureates, raising questions about its ability to identify truly outstanding scientific contributions compared to other metrics.

Methodology
The study employs a comparative quantitative analysis using two primary datasets for the 2024 publication cycle:

Clarivate HCR List (2024): Contains 6,886 researchers, sourced from the Web of Science corpus.
Stanford Top 0.5% Scientists Dataset (2025): Contains 57,508 researchers, sourced from Scopus, utilizing fractional counting, author disambiguation, and self-citation exclusion.

To quantify the divergence between these two methodologically distinct lists, the author introduces a Distortion Ratio (DR) defined as:
$DR = \frac{\text{Country HCR Share (\%)}}{\text{Country Stanford Annual Share (\%)}}$

Interpretation: A DR > 1.0 indicates over-representation in the HCR list relative to the Stanford benchmark; a DR < 1.0 indicates under-representation.
Statistical Rigor: The analysis calculates 95% confidence intervals for the DR using the delta method applied to binomial proportion standard errors. This accounts for sampling uncertainty, particularly for countries with small absolute HCR counts.
Scope: The analysis covers the top 20 countries by HCR count. It explicitly acknowledges that the DR measures cross-list discordance between two imperfect instruments rather than absolute bias, and notes limitations such as database differences (Web of Science vs. Scopus) and the lack of field-level decomposition in the aggregate data.

Key Contributions

Introduction of the Distortion Ratio (DR): A novel metric to systematically quantify national-level divergence between the HCR list and the Stanford ranking.
Empirical Evidence of Systematic Bias: The study provides statistical evidence that the HCR methodology produces significant distortions for specific nations, attributing these to structural advantages in large collaborations and field-specific citation practices.
Statistical Validation: By computing confidence intervals, the paper distinguishes between statistically robust findings and those that may be artifacts of sampling variation, cautioning against over-interpreting point estimates for countries with small sample sizes.
Reform Framework: The paper synthesizes existing scientometric literature into a concrete, three-pillar reform proposal for the HCR list.

Results
The analysis of the 2024 data reveals substantial variation in Distortion Ratios across countries:

China Mainland: Records the highest DR in the sample (2.10 for the annual dataset; 8.05 for the career-long dataset). The 95% confidence interval [1.99, 2.21] confirms this over-representation is statistically robust. The paper suggests this is likely driven by whole-counting inflation in large collaborative networks and concentration in high-citation fields.
Statistically Robust Under-representation: Several countries show DR values significantly below 1.0 with confidence intervals excluding 1.0, indicating they are under-represented in the HCR list relative to their Stanford performance. These include:
- Norway: DR = 0.46 (CI [0.24, 0.68])
- Italy: DR = 0.65 (CI [0.52, 0.77])
- Japan: DR = 0.69 (CI [0.53, 0.85])
- Sweden: DR = 0.74 (CI [0.54, 0.94])
- Canada: DR = 0.76 (CI [0.66, 0.87])
- United Kingdom: DR = 0.87 (CI [0.80, 0.95])
Statistically Indeterminate Cases: For 11 of the 20 countries (including Germany, France, and Australia), the 95% confidence intervals cross the null value of 1.0. For these nations, the point estimates cannot be statistically distinguished from parity, and the paper advises caution in drawing policy conclusions without longitudinal replication.
Nobel Laureate Discrepancy: The paper reiterates that the HCR list includes only ~10% of Nobel laureates, whereas the Stanford ranking (which uses fractional counting and disambiguation) includes >90%.

Significance and Claims
The paper claims that the current HCR methodology fails to accurately reflect individual scientific impact due to its reliance on whole counting and insufficient field normalization. It argues that these methodological flaws propagate into institutional rankings (such as the Academic Ranking of World Universities) and influence funding and hiring decisions.

The study proposes a Three-Pillar Reform Framework to address these issues:

Mandatory Fractional Counting: Replacing whole counting to ensure citation credit is divided among co-authors, thereby reducing the advantage of large collaborative networks.
Robust Cross-Field Normalization: Adopting a more granular field classification (e.g., the 54 fields used by ARWU) and applying percentile-rank normalization within each field to eliminate structural biases between disciplines.
Author Disambiguation and Self-Citation Exclusion: Implementing rigorous author identification (e.g., via ORCID) and excluding self-citations to prevent artificial inflation of impact.

The paper concludes that while the HCR list remains a prominent indicator, it should be interpreted with "considerable caution" until these reforms are adopted. It emphasizes that the findings regarding China, the US, UK, Canada, Italy, Belgium, Japan, Sweden, and Norway are statistically robust enough to warrant immediate attention, whereas findings for other countries require further longitudinal study. The analysis does not claim the Stanford dataset is a "ground truth" but uses it as a contrasting methodological reference to highlight systematic discordance in the HCR approach.