Modeling within-department homogeneity in research quality rankings: an application to the Italian ISPD

1. Problem Statement

The paper addresses a critical flaw in the Italian academic department ranking system, specifically the Indice Standardizzato di Performance Dipartimentale (ISPD). The ISPD is used by the National Agency for the Evaluation of the University and Research Systems (ANVUR) to allocate public funding and identify "Departments of Excellence."

The Core Issue:
The current ISPD formulation exhibits a marked tendency toward polarization, where a large number of departments receive extreme scores (0 or 100), reducing the index's discriminatory power. The authors identify the root cause not in the scoring itself, but in a statistical assumption: the ISPD assumes that research output scores within a department are mutually independent.

The Reality:
In reality, scores within a department are intra-departmentally correlated due to:

• Scholars producing multiple outputs.
• Shared research environments and strategies within departments.
• Homogeneity in scientific capabilities among department members.

Consequences of Ignoring Correlation:

• Variance Underestimation: The standard deviation of the "scaled average" (the core metric) is underestimated because the correlation ($\rho > 0$) increases the variance of the sum of scores.
• Size-Dependent Bias: The variance of the scaled average grows with department size ($N_d$). Larger departments are statistically more likely to receive extreme scores (0 or 100) simply due to their size and internal correlation, rather than superior/inferior quality.
• Unfairness: This creates an unfair ranking mechanism where large departments are penalized or rewarded disproportionately compared to smaller ones with similar average performance.

2. Methodology

The authors propose a statistical framework to model this intra-departmental correlation and adjust the ISPD accordingly.

A. Theoretical Adjustment

The authors define a theoretical "Virtual Department" (VD) that accounts for correlation.

• Let $\tilde{z}_d$ be the scaled average of department $d$.
• Under independence, $\tilde{z}_d \sim N(0, 1)$.
• With intra-departmental correlation $\rho_d$, the variance becomes $\sigma_d^2 = 1 + \rho_d(N_d - 1)$.
• The Adjusted ISPD is defined as:
  $$ISPD^{adj}_d = \lfloor 200 \cdot \Phi(\tilde{z}_d / \sigma_d) + 0.5 \rfloor / 2$$
  where $\Phi$ is the standard normal CDF. This rescaling ensures the index reflects the true percentile position relative to a correlated distribution.

B. Modeling Correlation as a Function of Size

Since micro-data (individual paper scores) are often unavailable to the public, the authors propose a parsimonious model linking correlation to department size ($N_d$). They hypothesize that larger departments have lower average correlations due to greater heterogeneity.

• Full Correlation Model (FCM): Uses a modified Fisher transformation to model $\rho_d$ as a function of $N_d$:
  $$\text{logit-like transformation: } F(\rho_d) = \alpha + \beta(N_d - 1)$$
  $$\rho_d = \frac{\exp(F(\rho_d)) - 1}{\exp(F(\rho_d)) + \tilde{N}}$$
  where $\tilde{N}$ is the maximum department size. The parameter $\beta$ is expected to be negative.

C. The "Betoidal" Distribution

To estimate the model parameters using publicly available data (which is coarsened by rounding to half-integers and, for the 2022 ranking, left-truncated to only show the top 350 departments), the authors introduce a novel probability distribution:

• Betoidal Distribution: The distribution of $X = \Phi(Z)$ where $Z \sim N(0, \sigma^2)$.
• It resembles a symmetric Beta distribution but is derived from the Normal CDF transformation.
• Left-Truncated Betoidal: Adapted to handle the 2022 data where only the upper tail is visible.
• This allows for Maximum Likelihood (ML) estimation of $\alpha$ and $\beta$ directly from the rounded ISPD values without needing raw micro-data.

D. Competing Estimators

The paper compares the proposed FCM-based adjustment against:

1. Original ISPD: Assumes $\rho = 0$.
2. Non-Parametric (ISPD$_{np}$): Estimates $\rho$ directly from micro-data (unstable for moderate $N_d$).
3. Random Intercept Model (ISPD$_{rim}$): Assumes constant correlation across all departments (Constant Correlation Model, CCM).

3. Key Contributions

1. Identification of a Structural Flaw: The paper formally demonstrates that the polarization of the Italian ISPD is a statistical artifact caused by ignoring intra-departmental correlation, which interacts with department size.
2. The Betoidal Distribution: Introduction of a new probability distribution tailored for analyzing transformed Normal variables subject to rounding and truncation, enabling robust estimation from coarse public data.
3. Size-Dependent Correlation Model: Proposing and validating a model where correlation decreases as department size increases, offering a more realistic structural assumption than constant correlation models.
4. Feasible Estimation Framework: Developing an ML estimation procedure that works with the limited, coarsened data typically available to the public, not just the agency holding the raw data.

4. Results

Empirical Analysis (2017 and 2022 Data)

• Correlation Existence: Likelihood Ratio Tests (LRT) overwhelmingly reject the Null Correlation Model ($\rho=0$). Intra-departmental correlation is statistically significant in both datasets.
• Size Effect: The estimate for $\beta$ is significantly negative in both years, confirming that larger departments have lower intra-departmental correlations.
• Polarization: The estimated standard deviations ($\hat{\sigma}_d$) are significantly greater than 1 (ranging from ~2.1 to ~3.2), explaining the U-shaped distribution of ISPD scores observed in reality.
• Model Fit: The Full Correlation Model (FCM) fits the data significantly better than the Constant Correlation Model (CCM) and the Null model.

Simulation Study

The authors simulated 1,000 datasets mimicking the 2017 ranking under various perturbation scenarios.

• Metrics: Mean Absolute Deviation (MAD) from the "true" theoretical index and Percentage of Discordant Comparisons (PDC).
• Findings:
  • The Original ISPD performed poorly (high MAD ~13.6, high PDC ~5.8%), confirming severe polarization and ranking errors.
  • The Non-Parametric (ISPD$_{np}$) was unstable due to second-order statistic noise.
  • The Proposed FCM-based ISPD ($ISPD^{fcm}$) consistently achieved the lowest MAD (0.02–0.94) and lowest PDC (0.08–1.64), outperforming all competitors, including the RIM/CCM approach.
  • The FCM-adjusted index produced a nearly uniform distribution of scores, restoring discriminatory power.

5. Significance and Implications

• Fairness in Funding: The study proves that the current ranking system systematically distorts funding allocation based on department size rather than pure research quality. The proposed adjustment ensures that large and small departments are compared on a statistically fair basis.
• Policy Transparency: It highlights that evaluation agencies must explicitly state and model statistical assumptions (like independence) that underpin performance indicators. Ignoring these assumptions leads to systematic bias.
• Methodological Rigor: The paper demonstrates that "technical" statistical choices in index construction have concrete distributive consequences. Properly modeling correlation is not just a refinement but a prerequisite for accountability in research evaluation.
• Generalizability: The framework and the Betoidal distribution can be applied to other national evaluation systems that rely on aggregated performance metrics and face similar issues of data coarsening and intra-group correlation.

In conclusion, Montanari and Doretti provide a rigorous statistical solution to a major flaw in the Italian research evaluation system, offering a path toward a more equitable and accurate ranking mechanism that accounts for the complex internal structures of academic departments.