📈 economics

Efficient Threshold-based Model for Scientific Productivity Evaluation: a Proposal

This paper proposes a streamlined, privacy-compliant model for evaluating scientific productivity in Italian universities by leveraging existing National Scientific Qualification thresholds to generate real-time, adaptable, and transparent indicators for both institutional governance and individual researcher tracking.

Original authors: Giuseppe Sergioli, Francesco Bianchini, Pierluigi Graziani

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Giuseppe Sergioli, Francesco Bianchini, Pierluigi Graziani

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive library where thousands of scholars write books and articles every year. Every few years, the library's management tries to figure out which departments are doing the best work. Usually, this process is like organizing a giant, multi-year Olympics: it's expensive, takes a long time, and only happens occasionally.

This paper proposes a much simpler, "real-time" scoreboard. Instead of waiting for the Olympics, it suggests using a set of pre-existing "passing grades" that the Italian academic system already calculates every day.

Here is the breakdown of their idea using everyday analogies:

1. The "Passing Grade" System (The ASN)

In Italy, to become a higher-level professor (like moving from a "Junior" to a "Senior" role), a scholar must meet specific requirements. The government has already calculated the "average" amount of work needed to pass these exams.

The Analogy: Think of these requirements as a video game level. To beat Level 1 (become a Senior Professor), you need to collect a certain number of "coins" (publications), "stars" (citations), and "badges" (top-tier journal articles).
The Data: The government already knows exactly how many coins, stars, and badges the average person needs to pass. These are the "Thresholds."

2. The New Scoreboard (The Productivity Index)

The authors suggest a new way to measure how well a university or a specific department is doing. Instead of waiting for a big report, they propose checking every scholar's current "coin count" against the "passing grade" for their next level.

The Calculation:
- If a scholar has exactly the number of coins needed to pass, their score is 1.0.
- If they have half the coins needed, their score is 0.5.
- If they have double the coins needed, their score is 2.0.
The Department Score: You simply add up all the scholars' scores in a department and take the average. This gives you a "Department Productivity Index" (DPI).
- DPI > 1.0: The department is, on average, beating the "passing grade."
- DPI < 1.0: The department is, on average, falling short of the "passing grade."

3. Why This is Useful (The "Dashboard" vs. The "Autopsy")

The paper compares their method to the current system (called VQR) using a medical analogy:

The Current System (VQR): This is like a full-body autopsy or a deep-dive medical exam. It's incredibly detailed and accurate, but it takes a long time, costs a lot of money, and only happens every few years. By the time the results are out, the patient (the university) might have already changed.
The Proposed Model: This is like a fitness tracker or a smartwatch. It doesn't do a full autopsy, but it gives you a live, constantly updating heart rate. It tells you right now if the department is running fast enough or if they are slowing down.

4. The Key Features

No New Data Needed: The "coins" and "passing grades" are already recorded in the university's digital filing cabinet (called IRIS). The authors' model just does the math on what's already there.
Privacy Friendly: The system can calculate the "average heart rate" of the whole department without ever revealing the specific heart rate of any single person. It's like knowing the average speed of traffic on a highway without tracking individual cars.
Fairness: Because the "passing grade" changes depending on the subject (e.g., a philosopher needs different "coins" than a physicist), the system automatically adjusts for different types of work. It doesn't compare apples to oranges; it compares apples to the specific "apple standard."

5. What the Paper Found

The authors tested this idea on nine different universities (some real, some simulated). They found that:

Some universities had high averages but were very "uneven" (some departments were superstars, others were struggling).
Some universities had lower averages but were very "consistent" (everyone was doing about the same amount of work).
The model could easily break down the data to see if "Junior" staff were doing better than "Senior" staff, or if specific departments were the engine of the university's success.

The Bottom Line

The paper argues that universities don't need to wait for a massive, expensive, multi-year evaluation to know how they are doing. By using the "passing grades" that are already calculated for promotions, they can build a simple, free, and constantly updating dashboard that shows exactly how productive their research teams are, right now. It's a tool for continuous monitoring rather than periodic judgment.

Technical Summary: Efficient Threshold-based Model for Scientific Productivity Evaluation

Problem Statement
Current research evaluation systems, particularly in Italy, face significant challenges regarding resource intensity, temporal lag, and complexity. The primary national evaluation mechanisms—the Valutazione della Qualità della Ricerca (VQR) and the Abilitazione Scientifica Nazionale (ASN)—are rigorous but require substantial financial and temporal resources. The VQR, for instance, operates on multi-year cycles (typically covering a three-year period with a one-year evaluation process), resulting in data that is not fully up-to-date regarding current research productivity. Furthermore, while the VQR assesses institutions, the ASN assesses individuals, creating a fragmented landscape where continuous, low-cost monitoring of institutional performance is difficult. There is a need for a streamlined, efficient tool that leverages existing data to provide real-time, transparent, and privacy-compliant insights into scientific productivity without requiring new data collection or complex algorithmic overhead.

Methodology
The authors propose a simplified, threshold-based model that utilizes the existing framework of the Italian National Scientific Qualification (ASN). The methodology relies on the "threshold values" already calculated by the Ministry of University and Research (MUR) for different academic ranks (Assistant Professor, Associate Professor, Full Professor) and scientific sectors (bibliometric and non-bibliometric).

The core technical components of the model are:

Data Sources:
- Threshold Triplets ( $T$ ): Official ASN threshold values for the next academic rank (e.g., an Associate Professor is evaluated against Full Professor thresholds; a Full Professor against Committee Member thresholds). These include three indicators:
  - Bibliometric sectors: Number of articles, total citations, and h-index.
  - Non-bibliometric sectors: Number of articles/contributions, number of Class A journal articles, and number of monographs.
- Actual Value Triplets ( $t$ ): Real-time data for individual researchers retrieved from the Institutional Research Information System (IRIS).
Individual Productivity Index (PI):
For each researcher $S_i$ , a Productivity Index is calculated as the arithmetic mean of the ratios between their actual values and the corresponding next-rank thresholds:
$PI(S_i) = \frac{1}{3} \sum_{j=1}^{3} \frac{t^i_j}{T^i_j}$
- A value $< 1$ indicates performance below the average threshold.
- A value $\ge 1$ indicates performance meeting or exceeding the threshold.
- In cases where a threshold is zero, the indicator is excluded, and the average is recalculated over the remaining values.
Departmental and Institutional Aggregation:
- Departmental Productivity Index (DPI): The arithmetic mean of the $PI(S_i)$ scores for all $n$ scholars in a department.
- University-Level Metrics: A weighted mean ( $\bar{X}$ ) is calculated across all departments, weighted by the number of faculty ( $n_j$ ) in each department.
- Dispersion Metrics: To assess homogeneity, the model calculates a weighted standard deviation ($WST$) and a Coefficient of Variation ($CoV$).
Implementation:
The model is designed to be implemented via simple code (provided as supplementary material) that interfaces with the IRIS repository. It allows for disaggregated analysis by scientific sector, academic rank, or specific faculty groups (e.g., newly hired staff).

Key Contributions

Immediate Applicability: The model requires no new data collection, as it relies entirely on data already present in ministerial tables and the IRIS database.
Real-Time Monitoring: Unlike the VQR, which provides a retrospective snapshot, this model offers a continuously updatable view of productivity that changes daily as researchers update their IRIS profiles.
Granularity and Flexibility: The framework allows for "drill-down" analyses, enabling comparisons across specific scientific sectors, academic ranks, or sub-groups within an institution.
Privacy Compliance: The model generates aggregated institutional indicators (DPI) without requiring the public disclosure of individual researchers' productivity indices, thus adhering to privacy constraints.
Disciplinary Adaptability: By using sector-specific thresholds, the model inherently accounts for the heterogeneity between bibliometric (STEM) and non-bibliometric (Humanities/Social Sciences) fields.

Results
The paper presents a simulation of the model applied to a virtual department and a case study analysis of nine Italian universities (categorized as large, medium, and small), using a mix of real and simulated data to ensure anonymity.

Simulation: A virtual department of ten scholars yielded a DPI of 1.24, indicating overall productivity above the national thresholds, though individual scholars showed variation (some below 1.0).
University Analysis:
- The analysis demonstrated that a high weighted average does not necessarily imply uniform excellence; high standard deviations and Coefficients of Variation revealed significant disparities between departments within the same university.
- For example, "Large University 1" had a high mean but high dispersion, driven by a few high-performing departments, whereas "Medium University 3" showed high variability with fewer than half of its departments exceeding the threshold.
- The breakdown by academic rank revealed that Assistant and Associate Professors often exhibited higher productivity indices relative to thresholds, likely driven by career progression incentives, while Full Professors showed slightly lower indices, potentially due to different institutional roles or internal evaluation metrics.
- The model successfully identified "pockets of excellence" and areas of underperformance within the same institutions.

Significance and Claims
The authors position this work not as a replacement for comprehensive, peer-reviewed national evaluations like the VQR, but as a complementary, low-cost tool for dynamic and continuous monitoring.

Strategic Value: The model serves as an "intermediate, dynamic layer" of monitoring that can inform internal governance, support individual career planning, and help institutions detect critical issues or pockets of excellence in a timely manner.
Efficiency: It offers a pragmatic solution for institutions seeking to track research performance without the substantial time and personnel costs associated with full-scale evaluations.
Generalizability: While developed within the Italian context, the authors claim the methodology is exportable to other national contexts that utilize threshold-based or publication-data-driven evaluation systems, provided comparable data is available.
Transparency: By leveraging existing, transparent thresholds, the model aims to reduce the opacity often associated with complex evaluation algorithms, fostering a culture of accountability and continuous improvement.

The paper concludes that this threshold-based approach provides a compact, interpretable, and sensitive measure of research performance that respects disciplinary diversity and career stages, offering a viable framework for refining evaluation strategies globally.