The Academic Journal Guide: A View from Category Theory.

Technical Summary: The Academic Journal Guide: A View from Category Theory

Problem Statement
The Academic Journal Guide (AJG), published by the Chartered Association of Business Schools (CABS), serves as a primary proxy for research quality in business and management disciplines. While the AJG methodology claims to balance proprietary citation impact data with qualitative expert judgment to allocate journals into a discrete hierarchy of five categories (1 to 4*), the theoretical robustness of this categorization remains unassessed. Specifically, there is a lack of understanding regarding whether the CABS categorization is justifiable through the lens of categorization theory. Key questions include: To what extent do impact data versus expert opinion drive the rankings? Do the defined criteria (e.g., review rigor, availability of metrics) actually distinguish the categories? Is there significant overlap between categories that undermines the utility of the discrete hierarchy?

Methodology
The study employs a mixed-methods approach grounded in categorization theory, testing four specific hypotheses against empirical data from the AJG 2022 edition. The analysis focuses on four subject areas: Marketing, Management, International Business (IB), and Strategy, comprising a total of 324 journals.

1. Data Construction: The study utilizes three proprietary impact metrics provided by the AJG (CiteScore, SNIP, and SJR) covering the period 2017–2021. These were summed to create a composite impact measure due to high inter-correlation (r > 0.85).
2. Statistical Testing:
   • H1 (Impact Variance): One-way ANOVA and Kruskal-Wallis tests were used to assess whether impact data differs significantly across and between adjacent AJG categories.
   • H1b (Dominance of Impact): Ordinal Logistic Regression was employed to determine the extent to which impact data predicts category membership.
   • H2 (Review Rigor): A qualitative audit of journal websites and internet searches was conducted to verify claims regarding "double-blind" review processes and the number of reviewers, specifically testing the definition that Level 1 journals are "relatively lightly" reviewed.
   • H3 (Metric Availability): The availability of citation impact factors across categories was analyzed to test the hypothesis that lower-rated journals lack such data.
   • H4 (Geographic Bias): Ordinal Logistic Regression identified journals that appeared as outliers (i.e., their impact data suggested a different category than their assigned AJG rating). The national origins of these journals were compared to test for a UK-based bias.

Key Results

• Impact Data Dominance (H1a & H1b): Impact data explains a substantial portion of the variance in categorization (approximately two-thirds to three-quarters, with pseudo R² values ranging from 0.66 to 0.73). However, the study finds significant overlap between adjacent categories (specifically 2–3 and 3–4), where p-values for pairwise comparisons were non-significant. This suggests that while impact data is a primary driver, it does not perfectly align with the discrete boundaries set by the AJG.
• Review Rigor (H2): The hypothesis that review standards vary significantly by category was not supported. Analysis of journal websites revealed that even Level 1 journals predominantly utilize double-blind peer review with multiple reviewers, contradicting the AJG definition that Level 1 papers are "relatively lightly" reviewed.
• Metric Availability (H3): The hypothesis that Level 1 journals lack citation metrics was largely unsupported. While some Level 1 journals lacked specific metrics, the majority possessed data for all three main indices used by the AJG.
• Geographic Bias (H4): Contrary to the hypothesis that UK-based journals are favored, the analysis of potential re-categorizations suggested that UK-based journals were actually more likely to be promoted (moved to a higher category based on impact data) than non-UK journals. However, the study notes that the current system does not allow for appeals based on metric discrepancies without a change in the journal's characteristics.
• Re-categorization Potential: The regression models indicated that approximately one-third of the journals in the sample could be re-categorized (either promoted or demoted) based solely on their impact data to reduce overlap between groups.

Key Contributions
The paper's primary contribution is the application of categorization theory to the evaluation of journal ranking systems. By treating the AJG as a social construction intended to impose coherence on the research landscape, the author identifies critical theoretical gaps:

1. Boundary Ambiguity: The study demonstrates that the boundaries between AJG categories are not well-defined in practice, as evidenced by the substantial overlap in impact metrics between adjacent groups.
2. Opacity of Judgment: While the AJG emphasizes the role of expert opinion to differentiate it from pure metric-based rankings, the data suggests that impact metrics drive the majority of the variance. The lack of publicly available data regarding the "qualitative judgment" component prevents users from understanding why specific journals deviate from metric-based expectations.
3. Misalignment of Definitions: The empirical findings show a disconnect between the AJG's written definitions (e.g., regarding review rigor and metric availability) and the actual characteristics of the journals within those categories.

Significance and Claims
The paper argues that the current AJG approach, while influential, fails to meet the standards of categorization theory regarding clear boundaries and transparent criteria. The author claims that:

• The reliance on proprietary impact data, combined with opaque expert judgment, creates a system where a significant portion of journals (approx. one-third) may be miscategorized relative to their impact data.
• The system currently lacks a mechanism for journals to appeal their ratings based on metric discrepancies, as re-categorization is restricted to journals with substantial changes in "nature" or metrics since the last review.
• To align with principles of transparency (such as those in the DORA declaration) and categorization theory, the CABS must either make the qualitative judgment data public or revise the category definitions to better align with the empirical reality of the journals they contain.

The study concludes that unless the criteria for categorization are made transparent and the definitions aligned with the data, the AJG will continue to function as a "surrogate measure" that may inadvertently encourage "rank chasing" rather than reflecting true research quality. The paper does not propose a new ranking system but calls for a theoretically informed revision of the existing one, noting that the next iteration of the AJG is due in 2027.