Aspect-Based Sentiment Evolution and its Correlation with Review Rounds in Multi-Round Peer Reviews: A Deep Learning Approach

Technical Summary: Aspect-Based Sentiment Evolution and its Correlation with Review Rounds in Multi-Round Peer Reviews

Problem Statement
While mining sentiment from peer review comments offers insights into scientific evaluation, existing research is constrained by two primary limitations: coarse-grained aspect analysis (e.g., broad categories like "innovation" or "clarity") and a lack of differentiation across multiple review rounds. Most studies treat the review process as a monolithic block or rely on overall review cycle duration, failing to capture the dynamic shifts in reviewer focus and sentiment tendencies as authors revise their manuscripts. Consequently, there is a gap in understanding how fine-grained aspect sentiments evolve from submission to publication and how these sentiments correlate with the specific number of review rounds a paper undergoes.

Methodology
The study employs a deep learning-based approach to analyze multi-round peer review comments from Nature Communications (NC), covering 11,063 accepted papers published between 2016 and 2020. The methodology proceeds through four key stages:

1. Data Collection and Preprocessing: The authors collected 41,373 peer review files. Using structural rule recognition and manual annotation, the texts were segmented by review rounds (ranging from 1 to 6 rounds). Noise was cleaned, and sentences were tokenized.
2. Aspect Identification: A fine-grained aspect extraction pipeline was implemented:
   • Noun phrases were identified using the spaCy toolkit.
   • The Double Propagation (DP) algorithm extracted candidate aspect words based on syntactic dependencies (e.g., mod, obj).
   • High-frequency aspect words were clustered using Affinity Propagation (AP) on 200-dimensional Word2Vec vectors, resulting in 11 distinct aspect clusters (e.g., "Experiments & Operations," "Result Analysis & Evaluation," "Methodology & Model").
3. Corpus Construction and Sentiment Classification:
   • A manually annotated corpus of 5,063 review sentences was created, labeling aspect sentiments as positive, neutral, negative, or ambiguous. Inter-annotator consistency was verified using Kappa coefficients (0.63–0.66).
   • A comparative experiment evaluated 14 deep learning models, including non-BERT architectures (LSTM, MEMNET, IAN) and BERT-based models (AEN-BERT, LCF-BERT, LCFS-BERT, RoBERTa variants).
   • The LCF-BERT-CDM model (Local Context Focus BERT with Contextual Distance Mechanism) was selected as the optimal model based on performance.
4. Statistical and Correlation Analysis:
   • The optimal model was used to classify sentiments across the entire corpus.
   • Sentiment scores were calculated at both the aspect-level and document-level using a normalized difference formula between positive and negative frequencies.
   • Spearman correlation analysis was conducted to examine the relationship between aspect sentiment scores and the total number of review rounds.

Key Results

• Model Performance: The LCF-BERT-CDM model achieved the highest performance with a Macro-F1 score of 82.65% and an Accuracy of 82.72%, outperforming both non-BERT models and other BERT variants. This suggests that emphasizing local context words near the aspect term is critical for peer review sentiment classification.
• Sentiment Distribution: Negative and neutral sentiments predominated across all review rounds, reflecting the critical nature of peer review. However, a consistent trend emerged: as the number of review rounds increased, the proportion of positive sentiments rose, while negative sentiments declined.
• Aspect Evolution:
  • Core aspects such as "Result Analysis & Evaluation," "Impact & Research Value," and "Methodology & Model" maintained high sentiment volume throughout the process.
  • In the final review rounds, attention to "Structure & Language" increased, indicating a shift toward finer details as major content issues were resolved.
  • For papers undergoing 2 or 3 rounds, positive sentiment generally increased with each subsequent round, though some aspects (like "Methodology") showed a dip in positivity in the second round before recovering in the third.
• Correlation with Review Rounds:
  • There is a significant negative correlation between aspect sentiment scores and the total number of review rounds. Lower sentiment scores (more negative) correlate with a higher number of rounds.
  • The aspects most strongly correlated with the number of review rounds were "Experiments & Operations" (EXP), "Impact & Research Value" (IMP), and "Result Analysis & Evaluation" (RES).
  • In later rounds (Round 4+), the "Methodology & Model" aspect showed a stronger correlation with the total number of rounds, suggesting that persistent methodological issues significantly prolong the review process.

Significance and Contributions
The paper claims significance in both theoretical and practical domains:

• Theoretical Contribution: The study advances the field of review comment mining by moving beyond coarse-grained analysis to fine-grained aspect-level sentiment analysis within a multi-round context. It demonstrates that reviewer focus and sentiment tendencies are not static but evolve dynamically across review stages, necessitating a round-specific analytical approach.
• Practical Application: The findings offer actionable guidance for authors. By identifying that negative sentiments regarding experiments, result analysis, and research significance are strongly linked to extended review cycles, the study suggests that authors should prioritize these specific aspects during writing and revision. Addressing these areas effectively can potentially reduce the number of review rounds and shorten the publication timeline.

Limitations
The authors acknowledge several limitations:

1. Data Scope: The dataset is restricted to accepted papers from Nature Communications, introducing a potential survivorship bias. Comments from rejected papers, which might contain critical sentiment patterns, were not included.
2. Generalizability: The findings are specific to the NC journal and may not fully generalize to other disciplines or journals with different review policies.
3. Model Performance: While LCF-BERT-CDM performed best, the model still struggles with implicit sentiment expressions, and errors may accumulate through the multi-step experimental pipeline (segmentation, aspect extraction, classification).
4. Causality: The correlation analysis does not incorporate control variables, meaning the observed relationships are associative rather than strictly causal.