Technical Summary: Semantic R-Precision for Keyphrase Evaluation

1. Problem Statement

The evaluation of automatically generated keyphrases remains a complex challenge due to a misalignment between traditional automated metrics and human judgment. Current evaluation paradigms generally fall into two categories, both of which have significant limitations:

• Exact Lexical Matching: Metrics like Precision, Recall, and F1 rely on exact string or stem matches. They fail to recognize semantically equivalent phrases with different lexical forms (e.g., "neural networks" vs. "deep learning"), leading to underestimation of model performance.
• Semantic Similarity without Ranking: Recent approaches using contextual embeddings (e.g., BERTScore, SemP/R/F1) capture semantic meaning but treat predictions as unordered sets. They ignore the ranking order, despite evidence that human perception of relevance is heavily influenced by position (the primacy effect), where users focus disproportionately on top-ranked results.

Existing metrics either lack semantic flexibility or lack rank awareness, failing to reflect how humans actually judge the informativeness and relevance of keyphrase lists.

2. Methodology: Semantic R-Precision (SemR-p)

The authors propose Semantic R-Precision (SemR-p), a novel metric that integrates semantic similarity scoring into the rank-sensitive R-Precision framework. The methodology is grounded in human cognitive principles and Information Retrieval practices.

Core Formulation

SemR-p evaluates the top $R$ predictions from an ordered list $\hat{Y}$, where $R$ is the total number of reference keyphrases ($|Y|$). For each predicted keyphrase $\hat{y}_i$ within the top $R$, a score is calculated using a two-part logic:

1. Exact Stem Match: If the stem of $\hat{y}_i$ exactly matches the stem of any reference keyphrase $y_j$, the score is 1.0.
2. Semantic Similarity Score: If no exact stem match exists, the score is derived from the semantic similarity between the prediction and the references.
   • Phrase-level embeddings are generated using a Sentence Transformer model (specifically uclanlp/keyphrase-mpnet-v1).
   • The similarity is computed as the average cosine similarity between the prediction embedding and the top-$k$ most similar reference keyphrases.
   • Formula: $score(\hat{y}_i) = \frac{1}{k} \sum_{j=1}^{k} \cos\_sim(emb(\hat{y}_i), emb(y'_j))$, where $y'_j$ are the $k$ most similar references.

The final SemR-p score for a document is the average of these individual scores across the top $R$ predictions.

Experimental Setup

• Datasets: kp20k (scientific abstracts) and kptimes (news articles).
• Models: A representative subset of 8 models covering unsupervised extraction, supervised extraction, supervised generation, and LLM-based generation.
• Baselines: Ten established metrics including F1@M, NDCG, AP, R-Precision, SemP/R/F1, and BERTScore variants.
• Parameter Tuning: The parameter $k$ (number of top similar references to average) was tested with values $\{1, 2, 3\}$.

3. Key Contributions

1. Formal Definition: A rigorous definition of SemR-p that balances exact lexical matching with phrase-level semantic similarity within a rank-aware structure.
2. Comprehensive Empirical Analysis: Extensive evaluation across multiple models and datasets to assess semantic sensitivity, ranking awareness, and discriminative power.
3. Qualitative Case Studies: Diagnostic examples demonstrating how SemR-p diverges from baselines in specific scenarios (e.g., rewarding semantic near-misses that exact matchers miss, or penalizing correct items placed too low in the list).
4. Factor Analysis: Identification of latent dimensions in keyphrase evaluation, positioning SemR-p as a "bridging metric" between traditional effectiveness and contextual semantic similarity.

4. Results

• Parameter Sensitivity ($k$): Increasing $k$ from 1 to 3 lowered absolute scores (as more diverse references were averaged) but improved alignment with a broader range of metrics (lexical, ranking, and semantic). The authors recommend $k=3$ as the default setting for a balanced evaluation.
• Discriminative Power: ANOVA results confirmed that SemR-p is statistically sensitive to differences between models (explaining 16% of variance) and across datasets, validating its utility for distinguishing system performance.
• Ranking Agreement: SemR-p showed positive correlation with all baseline metrics but distinct behavior compared to BERTScore variants. It aligns strongly with rank-aware metrics (NDCG, AP) while maintaining semantic sensitivity.
• Factor Analysis: Two latent factors were identified:
  1. General Effectiveness & Ranking: Dominated by exact-match and rank-based metrics. SemR-p loads heavily here (0.77), indicating it retains rank sensitivity.
  2. Contextual Semantic Similarity: Dominated by BERTScore variants. SemR-p shows moderate loading (0.47), indicating it incorporates semantic information without losing its rank-aware nature.
• Qualitative Findings:
  • SemR-p successfully rewards semantically relevant predictions that fail exact matching (unlike F1@M).
  • It penalizes models that place relevant keyphrases low in the list (unlike F1@M or BERTScore).
  • It can be stricter than approximate matchers (like R-Precision) by penalizing non-specific substrings that lack semantic depth.

5. Significance and Claims

The paper claims that SemR-p offers a complementary lens for evaluating keyphrase predictions. Its significance lies in:

• Human-Centric Alignment: It shifts evaluation from surface-level lexical matching to assessing semantic equivalence while preserving the user-centered perspective that top-ranked results are most critical.
• Bridging Dimensions: It addresses the gap between traditional ranking metrics and modern semantic similarity measures, providing a more holistic assessment of model performance.
• Practical Utility: By rewarding semantically relevant keyphrases that appear early in the output, it better reflects practical application needs where users prioritize top results.

The authors maintain a modest stance, noting that while SemR-p aligns with cognitive principles, its direct correlation with human judgments has not yet been empirically validated in this work. They also acknowledge limitations regarding semantic redundancy and the dependency on specific embedding models, suggesting future work on diversity-aware variants and dynamic parameter selection.