💬 NLP

Evaluating the Homogeneity of Keyphrase Prediction Models

This paper introduces a method to evaluate the homogeneity of keyphrase prediction models and reveals that, contrary to intuition, the ability to generate absent keyphrases does not improve homogeneity and can actually negatively impact it, with extraction methods proving competitive with generative models.

Original authors: Maël Houbre, Florian Boudin, Beatrice Daille

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maël Houbre, Florian Boudin, Beatrice Daille

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the librarian of a massive, chaotic library. Every day, thousands of new books arrive. Your job is to write a few "tags" (keyphrases) on each book so that when someone asks, "Where are the books about space travel?", you can find them instantly.

For a long time, librarians (and computer models) had two main ways to do this:

The "Scavenger" (Extraction): This librarian only writes tags using words they can find exactly inside the book. If the book says "The rocket ship flew to Mars," the tag must be "rocket ship" or "Mars." If the book talks about "space travel" but never uses those exact words, the Scavenger misses the tag.
The "Dreamer" (Generation): This librarian reads the book, understands the idea, and writes tags even if the words aren't there. If the book is about a rocket to Mars, the Dreamer might write "space travel" or "interplanetary exploration," even if the text only said "rocket ship."

The Big Question
The authors of this paper asked a very specific question: Which librarian is more consistent?

Imagine you have two different books about "Space Travel."

Book A says: "The rocket ship flew to Mars."
Book B says: "We launched a spacecraft to the Red Planet."

If you ask the Scavenger to tag both:

Book A gets: "rocket ship," "Mars."
Book B gets: "spacecraft," "Red Planet."
Result: The tags are totally different, even though the books are about the same thing. The Scavenger is inconsistent.

If you ask the Dreamer to tag both:

They should both get: "Space Travel," "Mars Mission."
Result: The tags are the same. The Dreamer is consistent (or "homogeneous," as the paper calls it).

The Surprise Twist
The researchers built a test to see if the "Dreamers" (Generative AI models) were actually better at being consistent than the "Scavengers" (Extraction models). They created pairs of similar documents and asked various AI models to tag them.

Here is the plot twist: The Dreamers were not always more consistent. In fact, sometimes they were less consistent than the Scavengers!

Why did this happen?

The paper uses a few clever analogies to explain this:

1. The "Too Many Choices" Problem
The Scavenger is limited. They can only pick words that are already on the page. It's like a chef who can only use ingredients sitting on the counter. They might make a slightly different dish every time, but the ingredients are always familiar.

The Dreamer, however, has access to the entire library's vocabulary (tens of thousands of words). It's like a chef who can go to a massive supermarket. While this allows them to create amazing new dishes (tags like "interplanetary travel"), it also means they might get distracted.

For Book A, the Dreamer might think, "Oh, let's call this 'Rocketry'."
For Book B, the Dreamer might think, "No, let's call this 'Spaceflight'."
Both are correct, but they aren't the same tag. The freedom to choose from a huge vocabulary actually made them less consistent.

2. The "Explicit vs. Implicit" Link
The researchers found that the answer depends on how the two books are similar.

Scenario A (The "Copy-Paste" Test): If the two books are very similar in their wording (like a book and a slightly rewritten version of itself), the Scavenger wins. Because the words are right there, it's easy to grab the same tag twice.
Scenario B (The "Deep Meaning" Test): If the two books are about the same topic but use totally different words (like a technical manual vs. a news article), the Dreamer wins. The Scavenger fails because it can't find the connecting words. The Dreamer succeeds because it understands the concept.

The Takeaway

The paper concludes that we shouldn't just assume "Generative AI" (the Dreamers) is automatically better at everything.

If you want consistency on documents that look similar, Extraction (Scavengers) is surprisingly strong.
If you want to connect documents that are conceptually similar but look different, Generation (Dreamers) is necessary, but we need to teach them to be more consistent.

The Future Solution?
The authors suggest a Hybrid Librarian. Imagine a robot that first uses the Scavenger to grab the obvious tags from the text, and then uses those tags to help the Dreamer come up with the "big picture" tags. This way, you get the best of both worlds: the consistency of the Scavenger and the creativity of the Dreamer.

In short: Being able to invent new tags is a superpower, but without training, it can make your library's filing system messy. Sometimes, sticking to the words you know is actually the most reliable way to keep things organized.

1. Problem Statement

Keyphrase prediction is a critical task for document indexing, summarization, and information retrieval. Models are generally categorized into two types:

Extraction Models: Select keyphrases that appear explicitly in the source text.
Generation Models: Can generate "absent keyphrases" (concepts not explicitly mentioned in the text), allowing them to associate a document with broader notions.

While generation models often outperform extraction models in standard metrics (like F1-score) and extrinsic retrieval tasks, a crucial property of indexing quality has been overlooked: Homogeneity. Homogeneity refers to the model's ability to consistently assign the same keyphrase to the same concept across different documents.

The authors hypothesize that generation models, by virtue of their ability to predict absent keyphrases, should theoretically offer better homogeneity than extraction models (which are limited to the specific vocabulary of each document). However, current benchmarks do not evaluate this property, leaving a gap in understanding whether generative capabilities actually improve indexing consistency.

2. Methodology

Evaluation Protocol

The authors propose a method to measure homogeneity by creating pairs of documents that share the same core concepts but differ in their textual expression. They evaluate the consistency of the predicted keyphrases for both documents in a pair using two metrics derived from inter-annotator consistency studies:

Hooper's CP: Measures the Jaccard index (Intersection over Union) at the keyphrase level.
Rodgers' CP: Measures the Jaccard index at the word level (more permissive).

Data Construction

To test homogeneity, two distinct datasets of document pairs were constructed:

Reformulation Pairs (Inspec Dataset):
- Original documents were paraphrased or back-translated (English $\to$ Spanish $\to$ English) using GPT-4o.
- Goal: Create semantically identical documents with significant lexical variation.
- Filtering: Ensured present keyphrases were preserved (either as exact matches or reordered words) and word count variation was within $\pm 10\%$ .
- Characteristics: High lexical overlap; majority of keyphrases are "present."
Shared Keyphrase Pairs (KP20k Dataset):
- Documents were paired based on a high Jaccard similarity of their author-assigned keyphrases (threshold $\ge 50\%$ ).
- Goal: Create documents sharing abstract concepts, even if the text differs significantly.
- Characteristics: Lower lexical overlap; majority of keyphrases are "absent" or "mixed" (concepts not fully explicit in the source text).

Models Evaluated

Generative Models (8): CopyRNN, CorrRNN, TG-Net, TGRF, BART, BART-60p (augmented), One2Set, One2Set-60p (augmented).
Extraction Models (2): MultipartiteRank, TF-IDF.
Training: Most generative models were trained on KP20k. Some (BART-60p, One2Set-60p) utilized self-compositional data augmentation to enhance absent keyphrase generation.

3. Key Contributions

New Evaluation Metric: Introduced a formal method to evaluate indexing homogeneity, a property previously ignored in keyphrase prediction benchmarks.
Counter-Intuitive Findings: Demonstrated that the ability to generate absent keyphrases does not automatically lead to higher homogeneity. In fact, in lexically similar contexts, generation models often perform worse than extraction models.
Context-Dependent Performance: Showed that the superiority of a model type depends on the nature of the document relationship:
- Lexical Similarity: Extraction models are superior.
- Semantic/Abstract Similarity: Generation models are superior.
Open Resources: Released the constructed document pairs, code, and prompts on Hugging Face and GitHub.

4. Results and Analysis

Scenario A: Reformulation Pairs (Lexically Similar)

Finding: Extraction models outperformed most generation models.
- TF-IDF (Extraction) achieved the second-highest Hooper's CP score (49.3%), outperforming 7 out of 8 generative models.
- The best generative model (TGRF) scored 57.6%, but this was only slightly better than the worst extraction model.
Correlation: There was a weak correlation between the amount of lexical variation (ROUGE-1) and homogeneity scores for generative models, suggesting that lexical variation alone does not explain the low homogeneity.
Hypothesis: Generative models suffer from a "wide vocabulary" problem. Unlike extraction models constrained to the source text, generative models sample from a massive vocabulary (tens of thousands of tokens), introducing variability in how they express the same concept.
Anomaly: One2Set-60p (augmented) performed relatively well (48.2%), suggesting that specific training paradigms (bipartite pairing) might mitigate homogeneity issues.

Scenario B: Shared Keyphrase Pairs (Semantically Similar/Absent Concepts)

Finding: Generation models outperformed extraction models.
- Extraction models (TF-IDF, MultipartiteRank) performed poorly (Hooper's CP < 15%) because they cannot generate concepts absent from the text.
- Generative models showed significant gains. For example, BART-60p improved by 3.6 points over base BART, and One2Set-60p improved by 17% over base One2Set.
Insight: When the link between documents is implicit (relying on broad, abstract concepts like "performance" or "algorithms" rather than specific text matches), the ability to generate absent keyphrases is essential for homogeneity.

5. Significance and Conclusion

The paper challenges the assumption that "generation is always better" for indexing. It reveals a trade-off:

Extraction models provide high consistency (homogeneity) when documents are textually similar because they are constrained to the source vocabulary.
Generation models provide high coverage (ability to index absent concepts) but suffer from variability when the text is similar, likely due to the stochastic nature of sampling from a large vocabulary.

Implications:

Future keyphrase prediction systems should consider hybrid approaches: using extraction to ensure consistency for explicit concepts and generation to capture abstract, absent concepts.
Evaluation benchmarks must move beyond simple F1-score matching to include homogeneity and consistency metrics to ensure reliable automated indexing.
The choice of model should be dictated by the specific indexing goal: strict lexical matching vs. abstract semantic retrieval.