TailNLG: A Multilingual Benchmark Addressing Verbalization of Long-Tail Entities

1. Problem Statement

Automatic verbalization of structured knowledge (Data-to-Text) is essential for making Knowledge Graphs (KGs) accessible to non-experts and for Retrieval-Augmented Generation (RAG) systems. While recent advances in Large Language Models (LLMs) have improved multilingual coverage, there is a critical gap in understanding how these models handle long-tail entities (rare, low-frequency entities).

Existing benchmarks, such as WebNLG, primarily focus on well-known ("head") entities. The authors hypothesize that LLMs exhibit systematic biases against long-tail entities, leading to:

• Lower generation quality.
• Higher model uncertainty.
• Inconsistent performance across languages.
• A lack of reliable evaluation metrics that can detect these specific biases.

2. Methodology

A. The TailNLG Benchmark

The authors introduce TailNLG, the first multilingual benchmark specifically designed to evaluate the verbalization of long-tail entities.

• Data Source: Extracted from Wikidata, covering three languages: English, Italian, and Spanish.
• Entity Selection:
  • Head Entities: Entities with a high number of claims (relations), falling within the top 20% of entities that account for 80% of all claims (Pareto cut-off).
  • Long-Tail Entities: Entities below the Pareto cut-off. To ensure they are truly rare, the authors filtered for entities that have a Wikipedia page in only one of the three target languages.
• Data Structure: The dataset consists of RDF triples (Subject, Predicate, Object) organized into "data units" with varying structural complexities (Chain, Sibling, Mixed).
• Dataset Composition:
  • Total: 1,845 data units.
  • Gold Standard: 616 manually verbalized sentences by native speakers.
  • Silver Standard: 1,232 sentences generated via Machine Translation (DeepL) and refined using Automatic Post-Editing (APE) with models like xCOMET and xTOWER.
  • Distribution: 1,224 long-tail entities and 654 head entities.

B. Experimental Setup

• Models Evaluated: Three families of open-source, instruction-tuned multilingual LLMs in zero-shot settings:
  • Llama-3 (3B and 8B parameters)
  • Qwen2.5 (3B and 7B parameters)
  • Gemma-3 (4B and 12B parameters)
• Task: Convert RDF triples into a single-paragraph natural language description.
• Baselines: Performance was compared against the established WebNLG benchmark (head entities) and within the TailNLG corpus (comparing long-tail vs. head).

C. Evaluation Metrics

The study employs a diverse set of metrics to capture different aspects of performance:

1. Semantic Similarity: BERTScore (rescaled with a multilingual baseline).
2. Surface-Level Overlap: BLEU, chrF++, and ROUGE (1, 2, L).
3. Model Uncertainty: Perplexity (PPL), where lower PPL indicates higher confidence. This is used as a proxy to measure how "surprised" the model is by the entity.
4. Statistical Analysis: Mann-Whitney U and Wilcoxon tests to determine significance.

3. Key Contributions

1. TailNLG Benchmark: The first systematic, multilingual dataset dedicated to long-tail entity verbalization, bridging the gap between standard RDF-to-Text tasks and the reality of rare entities in KGs.
2. Systematic Bias Discovery: Empirical evidence demonstrating that LLMs consistently underperform on long-tail entities compared to head entities, regardless of the model family.
3. Metric Analysis: A critical analysis showing that standard overlap-based metrics (BLEU, ROUGE) often fail to capture the nuances of long-tail bias, sometimes even favoring long-tail outputs due to lexical divergence, whereas Perplexity (PPL) provides a more consistent signal of model difficulty.
4. Multilingual Insights: Identification of language-specific biases, where English generally yields better results than Italian or Spanish, likely due to morphological complexity and training data distribution.

4. Key Results

A. Performance on Long-Tail vs. Head Entities (RQ1)

• Semantic Similarity: Embedding-based metrics (BERTScore) are consistently lower for long-tail entities, indicating that the generated text is semantically further from the reference.
• Overlap Metrics: Results are inconsistent. In some cases, overlap metrics (BLEU, ROUGE) are higher for long-tail entities. The authors attribute this to "verbosity": models tend to generate more verbose or divergent sentences for rare entities, which can accidentally increase n-gram overlap with references or simply fail to match the concise gold standard.
• Model Uncertainty (PPL): Long-tail entities consistently yield higher PPL scores (indicating higher uncertainty) for Llama and Qwen models. This confirms that the models find rare entities harder to predict.
• Language Bias: English outputs generally achieve higher scores and lower PPL than Italian and Spanish. Italian outputs, in particular, showed issues with grammatical agreement and gender, suggesting models struggle more with morphologically rich languages when dealing with rare entities.

B. Comparison with WebNLG (RQ2)

• Models perform significantly better on the WebNLG benchmark (head entities) than on TailNLG (long-tail).
• PPL is significantly lower for WebNLG, suggesting that models have seen these entities frequently during pre-training, whereas long-tail entities are effectively "out-of-distribution" or under-represented.
• Even when comparing WebNLG (gold standard) against the long-tail portion of TailNLG (mixed gold/silver), models still outperform on WebNLG.

C. Gold vs. Silver Data

• Surprisingly, models often achieved higher scores on Silver (machine-translated) data than Gold (human-written) data.
• This suggests that LLMs are better aligned with the patterns found in machine-translated text, potentially because the models themselves were trained on similar synthetic data. This implies that for downstream applications, silver-standard data might be sufficient, though it masks the true difficulty of the task.

5. Significance and Future Work

• Reliability of RAG: The findings highlight a critical risk for Retrieval-Augmented Generation systems. If an LLM cannot reliably verbalize rare entities retrieved from a KG, the safety and accuracy of the generated response are compromised.
• Evaluation Frameworks: The paper argues for a shift in evaluation. Relying solely on BLEU or ROUGE is insufficient for detecting bias against long-tail entities. Perplexity and embedding-based metrics are proposed as more reliable indicators of model confidence and semantic alignment.
• Fairness in NLP: The study underscores that current LLMs perpetuate a "rich-get-richer" dynamic in knowledge representation, where well-known entities are handled well, but rare (often culturally specific or less documented) entities are marginalized.

Future Directions: The authors plan to expand TailNLG to more languages, incorporate culturally specific data units beyond just claim counts, and develop gold standards that reflect diverse human annotator perspectives to better measure human label variation.