NLG Evaluation: Past, Present, Future

Technical Summary: NLG Evaluation: Past, Present, Future

Problem Statement
The paper addresses the dramatic evolution and current state of Natural Language Generation (NLG) evaluation. Historically, NLG research relied on qualitative linguistic or engineering arguments rather than quantitative experimental validation. As the field has shifted from rule-based systems to Large Language Models (LLMs) by 2026, traditional evaluation methods (such as reference-based metrics like BLEU/ROUGE and standard human Likert scales) have become insufficient. New challenges include the high quality of LLM outputs (rendering reference comparisons less meaningful), data contamination (memorization of test sets), the need for semantic and pragmatic assessment over fluency, and the critical requirement for safety and worst-case scenario analysis in safety-critical domains like medicine. Furthermore, the research culture often prioritizes rapid publication over experimental rigor, leading to issues with replicability, construct validity, and benchmark saturation.

Methodology
This paper employs a historical and analytical review methodology, tracing the trajectory of NLG evaluation from 1990 to a projected 2036. The author synthesizes observations from conference proceedings (specifically INLG and ACL), analyzes trends in evaluation techniques across different eras, and reviews existing literature on evaluation challenges and best practices. The analysis is structured chronologically:

• 1990: Characterized by the absence of quantitative hypothesis testing.
• 2000: Emergence of a wide mix of techniques (human ratings, metrics, task-based) without standardization.
• 2010: Adoption of shared tasks and standardized metrics (BLEU, ROUGE) alongside human evaluation.
• 2020: Evaluation becoming a distinct research sub-field with a focus on methodology.
• 2026: The current state dominated by LLMs, introducing "LLM-as-Judge," expert human annotation, private test data, and interdisciplinary safety evaluations.
• 2036 (Projection): A future shift toward impact, qualitative, and safety evaluations.

Key Contributions

1. Historical Taxonomy: The paper provides a structured timeline (summarized in Table 1) documenting the shift from qualitative linguistic justification to rigorous experimental evaluation, and finally to the complex, interdisciplinary evaluation landscape of the LLM era.
2. Identification of LLM-Specific Challenges: It details specific hurdles introduced by LLMs, including:
   • The inadequacy of reference-based metrics for high-quality, potentially human-level or super-human text.
   • The risk of data contamination rendering public test sets obsolete.
   • The necessity of evaluating semantic accuracy, hallucinations, and pragmatic appropriateness rather than just fluency.
   • The need for "worst-case" safety evaluations (e.g., red teaming) rather than just average-case performance.
3. Critique of Current Research Culture: The paper highlights systemic issues such as poor experimental design, lack of replicability, construct validity failures in benchmarks, "reward hacking" by models, and commercial bias in evaluation funding.
4. Future Roadmap: It proposes a shift in evaluation paradigms for the next decade (2036), arguing that the community must move beyond test-set performance to:
   • Impact Evaluation: Measuring real-world changes in Key Performance Indicators (KPIs) using methods like Randomized Controlled Trials (RCTs) and A/B testing.
   • Qualitative Evaluation: Utilizing focus groups, interviews, and thematic analysis to understand system behavior in messy, complex real-world contexts.
   • Safety Evaluation: Prioritizing the identification of harmful behaviors and adversarial vulnerabilities, drawing on methodologies from medicine and cybersecurity.

Results and Observations

• Evolution of Rigor: In 1990, no structured quantitative hypothesis tests were found in INLG papers. By 2020, evaluation had become a primary research agenda, with best papers dedicated to evaluation methodology.
• Current State (2026): The field is currently grappling with the "LLM-as-Judge" paradigm, where LLMs evaluate other LLMs, though this lacks universal validation. Human evaluation is shifting from crowd-sourced Likert scales to expert annotation of specific errors.
• Interdisciplinary Divergence: Evaluations in fields like medicine are beginning to use clinical trial methodologies (e.g., RCTs), which often yield different results than classical NLP metrics, suggesting that current NLP evaluation may not fully capture real-world utility.
• Cultural Deficits: The paper observes that the NLP research culture often discourages high-quality evaluation due to publication pressures, leading to "quick and dirty" studies and a lack of support for replication.

Significance
The paper argues that as NLG technology becomes ubiquitous and integrated into critical real-world applications (medicine, law, etc.), the definition of "successful" evaluation must expand. The significance lies in the call to action for the NLG community to adopt evaluation frameworks used in other high-stakes disciplines. The author posits that while performance evaluation remains relevant for developers, safety, impact, and qualitative evaluation are becoming more critical for society at large. The paper serves as a motivational guide to evolve evaluation practices from mere benchmarking to holistic assessments of system utility, safety, and real-world consequence, ensuring that NLG research remains robust and relevant in an era of widespread AI deployment.