Technical Summary: Illusions of the Gold Standard

Problem Statement

Human evaluation remains the "gold standard" for assessing long-form and open-ended natural language generation (NLG), particularly in high-expertise domains where automated metrics fail. However, the reliability and reproducibility of these evaluations are compromised by a lack of transparent and well-documented protocols. Current practices often omit critical details regarding study design, annotation operationalization, and data analysis. Furthermore, as Large Language Models (LLMs) increasingly serve as "LLM-judges" for meta-evaluation, the foundational role of human judgment requires even greater rigor. Despite the existence of reproducibility frameworks in NLP, there is a lack of large-scale empirical evidence regarding the specific reporting norms and deficiencies within the *CL (Computational Linguistics) community.

Methodology

The authors conducted a large-scale analysis of human evaluation protocols in *CL conference publications from 2023 to 2025.

1. Criteria Definition and Codebook Development:
The team defined 20 core reportable criteria grounded in scientific reproducibility literature, categorized into three dimensions:

• Task Documentation: Evaluation dimensions, justifications, and task guidelines.
• Annotation Design: Annotator demographics, recruitment platforms, payment, sample size, and quality control measures.
• Analysis & Interpretation: Statistical metrics, inter-annotator agreement (IAA), disagreement resolution, and discussion of limitations.
  A codebook of 37 questions (including the 20 core criteria) was developed iteratively through inductive and deductive processes involving four authors.

2. Corpus Curation and Sampling:

• Initial Corpus: 9,172 papers from ACL, EMNLP, NAACL, EACL, and AACL (2023–2025).
• Filtering: A two-stage filtering process was employed. First, keyword matching identified papers on long-form generation. Second, a majority vote among three LLMs (Gemini-2.5-Pro, Claude-3.7-Sonnet, GPT-4o-mini) screened for long-form generation tasks involving human evaluation. This yielded 1,891 papers meeting inclusion criteria.
• Sampling: A stratified sample of 356 papers from 2024 and 2025 was selected for manual annotation.

3. Annotation and Analysis:

• Manual Annotation: 284 papers were fully annotated by five researchers (two lead authors and three contributors) using the codebook. Inter-annotator agreement (IAA) was established at 73% for binary questions after training.
• LLM-Assisted Annotation: The remaining papers in the filtered set were analyzed using GPT-4o-mini to extract relevant information, with validation accuracy thresholds set at >0.75 for reported results.
• Statistical Estimation: Bootstrap resampling (n=500) was used to estimate the true proportion of papers reporting each criterion.

Key Results

The analysis reveals pervasive under-reporting of essential elements required for reproducible human evaluation studies:

• Reporting Deficiencies: While most papers report evaluation dimensions (98%), annotator count (77%), and sample size (85%), critical details are frequently missing.
  • Task Guidelines: Only ~50% of papers provide guidelines for how annotators should judge dimensions.
  • Justification: Only ~50% justify the chosen evaluation dimensions, and only 25% justify them using prior work.
  • Annotator Details: 29% of papers report no demographic information; 65% omit recruitment platform details.
  • Quality Control: Only 6% of papers include data filtering steps (e.g., attention checks), and 22% report procedures to ensure annotation quality.
  • Statistical Rigor: Statistical metrics are reported in only 9% of papers. No papers were found to use power analysis to determine sample sizes.
• Distribution of Reporting: The modal number of reported criteria is 7 out of 20. Over half of the reviewed papers report 7 or fewer criteria. No paper reported all 20 criteria.
• Temporal Trends:
  • The proportion of papers studying long-form generation has increased (from ~30% to ~50% in 2025), but the proportion of these papers utilizing human evaluation has declined.
  • There is a rising trend in using human evaluation for the meta-evaluation of LLM-judges, yet reporting standards for these meta-evaluations have not shown corresponding improvement.
• Annotator Profiles: Among papers reporting demographics, 31% recruit students, 50% recruit domain experts, and 13% recruit paper authors.

Contributions

The paper makes three primary contributions:

1. Framework: It defines a set of 20 core reportable criteria and a corresponding codebook for assessing the reproducibility of human evaluation studies in NLP.
2. Empirical Analysis: It provides a large-scale, systematic analysis of reporting practices across 9,100+ papers, with deep manual annotation of 284 papers, establishing a baseline for current norms.
3. Recommendations: It outlines actionable recommendations to improve transparency, including a minimum reporting template and the need for higher standards in meta-evaluation contexts.

Significance and Claims

The authors argue that the current state of human evaluation reporting creates an "illusion" of rigor. The widespread under-reporting leads to ambiguity regarding what was measured, how it was measured, and how results should be interpreted. This lack of transparency limits the validity of conclusions, the robustness of findings, and the ability to compare studies.

The paper claims that as human evaluation increasingly serves as a meta-evaluative tool for LLM-judges, the community must hold these protocols to higher standards of documentation. The authors assert that modest changes in authoring norms—specifically adopting the proposed reporting criteria—can substantially improve the interpretability and reproducibility of NLP research without imposing prohibitive overhead. They emphasize that their findings represent a conservative estimate of the problem, as their annotation protocol was lenient (crediting any mention of a criterion regardless of sufficiency).