LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

1. Problem Statement

Long-Form Question Answering (LFQA) requires models to generate detailed, multi-sentence explanatory responses. However, evaluating these responses is challenging due to several limitations in current methodologies:

• Inadequacy of Reference-Based Metrics: Traditional metrics like ROUGE, BLEU, and even learned models like BERTScore rely on lexical overlap or fixed human references. They fail to capture semantic equivalence, paraphrasing, and the nuanced quality of long-form explanations, especially under domain adaptation.
• Lack of Standardized Definitions: There is no unified definition of what constitutes a "long-form question," leading to inconsistent dataset construction and evaluation.
• Black-Box Evaluation: The emerging "LLM-as-a-Judge" paradigm offers reference-free evaluation but lacks interpretability. These models often suffer from biases (positional, verbosity) and lack transparency regarding why one answer is preferred over another.
• Data Scarcity: There is a shortage of large-scale, high-quality human preference datasets specifically curated for LFQA to train and benchmark evaluators.

2. Methodology

The authors propose a comprehensive framework (illustrated in Figure 1 of the paper) involving dataset construction, rubric definition, and model evaluation.

A. Dataset Construction (LFQA-HP-1M)

• Source Aggregation: The dataset aggregates human preference data from three heterogeneous sources: Chatbot Arena (conversational), LFQA Eval (reasoning-focused), and SHP-2 (Stanford Human Preferences, derived from Reddit/StackExchange).
• Rigorous Filtering: To ensure data quality, the authors developed a strict operational definition of LFQA. A valid LFQA must:
  • Express a single, well-defined information need.
  • Require a detailed, multi-sentence answer involving exposition, reasoning, or process description.
  • Exclude: Personal advice, simple yes/no questions, creative writing, code generation, or single-fact answers.
• Automated Filtering Pipeline: They trained LLMs (GPT-4o, Llama-4, Gemini-2.5) to filter non-LFQA prompts using the new definition. To balance cost and performance, they utilized Llama-4 with a batch size of 10 and a confidence filter (log_score = 0), achieving high precision at a fraction of the cost of GPT-4o.
• Result: The final dataset, LFQA-HP-1M, contains 1.3 million human pairwise preference annotations across 300,000 unique questions spanning 129 domains.

B. Rubric-Based Evaluation Framework

To address the "black box" nature of LLM judges, the authors defined nine fine-grained rubrics for answer quality:

1. Specificity: Precision and depth of explanation.
2. Grammar: Adherence to formal writing style.
3. Fluency: Clarity of communication.
4. Completeness: Coverage of all query parts.
5. Coherence: Logical organization.
6. Relevance: Direct addressing of the question.
7. Conciseness: Succinctness without redundancy.
8. Use of Examples: Presence of concrete supporting examples.
9. Factuality: Accuracy of verifiable claims.

Implementation:

• Most rubrics are computed using established non-LLM tools (e.g., spaCy for grammar, Veriscore for factuality).
• Complex dimensions (Coherence, Fluency, Relevance, Completeness) are evaluated using G-EVAL (an LLM-driven framework).
• Scores are normalized to a 1–5 scale (or binary) to create a feature vector for each answer pair.

C. Model Evaluation & Robustness Testing

The authors benchmarked the performance of Logistic Regression (LR) models (trained on the rubric features) against state-of-the-art LLM-as-a-Judge models (GPT-4o, Gemini-2.5, Llama-4).

• Bias Analysis: They tested for Position Bias (does answer order matter?) and Verbosity Bias (does length correlate with preference?).
• Transitivity Check: They evaluated logical consistency (if A > B and B > C, does A > C hold?).
• Adversarial Robustness: They applied TextFooler (synonym swaps) and DeepWordBug (character-level perturbations) to test if evaluators rely on superficial lexical patterns rather than semantics.

3. Key Contributions

1. LFQA-HP-1M Dataset: The creation of the largest public LFQA preference dataset to date (1.3M pairs), enabling large-scale empirical analysis of human judgment.
2. Operational LFQA Definition: A rigorous, annotated definition and a scalable filtering pipeline to distinguish true long-form questions from other text types.
3. Rubric-Driven Framework: A transparent, interpretable evaluation system using nine specific quality dimensions.
4. Empirical Insights: Demonstration that simple linear models based on these rubrics can match the performance of complex, closed-source LLMs while offering interpretability and resistance to specific biases.

4. Key Results

• Performance Parity: A lightweight Logistic Regression model trained on the nine rubrics achieved an accuracy of 0.6826 and F1-score of 0.6817. This is comparable to Gemini-2.5 (0.6826) and Llama-4 (0.6840), and only marginally lower than the best-performing GPT-4o (0.6943).
• Feature Importance: The LR model weights revealed that Coherence (0.5910) and Completeness (0.4065) are the strongest predictors of human preference. Interestingly, Relevance and Grammar showed negative weights, suggesting that in pairwise comparisons, these factors are often constant or that overly verbose "relevant" answers are penalized.
• Bias Vulnerability:
  • Position Bias: Llama-4 and Gemini-2.5 showed significant performance drops (~2%) when answer positions were swapped. GPT-4o was relatively stable.
  • Verbosity Bias: GPT-4o and Gemini-2.5 exhibited a "Long-biased" tendency, preferring longer answers even when humans preferred concise ones. Llama-4 showed a more balanced preference.
• Transitivity: LLM judges violated transitivity in 4.7% to 15.3% of cases (GPT-4o being the most consistent, Gemini-2.5 the least), indicating logical inconsistencies in their ranking.
• Adversarial Robustness: LLMs are highly vulnerable to orthographic perturbations. DeepWordBug (character-level attacks) caused a ~10% performance drop in GPT-4o, whereas semantic synonym swaps (TextFooler) caused only a ~2.6% drop. This suggests LLMs rely heavily on surface-level text patterns.

5. Significance

This work fundamentally shifts the paradigm of LFQA evaluation from "black-box" LLM scoring to transparent, rubric-driven assessment.

• Cost-Effectiveness: It proves that expensive, closed-source LLMs are not strictly necessary for high-quality preference modeling; a simple linear model with well-defined features can achieve comparable results.
• Reliability: By identifying and quantifying biases (position, verbosity) and logical inconsistencies (transitivity), the paper provides a roadmap for building more reliable evaluation pipelines.
• Interpretability: The rubric framework allows researchers and developers to understand why an answer is preferred (e.g., "it was more coherent" rather than just "it scored higher"), facilitating better model debugging and training.
• Resource Availability: The release of LFQA-HP-1M and the associated code provides a critical foundation for future research into long-form generation and evaluation.