Explanation Quality Assessment as Ranking with Listwise Rewards

1. Problem Statement

The paper addresses a critical bottleneck in training Large Language Models (LLMs) to generate high-quality explanations: the failure of standard reward modeling to capture graded quality distinctions.

• The Limitation of Current Approaches: Most existing methods treat explanation evaluation as either a binary classification task (good/bad) or a pointwise regression task (predicting a scalar score).
• Score Compression: The authors identify "score compression" as a primary failure mode. Regression (MSE) and binary preference objectives tend to compress the predicted scores of diverse explanations into a narrow range.
• Impact on Policy Optimization: In Reinforcement Learning from Human Feedback (RLHF), specifically Proximal Policy Optimization (PPO), the reward signal drives policy updates. When score separation is minimal (e.g., a "gold" explanation and a "nonsense" explanation receive scores of 0.51 and 0.49), the advantage estimates become dominated by noise. This renders the gradient signal too weak for effective policy learning, causing PPO to fail or converge to suboptimal solutions.

2. Methodology

The authors propose reformulating explanation quality assessment as a Learning-to-Rank (LTR) problem rather than a generation or regression problem.

A. Graded Dataset Construction

To train ranking models, the authors constructed a dataset with five distinct quality tiers for each Natural Language Inference (NLI) instance:

1. Gold: Human-annotated explanations from e-SNLI (high inter-annotator agreement).
2. Good, Fair, Poor: Synthetic explanations generated via deterministic, label-aware templates that systematically degrade in logical consistency and relevance.
3. Nonsense: Semantically irrelevant text.

• Key Innovation: The authors introduced overlapping score ranges for adjacent tiers (e.g., "Good" overlaps with "Fair"). This emulates natural human disagreement and ambiguity, preventing the model from learning trivial boundaries and forcing it to learn nuanced features.

B. Ranking-Based Reward Modeling

Instead of predicting a single scalar score, the model learns to order a set of candidate explanations $\{e_1, ..., e_n\}$ for a given query $q$.

• Architecture: They utilize encoder models (BERT, RoBERTa, DeBERTa) and decoder models (Phi-2, Falcon, Mistral) with a projection head to output scores.
• Loss Functions: They compare standard objectives against ranking-specific losses:
  • MSE Regression: Pointwise prediction.
  • Binary Classification: Bradley-Terry preference model (pairwise).
  • ListNet: Listwise loss optimizing the probability distribution over the entire list.
  • RankNet: Pairwise loss.
  • ApproxNDCG: Optimizing Normalized Discounted Cumulative Gain.
• Integration with PPO: The ranking model serves as the reward estimator $f_\theta(q, e)$. The advantage function is calculated as $\hat{A}_t = f_\theta(q_t, e_t) - V_\phi(q_t)$. Because the ranking model preserves ordinal gaps, the advantage estimates reflect meaningful relative quality differences, providing stable gradients for PPO.

3. Key Contributions

1. Identification of Score Compression: The paper formally identifies score compression in regression and binary preference models as a root cause of PPO failure in explanation tasks, demonstrating that it destroys the learning signal.
2. Ranking as a Solution: They establish an end-to-end link showing that ranking objectives preserve score separation, leading to more informative advantage estimates and faster, more stable PPO convergence.
3. Data-Centric Insights: The study reveals that data quality and structure matter more than model scale. Small encoder models (e.g., 110M parameters) trained on carefully curated, graded data outperform or match much larger decoder models (7B parameters).
4. Loss Function Analysis: They demonstrate that the optimal ranking loss depends on data characteristics:
   • Listwise (ListNet) excels when quality tiers are well-separated.
   • Pairwise (RankNet) is more robust to noisy annotations but generally yields lower score separation than ListNet.
5. Graded Dataset Paradigm: They introduce a method to convert standard NLI benchmarks (which have zero score variance per query) into ranking benchmarks by injecting synthetic, graded candidates with overlapping score distributions.

4. Experimental Results

The experiments were conducted on 50,000 human-annotated e-SNLI explanations and validated across four NLI benchmarks (e-SNLI, MultiNLI, Delta-NLI, WinoWhy) and three commonsense reasoning tasks.

• Score Separation: ListNet achieved a Score Separation Ratio of 0.920, compared to 0.089 for MSE regression. This means ListNet preserved the relative distance between quality levels almost perfectly, whereas MSE collapsed them.
• Ranking Performance:
  • ListNet achieved near-perfect discrimination (NDCG@5 $\approx$ 1.0, Spearman $\rho \approx$ 0.92) across all model sizes.
  • MSE Regression struggled significantly (NDCG@5 $\approx$ 0.79).
• PPO Convergence:
  • ListNet-based PPO converged in 1,000 steps with a final explanation quality of 4.1/5.0 and only 9% errors.
  • MSE-based PPO failed to converge entirely.
  • Binary Classification required 3,200 steps and yielded lower quality (3.2/5.0).
• Efficiency: Despite the computational overhead of listwise softmax, ListNet was the most efficient end-to-end approach because the stronger gradient signals drastically reduced the number of training epochs and PPO steps required.
• Model Scale: Small encoders (RoBERTa-base, 125M) achieved near-perfect ranking performance, matching 7B parameter decoders, proving that high-quality structured data is the limiting factor, not model capacity.

5. Significance and Impact

• Paradigm Shift: The paper argues that explanation quality is inherently a ranking problem, not a generation or regression problem. This shifts the focus from "generating the best token" to "discriminating the best explanation."
• Stabilizing RLHF: By solving the score compression issue, this approach provides a robust mechanism for training LLMs via PPO in tasks requiring complex reasoning and justification, where sparse rewards typically fail.
• Cost-Effectiveness: The findings suggest that organizations can achieve state-of-the-art explanation quality using smaller, more efficient encoder models if they invest in constructing high-quality, graded ranking datasets, rather than relying solely on scaling up model parameters.
• Generalization: The method successfully transferred from NLI tasks to commonsense reasoning (WinoGrande, CommonsenseQA) without additional fine-tuning, indicating the learned ranking capabilities are domain-agnostic.

In conclusion, the paper demonstrates that preserving ordinal structure through listwise ranking losses is essential for effective policy optimization in explanation generation, offering a scalable and data-efficient path to improving LLM reasoning capabilities.