Post-training Large Language Models for Diverse High-Quality Responses

1. Problem Statement

While Reinforcement Learning (RL) is the standard for post-training Large Language Models (LLMs) to align with human preferences and improve task performance, it suffers from a critical side effect: output homogenization.

• The Issue: RL optimization often causes models to converge on a narrow set of "canonical" responses, drastically reducing output diversity. This limits reasoning capabilities (by restricting alternative solution paths), personalization, and robustness to distribution shifts.
• Limitations of Existing Solutions:
  • Inference-time interventions (e.g., temperature scaling, top-k sampling) often introduce only shallow, token-level variations and can degrade output quality or coherence. They cannot recover semantic modes absent from the base model's distribution.
  • Training-time methods largely focus on lexical diversity (e.g., token-level entropy regularization or pairwise word differences). These fail to capture semantic diversity, which is more meaningful to humans. Furthermore, simple pairwise distance metrics are prone to "degenerate clustering," where responses form distinct but internally redundant clusters, misleadingly appearing diverse.

2. Methodology: Diversity Quality Optimization (DQO)

The authors propose DQO, a principled training framework that jointly optimizes for response quality and semantic diversity using Determinantal Point Processes (DPPs).

Core Concept: Semantic Diversity via DPPs

Instead of measuring diversity at the token level, DQO operates in the semantic embedding space.

1. Sampling: For a given prompt $x$, the policy $\pi_\theta$ samples a group of $k$ responses $\{y_1, \dots, y_k\}$.
2. Embedding: These responses are mapped to a high-dimensional semantic space using a pre-trained encoder $\phi(\cdot)$.
3. Similarity Matrix: A kernel-based similarity matrix $L$ is constructed where $L_{ij} = f(\phi(y_i), \phi(y_j))$. The authors primarily use the dot product kernel.
4. Diversity Metric: The diversity score is defined as the determinant of this matrix:
   $$\text{Div}(y_{1:k}) = \det(L_\phi(y_{1:k}))$$
   • Geometric Interpretation: The determinant corresponds to the squared volume spanned by the response embedding vectors. Maximizing this volume forces the model to generate responses that are not only distinct but also linearly independent, effectively covering the semantic space rather than forming tight clusters. This overcomes the limitations of pairwise distance metrics.

The Objective Function

DQO modifies the standard RL objective (e.g., PPO or GRPO) to include a diversity term. The new objective $J_{\text{Div}}$ is:
$$J_{\text{Div}}(\pi_\theta) = \mathbb{E}_{x, y_{1:k} \sim \pi_\theta} \left[ \sum_{i=1}^k r(x, y_i) + \alpha \log \det(L_\phi(y_{1:k}) + I_k) - \beta \text{KL}(\pi_\theta || \pi_{\text{ref}}) \right]$$

• Reward ($r$): Represents response quality.
• $\alpha$: A hyperparameter balancing quality and diversity.
• $\log \det(L + I_k)$: The diversity term. Adding the identity matrix $I_k$ acts as a regularizer to prevent numerical instability (log of zero) and ensures the term is bounded.
• Reward-Augmented Embeddings: Theoretically, the optimal policy selects groups of responses proportional to the determinant of a Gram matrix constructed from reward-augmented embeddings $\psi(x, y) = \sqrt{\exp(r(x,y)/\alpha)} \cdot \phi(y)$. This creates a geometric trade-off: the model seeks vectors that are both long (high reward/quality) and widely separated (diverse).

Algorithmic Stability

Direct optimization of the determinant poses challenges:

• Numerical Instability: $\log(\det(L))$ can be unbounded if $L$ is singular.
• High Variance: The gradient involves the sum of $k$ responses.
• Solution:
  1. Identity Regularization: Use $\det(L + I_k)$ instead of $\det(L)$.
  2. Leave-One-Out (LOO) Estimator: To reduce gradient variance, the diversity term is computed as a ratio: $\log \frac{\det(L(y_{1:k}) + I_k)}{\det(L(y_{-i}) + I_{k-1})}$. This estimator is unbiased and bounded, making training stable even for larger $k$.

3. Key Contributions

1. Principled Framework (DQO): A flexible algorithm applicable on top of existing RL methods (PPO, GRPO) that optimizes for both quality and semantic diversity.
2. Semantic vs. Lexical: Demonstrates that DPP-based formulation captures meaningful semantic diversity, avoiding the pitfalls of token-level entropy or pairwise distance metrics.
3. Quality-Diversity Trade-off: Shows that DQO can significantly enhance diversity without sacrificing task performance (quality), achieving a superior Pareto frontier compared to baselines.
4. Theoretical Connection: Links the optimization objective to D-Optimal Design in experimental design, framing the selection of diverse responses as maximizing information gain in the embedding space.

4. Experimental Results

The authors evaluated DQO across four task categories: Instruction-following (Dolly), Summarization (CNN-DailyMail), Story Generation (CommonGen), and Reasoning (GSM8K).

• Performance vs. Baselines:

  • DQO consistently outperformed reward-only baselines (PPO/GRPO) and other diversity-aware methods (GRPO-likelihood, GRPO-entropy) in both diversity and quality metrics.
  • Diversity Metrics: Measured via Distinct-n, Self-BLEU, Self-ROUGE, and LLM-as-a-Judge. DQO showed substantial gains (e.g., +61.7% diversity improvement in reasoning tasks).
  • Quality Metrics: Measured via Pass@n (the probability that at least one of $n$ samples is correct). DQO maintained or improved Pass@1 performance while significantly boosting Pass@n, indicating that the model generates a wider variety of correct solutions.

• Ablation Studies:

  • Hyperparameters ($\alpha, k$): The method is robust across a range of $\alpha$ (diversity weight) and $k$ (sample size) values. Increasing $\alpha$ or $k$ generally boosts diversity with minimal quality degradation.
  • Embedding Models: DQO performs well with various embedding models (e.g., MiniLM, MPNet, GTE), showing robustness to the choice of semantic encoder.
  • Kernel Functions: Both dot-product and Gaussian kernels yielded similar results.

• Case Study (City Recommendation):

  • Reward-only models converged on a single city (Tokyo).
  • Pairwise-distance methods converged on two clusters.
  • DQO (Determinant) successfully distributed recommendations across a wide variety of cities, demonstrating true semantic coverage.

• Reward Hacking: The paper notes that using outcome-based rewards (e.g., binary correct/incorrect for math) can lead to reward hacking (generating a correct answer followed by random text to boost diversity). DQO mitigates this by using a Reward Model that evaluates the entire response, ensuring quality is maintained across the whole generation.

5. Significance and Conclusion

This paper addresses a fundamental bottleneck in LLM post-training: the trade-off between alignment (quality) and diversity. By leveraging the mathematical properties of Determinantal Point Processes, DQO provides a theoretically grounded mechanism to encourage models to explore the full semantic space of possible answers.

• Practical Impact: DQO enables LLMs to be more creative, personalized, and robust, which is crucial for applications like creative writing, brainstorming, and complex reasoning where multiple valid paths exist.
• Efficiency: The computational overhead is negligible (microseconds per step) compared to the cost of generating responses and running the reward model.
• Future Directions: The authors suggest that while DQO is effective, future work could focus on adaptive diversity measurement that adjusts to specific task requirements without relying solely on fixed embedding models.

In summary, DQO represents a significant step forward in making LLMs not just "correct," but diversely correct, preserving the richness of human language while maintaining high performance.