Technical Summary: PolyAlign – Conditional Human-Distribution Alignment

Problem Statement

Current post-training methods for Large Language Models (LLMs), such as Supervised Fine-Tuning (SFT) and preference optimization (e.g., DPO), typically align models toward a single global assistant behavior. While effective for improving average helpfulness, this approach often suppresses the natural variation of human responses across different languages, tasks, and dialogue settings. Standard alignment tends to collapse diverse interaction contexts into a generic assistant style, failing to capture how humans naturally vary their response style, length, discourse structure, and language use depending on the specific interaction regime.

The paper identifies the need for conditional human-distribution alignment: the goal is for models to match the human response distribution appropriate to the current interaction context, rather than optimizing for a universal response style.

Methodology: PolyAlign Framework

The authors propose PolyAlign, a distribution-aware alignment framework that organizes bilingual interaction data into "buckets" defined by metadata (language, interaction track, response family, and length). The framework consists of two primary components:

1. Bucket-Aware Supervised Fine-Tuning (Bucket-SFT)

Standard SFT often suffers from bucket imbalance, where frequent interaction regimes dominate the optimization, causing the model to neglect less frequent but distinct human response patterns.

• Mechanism: PolyAlign partitions training data into buckets $B$. It assigns a specific weight $e_{w_b}$ to each instance $i$ in bucket $b$ to ensure macro bucket risk minimization.
• Objective: The method assigns equal optimization mass to every bucket, regardless of the number of samples within it. The loss function is formulated as:
  $$L_{\text{Bucket-SFT}}(\theta) = \frac{1}{|B|} \sum_{b \in B} \frac{1}{n_b} \sum_{i: b_i=b} \ell_i(\theta)$$
• Theoretical Guarantee: Theorem 1 proves that this weighting scheme exactly converts supervised learning into macro bucket risk minimization, ensuring every interaction regime receives equal optimization mass.

2. Human-Distribution Preference Optimization (HDPO)

To further refine alignment, PolyAlign introduces a preference optimization objective that incorporates distributional information via a critic.

• Bucket-Support Distance: A critic estimates the distance of a candidate response's feature vector $z$ from the empirical human support region $\Lambda_b$ of its specific bucket. This distance, $D_b(z)$, is zero if the response lies within the human support region and increases as it deviates.
• Regularization: HDPO augments the standard sigmoid-DPO loss with a critic-based regularizer. The regularizer $R_{\text{HDPO}}$ favors responses that are both preferred by the human labeler and closer to the target human bucket support.
  $$L_{\text{HDPO}}(\theta) = \frac{1}{K} \sum_{i=1}^K w_i \left( L_{\text{DPO}}^{(i)}(\theta) + \lambda_{hd} R_{\text{HDPO}}^{(i)}(\theta) \right)$$
• Alignment: Theorem 3 demonstrates that under the sigmoid-DPO variant, the HDPO regularizer is locally distributionally aligned with preference learning: if the chosen response is distributionally closer to the human support than the rejected response, both objectives push the policy toward increasing the margin between them.

Key Contributions

1. Formulation: The paper formulates naturalistic alignment as matching the human response distribution appropriate to each interaction context, capturing variations in style, length, and structure across regimes.
2. Framework: Introduction of PolyAlign, which integrates conditional distributional targets into SFT (via Bucket-SFT) and preference optimization (via HDPO).
3. Evaluation Protocol: Development of a dual-axis evaluation measuring Task Utility (e.g., QA-F1, ROUGE-L) and Conditional Naturalness (e.g., Bucketed Naturalness Gap, Conditional MAUVE, Naturalness-Utility Frontier).

Experimental Results

The framework was evaluated on a bilingual (English and Chinese) corpus covering five interaction situations (e.g., assistant-like, long-form QA, open chat) using compact open models (Qwen2.5-1.5B/3B, Gemma2-2B, Llama-3.2-3B).

• Performance vs. Baselines: PolyAlign consistently outperformed BaseLM, Chain-of-Thought (CoT), Full-SFT, and standard DPO.
  • Bucket-SFT: Proved to be the most reliable component, significantly improving distributional alignment (lowering Bucketed Naturalness Gap) while preserving task utility. For example, on Qwen2.5-1.5B (English), Bucket-SFT improved the aggregate score from 0.371 (Full-SFT) to 0.643 and reduced the BNG-macro from 5.012 to 0.427.
  • HDPO: Provided additional refinements, particularly in Naturalness-Utility Frontier (NUF) and human-likeness metrics, when the critic signal was reliable.
• Human-Likeness: PolyAlign models demonstrated significantly lower AI-detection rates (measured by Binoculars, Fast-DetectGPT, and RADAR) compared to baselines, indicating higher human-likeness.
• Stability: Explicit human-support conditioning (Bucket-SFT) was found to be more stable than global preference optimization (DPO), which often drifted away from desired human-like response supports, especially in Chinese datasets.
• LLM-as-a-Judge: Judge evaluations confirmed that Bucket-SFT improves conditional appropriateness and distribution faithfulness without sacrificing task correctness.

Significance and Claims

The paper argues that post-training should evolve from optimizing for "how to make a model better on average" to "how to make a model produce the right kind of answer for the right kind of interaction."

• Shift in Paradigm: The work suggests moving beyond generic alignment toward conditional, naturalistic, and interaction-aware alignment.
• Natural Variation: By explicitly modeling interaction-specific human response patterns, PolyAlign generates responses that are more natural and situation-appropriate than those produced by standard Full-SFT or DPO.
• Modest Scope: The authors acknowledge this is an initial study focused on English and Chinese with compact models. They note that future work is needed to extend this to broader multilingual settings, richer bucket representations, and larger-scale models. The paper does not claim to solve all alignment issues but demonstrates that explicitly modeling human response distributions is a necessary step toward more natural AI interaction.