Technical Summary: SHALA-LLM

Problem Statement

Many human-centered natural language tasks, such as Natural Language Inference (NLI) and Emotion Recognition (ER), inherently possess multiple plausible interpretations. This leads to label ambiguity and significant disagreement among human annotators, driven by factors like contextual understanding, cultural background, and personal experience.

Current Large Language Model (LLM) alignment paradigms predominantly treat this annotator disagreement as noise. They typically collapse ambiguous annotations into a single "gold standard" label using majority voting or averaging. This approach discards the rich information embedded within human disagreement, particularly in highly ambiguous settings. While recent studies have explored distributional alignment, they often operate at inference time (via prompting) rather than directly optimizing LLMs during training to account for annotator disagreement. Consequently, existing methods fail to preserve the full distribution of human judgments or dynamically prioritize samples where ambiguity is most informative.

Methodology: SHALA-LLM

The authors propose SHALA-LLM (Smartly Handling Ambiguous Labels in Aligning LLMs), a reinforcement learning (RL) framework designed to align LLMs directly with empirical annotator distributions rather than single collapsed labels. The framework operates through the following key components:

1. Problem Formulation

Instead of collapsing annotations into a single label, SHALA-LLM derives an Empirical Label Distribution ($p_q$) from $N$ human annotators for a given sample $q$. This distribution represents the probability of each candidate class based on the frequency of annotator votes. The goal is to train the LLM to generate predictions ($\hat{p}$) that align with this underlying human judgment distribution.

2. Verbalized Distribution Prediction

To facilitate distributional learning, the LLM is prompted to generate verbalized distributions. Instead of outputting a single class, the model generates a textual response containing probability estimates for all candidate classes (e.g., in a JSON format). These textual outputs are parsed into a predicted probability distribution $\hat{p}^{(q,i)}$.

3. Reward Design: Ambiguity-Enhanced Optimization

The core innovation lies in the reward function used within the Group Relative Policy Optimization (GRPO) framework. The reward $r^{SHALA}_{(q,i)}$ for a sampled rollout is defined as:

$$r^{SHALA}_{(q,i)} = \tilde{H}(p_q) \cdot [1 - D_{JS}(\hat{p}^{(q,i)}, p_q)]$$

Where:

• Distributional Alignment Term: $[1 - D_{JS}(\hat{p}^{(q,i)}, p_q)]$ measures the agreement between the model's predicted distribution and the annotator distribution using Jensen-Shannon (JS) Distance. This replaces the sparse, binary rewards of majority-label supervision with continuous rewards proportional to distributional agreement.
• Ambiguity Enhancement Term: $\tilde{H}(p_q)$ is the normalized entropy of the annotator distribution. It quantifies the degree of disagreement (ambiguity) among annotators.
  • Dynamic Reweighting: Samples with higher annotator disagreement (higher entropy) receive proportionally amplified rewards. This dynamically prioritizes highly ambiguous samples during optimization, forcing the model to learn from richer disagreement structures rather than just dominant labels.

4. Optimization

The framework utilizes GRPO to optimize the policy. By computing group-normalized advantages based on the ambiguity-enhanced rewards, SHALA-LLM encourages the model to capture uncertainty and diverse human interpretations. This reshapes the reward geometry, yielding smoother advantage distributions and more informative policy updates under ambiguous supervision.

Key Contributions

1. Paradigm Shift: The paper challenges the assumption that annotator disagreement is noise, reframing it as informative supervision that improves model behavior.
2. Algorithmic Innovation: It introduces a novel RL-based algorithm (SHALA-LLM) that integrates ambiguity-aware reward modulation directly into the training loop, specifically utilizing GRPO.
3. Dual Improvement: The method demonstrates that modeling annotator disagreement not only improves distributional alignment (capturing human uncertainty) but also enhances standard classification performance (F1 scores), contrary to the belief that handling ambiguity compromises accuracy.

Experimental Results

The framework was evaluated on ambiguity-sensitive NLI and ER benchmarks: ChaosNLI, GoEmotions, and MSP-Podcast.

• Distributional Alignment: SHALA-LLM significantly reduced the Jensen-Shannon Distance (JSD) between model predictions and human distributions. On ChaosNLI, JSD was reduced by 62.1% compared to Majority-Label Supervision (MLS). On MSP-Podcast, JSD was reduced by 6.2%.
• Classification Performance: Contrary to the expectation that distributional supervision might hurt point-estimate accuracy, SHALA-LLM improved F1 scores. On ChaosNLI, F1 increased by 16.6%; on MSP-Podcast, it increased by 29.2% compared to MLS.
• Robustness to Ambiguity: Analysis across different ambiguity levels (low, medium, high) revealed that SHALA-LLM maintains stable performance as ambiguity increases, whereas baseline methods (like MLS) suffer significant degradation in high-ambiguity settings.
• Category-Specific Gains: The model showed particularly strong improvements on inherently ambiguous semantic categories (e.g., "Contradiction" in NLI and "Surprise" in ER), where baseline models often struggle.
• Reasoning Behavior: Qualitative analysis showed that SHALA-LLM generates reasoning patterns that reflect multiple valid interpretations and uncertainty, rather than overconfident, single-interpretation outputs typical of zero-shot or majority-label baselines.

Significance and Claims

The authors claim that SHALA-LLM demonstrates the value of treating human disagreement as a signal rather than noise. By preserving disagreement structures and dynamically prioritizing ambiguous samples, the framework leads to more robust and human-aligned LLM behavior in subjective, real-world settings.

The paper concludes that:

1. Preserving Disagreement: Maintaining the full structure of human judgments allows models to learn from diverse and potentially conflicting labels, leading to better alignment with human distributions.
2. Performance Synergy: Ambiguity-aware optimization does not trade off classification accuracy for alignment; rather, it strengthens both.
3. Generalizability: The approach is effective across different modalities (text and speech) and task types (inference and emotion recognition).

The authors acknowledge limitations, noting that the current evaluation focuses on structured label distributions and does not yet address open-ended generative tasks or complex multi-turn interactions. They also note that the framework assumes empirical distributions adequately represent underlying factors like annotator expertise, which may not fully capture all sources of ambiguity. Nevertheless, the work suggests a promising direction for developing more robust, human-centered large language models.