💬 NLP

Diversity-Enhanced Reasoning for Subjective Questions

This paper introduces MultiRole-R1, a diversity-enhanced training framework that leverages unsupervised perspective synthesis and token-level diversity within a reinforcement learning setup to significantly improve large reasoning models' performance on subjective tasks while also boosting their objective reasoning capabilities.

Original authors: Yumeng Wang, Zhiyuan Fan, Jiayu Liu, Jen-tse Huang, Yi R. Fung

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yumeng Wang, Zhiyuan Fan, Jiayu Liu, Jen-tse Huang, Yi R. Fung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant (a Large Reasoning Model) that is incredibly good at math and coding. It solves these problems by "thinking out loud" in a long chain of thoughts, like a detective writing down every clue. This works great for math because there's only one right answer.

But what happens when you ask the robot a subjective question, like "Is it better to live in the city or the country?" or "What is the fairest way to distribute resources?"

Here, there is no single "right" answer. The answer depends entirely on who you are asking. A city planner, a farmer, a teenager, and a retiree will all have different, equally valid perspectives.

The problem the paper tackles is that current AI training methods force the robot to find that "one right answer" (like in math). This makes the robot boring and repetitive. It stops exploring different viewpoints and just gives the same safe, generic answer every time. It's like asking a chef to cook a meal but only allowing them to use salt, no matter what the dish is.

The Solution: MultiRole-R1 (The "Council of Voices")

The authors propose a new training method called MultiRole-R1. Think of it as turning the robot from a lone detective into a town hall meeting.

Here is how it works, using simple analogies:

1. The "Council of Voices" (Perspective Diversity)

Instead of letting the robot think alone, the researchers tell it to imagine it is three different people at the same time.

The Analogy: Imagine you are trying to decide where to build a new park.
- Role 1 (The Teenager): "We need skate ramps and Wi-Fi!"
- Role 2 (The Grandparent): "We need quiet benches and safe walking paths."
- Role 3 (The Local Business Owner): "We need parking and foot traffic."
The Magic: The robot is trained to generate a reasoning chain where it explicitly switches hats, arguing from each of these perspectives before making a final decision. This ensures the answer isn't just "one thing," but a rich tapestry of different human viewpoints.

2. The "Exploration Reward" (Token-Level Diversity)

In standard AI training, if the robot finds an answer that is "correct," it stops trying to find other ways to say it. It gets lazy.

The Analogy: Imagine a student taking a test. If they get the answer "C" right, they stop thinking.
The Fix: The researchers added a special "bonus point" system. If the robot's reasoning chain uses different words, different sentence structures, or explores unique angles (even if the final answer is the same), it gets extra points.
The Result: The robot learns that variety is good. It stops repeating the same phrases and starts exploring a wider "search space" of ideas, just like a human brainstorming session.

Why This Matters (The "Aha!" Moments)

The paper found three surprising things:

Diversity is the Secret Sauce, Not Length:
- Old Belief: "If the AI thinks longer (writes more words), it must be smarter."
- New Discovery: Not necessarily! The paper found that diversity (how many different angles the AI looks at) is a much better predictor of a good answer than just length (how many words it writes). A short, diverse answer is often better than a long, repetitive one.
- Metaphor: It's better to have a team of 3 people looking at a problem from 3 different angles than one person staring at it for 10 hours.
It Works on Math Too (The "Transfer" Effect):
- Even though they only trained the robot on subjective questions (opinions, ethics, culture), the robot got better at math (like the AIME 2024 competition).
- Why? Because the robot learned to explore more possibilities. In math, this means it didn't just jump to the first solution; it checked multiple paths, leading to fewer mistakes. It learned how to think, not just what to think.
Unsupervised Learning (Learning from Noise):
- They didn't need a human to grade every single answer. They used a "self-consistency" filter.
- The Analogy: Imagine asking the robot to solve a problem 10 times. If 8 times it says "C" and 2 times it says "A," they assume "C" is the most reliable answer. They use this self-checking mechanism to build their training data without needing perfect human labels.

The Bottom Line

MultiRole-R1 is like teaching an AI to be a diplomat instead of a calculator. By forcing it to wear different "hats" (roles) and rewarding it for being creative (diverse), the AI becomes better at handling complex human questions.

And the best part? By learning to be a good diplomat, it accidentally became a better mathematician too. It proves that thinking in many ways is a superpower that applies to almost everything.

1. Problem Statement

Large Reasoning Models (LRMs) optimized via Reinforcement Learning with Verifiable Rewards (RLVR), such as DeepSeek-R1 and OpenAI o1, excel at objective tasks (e.g., math, coding) where a single ground truth exists. However, these models suffer from diversity degradation (mode collapse), leading to repetitive and homogeneous reasoning chains.

This limitation is particularly detrimental for subjective reasoning tasks (e.g., ethical dilemmas, cultural opinion QA), where:

There is no single "correct" answer; validity depends on the stakeholder perspective (e.g., a religious leader vs. a secular activist).
Existing diversity-enhanced training methods for objective tasks rely on a single ground truth, making them incapable of generating the multiple valid outcomes required for subjective questions.
Current approaches (multi-agent debate, prompting) lack specific training paradigms designed to internalize perspective diversity.

2. Methodology: MultiRole-R1

The authors propose MultiRole-R1, a two-stage training framework designed to enhance both semantic-level diversity (perspective diversity) and token-level diversity (reasoning chain variation).

Stage 1: Multi-Role Reasoning Paths Synthesis & Fine-tuning (SFT)

Goal: Enhance perspective diversity by teaching the model to "think from which perspective" rather than just "think deeper."
Process:
1. Role Generation: The model generates $n$ context-relevant roles (e.g., experts, stakeholders) with conflicting viewpoints using few-shot prompting.
2. Selection & Sampling: Roles are selected based on relevance to the query and contrastiveness to existing opinions. For each role, multiple reasoning paths are sampled.
3. Self-Consistency Filtering: To ensure coherence, the most consistent answer for each role is selected via majority voting.
4. Path Synthesis: The filtered reasoning paths from different roles are concatenated into a single long Chain-of-Thought (CoT).
5. Merging Strategies:
  - Divergent Merging: For tasks where roles yield different answers, the final prediction is a weighted aggregation.
  - Convergent Merging: For tasks requiring consensus, a majority vote within the reasoning sequence determines the answer.
6. Filtering: The dataset is filtered to remove verbosity bias and formatting errors, resulting in a high-quality synthetic dataset (2,700 entries).

Stage 2: Diversity-Enhanced Reinforcement Learning (GRPO)

Goal: Enhance token-level diversity and broaden the answer search space.
Algorithm: Uses Group Relative Policy Optimization (GRPO).
Reward Shaping: The total reward $R$ is a combination of a verifiable accuracy reward ( $R_{acc}$ ) and a diversity reward ( $R_{div}$ ):
$R = \delta R_{acc} + (1 - \delta) R_{div}$
Diversity Metric ( $R_{div}$ ): A composite metric weighted equally across eight signals: lexical diversity, token entropy, sentence length, sentence pattern, adjacent sentence similarity, Yule's K, distinct N-grams, and function word diversity.
Mechanism: The diversity reward acts as a shaping signal to prevent the "reward hacking" (excessive verbosity) often seen in standard RLVR. It ensures intra-group reward variance in GRPO, preventing gradient stagnation when all samples in a group have similar accuracy.

3. Key Contributions

First Diversity-Enhanced Training for Subjective Tasks: Introduces a paradigm specifically for subjective reasoning, moving beyond the single-ground-truth assumption of standard RLVR.
Dual-Level Diversity Framework:
- Perspective Diversity: Anchors reasoning to real-world stakeholder groups, providing a coherent scaffolding for diverse outputs.
- Token-Level Diversity: Broadens the search space of reasoning chains via reward shaping.
Unsupervised Data Construction: Demonstrates that self-consistency filtering on self-synthesized data is as effective as, or better than, supervised ground-truth filtering for generating diverse training data.
Diversity as a Superior Indicator: Establishes that diversity is a more consistent indicator of accuracy than reasoning length in subjective tasks (correlation $r=0.74$ vs. length $r=0.55$ ).

4. Experimental Results

The framework was evaluated on four models (DeepSeek-R1 variants and Qwen3-8B) across subjective and objective benchmarks.

In-Domain (ID) Performance: MultiRole-R1 achieved an average 14.1% accuracy gain on three subjective tasks (BBQ, GlobalOpinionQA, ETHICS) compared to zero-shot CoT.
Out-of-Domain (OOD) Generalization:
- Improved accuracy by 7.64% on four OOD tasks (including cultural NLI and commonsense reasoning).
- Math Reasoning Transfer: Remarkably, training only on subjective questions improved performance on the advanced math benchmark AIME 2024 by 5.78%.
Efficiency: Contrary to the "longer is better" trend in some scaling laws, MultiRole-R1 achieved higher accuracy with shorter reasoning chains (avg. 657.8 words) compared to SFT baselines (1572.9 words), indicating higher reasoning efficiency.
Ablation Studies:
- Perspective diversity (SFT stage) contributed ~7.5% to the total gain.
- Token-level diversity (GRPO stage) contributed ~3.1%.
- On-policy GRPO with diversity shaping significantly outperformed off-policy DPO.

5. Significance and Implications

Redefining Reasoning Scaling: The paper challenges the notion that simply extending reasoning length improves performance. It argues that diversity (exploring multiple valid perspectives and token variations) is the true driver of accuracy in complex reasoning tasks.
Generalizability: The ability of a model trained solely on subjective questions to improve on hard math problems suggests that perspective diversity acts as a robust inductive bias, helping models escape local optima and explore broader solution spaces even in objective domains.
Practical Application: The framework offers a scalable solution for deploying LLMs in real-world scenarios (policy making, ethics, cultural analysis) where rigid, single-answer reasoning is insufficient and potentially harmful.

In conclusion, MultiRole-R1 demonstrates that integrating role-based perspective diversity and token-level diversity rewards creates more robust, accurate, and efficient reasoning models, bridging the gap between objective reasoning capabilities and the nuanced demands of subjective human tasks.