Technical Summary: Diverse LLMs or Diverse Question Interpretations?

Problem Statement

The integration of diversity into machine learning systems is a well-established strategy for improving robustness and performance, particularly in ensemble methods where components fail in complementary ways. However, in the context of Large Language Models (LLMs), effectively leveraging diversity remains challenging. Traditional ensemble approaches often rely on model diversity, aggregating outputs from multiple distinct LLMs. A critical limitation of this approach is that independently developed LLMs may share underlying training data or architectural biases, leading to positively correlated errors that limit accuracy gains.

This paper addresses the research question: "Can diversity in question interpretation serve as an effective alternative to model diversity?" The authors investigate whether generating multiple semantic interpretations of a single question and aggregating the answers from a single model yields better performance than aggregating answers from multiple models on a single interpretation.

Methodology

The study compares two ensemble strategies for answering binary questions (Yes/No) across three datasets: BoolQ, StrategyQA, and PubMedQA.

1. Diversity Approaches

• Model Diversity: Three distinct LLMs answer the same question using a single, fixed interpretation. The final answer is determined by majority voting.
• Question Interpretation Diversity: A single LLM generates three distinct semantic interpretations of the same question. The model then answers each interpretation individually. The final answer is determined by majority voting across these interpretations.

2. Experimental Setup

• Models: The study evaluates both commercial models (OpenAI's GPT-3.5-turbo, GPT-3.5-turbo-instruct, GPT-4) and open-weight models (LLaMA-2-7b, LLaMA-3-8B, LLaMA-3-8B-Instruct).
• Prompting Strategy:
  • To generate interpretations without introducing model variability, a single model (GPT-3.5-turbo) is used with a specific prompt (pmulti-step1) to produce at least three distinct semantic interpretations.
  • Each model in the ensemble then answers the specific interpretations using a second prompt (pmulti-step2) that instructs the model to restate the interpretation, provide relevant facts, and summarize the answer as <answer>Yes</answer> or <answer>No</answer>.
• Answer Extraction & Validation:
  • Raw LLM outputs are processed through extraction heuristics (Match First and Match Pattern) to normalize responses into a set of summarized answers: {yes, no, none, depends, unclear}.
  • A correctness score is computed by comparing the extracted answer against the ground truth.
• Ensemble Consensus: Majority voting is applied to the summarized answers of the ensemble members (either different interpretations or different models) to determine the final output.

Key Results

The experiments demonstrate that Question Interpretation Diversity consistently outperforms Model Diversity in terms of ensemble accuracy.

1. Performance Comparison

• Interpretation Diversity: Voting across diverse interpretations consistently improved accuracy compared to the baseline (direct questions) and often surpassed the performance of the best individual ensemble member. This was particularly pronounced for LLaMA-based ensembles on the BoolQ and StrategyQA datasets.
• Model Diversity: Voting across diverse models typically produced results that fell between the best and worst individual ensemble members. It rarely provided a significant improvement over the best single model and, in some cases (e.g., LLaMA models on PubMedQA), resulted in lower performance than the best member.

2. Quantitative Findings

• Accuracy Delta: When comparing the ensemble accuracy against the best individual member:
  • Model Diversity: Showed negative mean accuracy differences (-3.73 for ChatGPT, -10.53 for LLaMA), indicating the ensemble often underperformed the best single model.
  • Interpretation Diversity: Showed a mean accuracy difference of -1.35 for ChatGPT and a positive +11.12 for LLaMA, indicating that interpretation diversity consistently brought the ensemble closer to or above the performance of the best individual member.
• Dataset Specifics: The advantage of interpretation diversity was most significant for open-weight LLaMA models, suggesting that varying the semantic framing of a question is a more effective mechanism for error correction than varying the model architecture or weights.

Significance and Claims

The paper claims to be the first work to directly compare model diversity and question interpretation diversity in LLM-based ensembles.

• Primary Contribution: The authors establish that for binary QA tasks, diversity in question interpretation is a more effective design choice than model diversity. This approach allows for performance gains using a single model, thereby streamlining development by enabling focused tuning and prompt engineering without the cost of managing multiple distinct models.
• Theoretical Implication: The results suggest that the "failure diversity" required for robust ensembles can be achieved by varying the semantic context of the input (interpretation) rather than relying on architectural diversity, which may suffer from shared failure patterns.
• Modesty in Claims: The authors acknowledge limitations, noting that the study is restricted to binary questions and specific datasets. They do not claim universal applicability to open-ended generation tasks, noting that answer validation for non-binary questions would require different heuristics (e.g., semantic similarity). Furthermore, they note that the overhead of generating interpretations introduces latency, and the scalability of adding more interpretations versus more models remains an open question.

In conclusion, the paper argues that leveraging inference-time diversity through question interpretation is a superior and more efficient strategy for enhancing LLM ensemble accuracy than relying on model diversity alone.