💬 NLP

Ensemble Learning for Large Language Models in Text and Code Generation: A Survey

This survey reviews emerging ensemble learning techniques for Large Language Models in text and code generation, categorizing them into seven distinct methods to highlight their benefits in improving output quality, diversity, and flexibility while laying the groundwork for future multimodal applications.

Original authors: Mari Ashiga, Wei Jie, Fan Wu, Vardan Voskanyan, Fateme Dinmohammadi, Paul Brookes, Jingzhi Gong, Zheng Wang

Published 2026-06-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mari Ashiga, Wei Jie, Fan Wu, Vardan Voskanyan, Fateme Dinmohammadi, Paul Brookes, Jingzhi Gong, Zheng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why One Brain Isn't Enough

Imagine you are trying to solve a very difficult puzzle. You could ask one expert to do it. But what if that expert is tired, biased, or just has a bad day? They might give you a wrong answer or a boring one.

This paper argues that Large Language Models (LLMs)—the AI brains behind tools like ChatGPT—often make the same mistakes. They can be inconsistent, biased, or "hallucinate" (make things up).

The solution? Ensemble Learning. Instead of relying on one AI, we combine the brains of many. Think of it like a committee of experts rather than a single dictator. By pooling their knowledge, the group makes better, more diverse, and more accurate decisions than any single member could alone.

The paper reviews seven different ways to build these "AI committees" for writing text and generating computer code.

The Seven Ways to Build an AI Committee

The authors categorize these methods into two groups: those that mix the AI's "brain structure" (Architecture-Level) and those that mix the AI's "answers" (Model-Level).

1. Weight Merging: The "Smoothie" Approach

How it works: Imagine you have three different smoothies (AI models), each made with different fruits (knowledge). Instead of drinking them separately, you blend them all into one giant smoothie.
The Paper's Claim: You take the mathematical "weights" (the internal settings) of several trained models and average them together. This creates a new, single model that knows a bit about everything the originals knew.
The Benefit: It's fast and doesn't require retraining. It's like instantly upgrading your phone by merging software updates from three different versions.

2. Knowledge Fusion: The "Tutor" Approach

How it works: Imagine a smart student (the target model) sitting in a classroom with three different teachers (source models). The teachers don't just give answers; they teach the student how to think by combining their different teaching styles.
The Paper's Claim: This method trains a new model to mimic the combined "probabilities" of several other models. It's like a student learning from a panel of experts to become a super-student.
The Benefit: You can combine models that are built differently (like mixing a sports car engine with a truck engine) to create something unique.

3. Mixture-of-Experts (MoE): The "Specialized Team" Approach

How it works: Imagine a hospital with one giant building. Instead of one doctor trying to know everything, you have a team of specialists: a cardiologist, a neurologist, and a dermatologist. When a patient walks in, a "router" (a triage nurse) sends them to the right doctor.
The Paper's Claim: The AI has many "expert" sub-networks. For every question, the AI only "wakes up" the specific experts needed to answer it.
The Benefit: You get the power of a massive brain without the massive cost. It's efficient because the AI doesn't use its whole brain for simple tasks.

4. Reward Ensemble: The "Panel of Judges" Approach

How it works: Imagine a talent show. Instead of one judge deciding who wins, you have a panel of five judges. If they all agree, the decision is solid. If they disagree, the system knows to be careful.
The Paper's Claim: In AI training, a "reward model" tells the AI if an answer is good. By using a panel of these judges, the AI learns to avoid "cheating" (trying to trick the system) and produces more honest, human-like answers.
The Benefit: It makes the AI safer and more aligned with what humans actually want.

5. Output Ensemble: The "Group Brainstorm" Approach

How it works: Imagine asking five different writers to draft a story. Then, you ask a sixth, very smart editor to read all five drafts and write the best version by combining the good parts of each.
The Paper's Claim: You run the same question through multiple models, get their different answers, and then use a "synthesizer" model to pick the best one or blend them together.
The Benefit: No need to retrain anything. You just use the models you already have and let them vote or collaborate.

6. Routing: The "Traffic Cop" Approach

How it works: Imagine a busy office. A receptionist (the router) looks at a question. If it's simple ("What's the weather?"), they send it to a cheap intern. If it's complex ("Fix this legal contract"), they send it to the expensive senior lawyer.
The Paper's Claim: A small, fast AI looks at the question and decides which of the many available AI models is the best one to answer it.
The Benefit: You save money by using cheap models for easy tasks and saving the expensive, powerful models only for the hard stuff.

7. Cascading: The "Escalation Ladder" Approach

How it works: Imagine a customer service line. You start with a basic bot. If the bot says, "I'm not sure," it automatically passes the call to a human. If the human is unsure, they pass it to a manager.
The Paper's Claim: You ask a small, cheap AI first. If it gives a confident, good answer, you stop there. If it's unsure or the answer looks weak, you "cascade" the request to a bigger, more expensive AI.
The Benefit: You get high-quality answers most of the time but only pay the high price when absolutely necessary.

Text vs. Code: The Same Team, Different Goals

The paper notes that while these methods work for both writing stories and writing computer code, the goals are different:

Writing Text: It's like painting. There are many "right" ways to do it. The goal is to be creative, fluent, and follow instructions well. The "committee" helps blend different styles and ideas.
Writing Code: It's like building a bridge. There is only one "right" way (it must work without crashing). The goal is strict logic and correctness. The "committee" helps generate many different solutions to ensure at least one of them actually works.

The Bottom Line

This paper is a survey of how to stop relying on a single, potentially flawed AI. By using these seven "teamwork" strategies, we can:

Improve Quality: Get better answers.
Save Money: Use cheaper models for simple jobs.
Be More Flexible: Mix and match different types of AI.

The authors conclude that while these methods are great for text and code, the next big step is to apply this "teamwork" logic to multimodal AI (models that can see, hear, and speak), creating even smarter, more collaborative systems for the future.

Technical Summary: Ensemble Learning for Large Language Models in Text and Code Generation

1. Problem Statement

Generative Pretrained Transformers (GPTs) have become foundational for text and code generation. However, individual Large Language Models (LLMs) face significant limitations that hinder their reliability and applicability in diverse real-world scenarios:

Inconsistency and Bias: Single LLMs often produce inconsistent outputs and exhibit biases due to their training data and fixed architectural parameters, limiting their ability to represent diverse language patterns.
Performance Ceiling: Recent studies indicate that even the best single LLMs achieve only approximately 57% in chat generation quality due to architectural constraints.
Closed-Source Restrictions: The proprietary nature of many powerful LLMs restricts industrial adoption due to data privacy concerns and limits the ability to integrate diverse models.
Hallucinations: Reliance on a single model can lead to hallucinations and degraded text quality, particularly when the model lacks external knowledge access.

While techniques like Mixture-of-Experts (MoE) and improved training criteria exist, they often remain within the scope of single-model architectures. There is a need for a comprehensive review of LLM ensemble techniques that combine multiple models to overcome these single-model limitations, specifically for text and code generation.

2. Methodology

This paper provides a survey and systematic analysis of emerging LLM ensemble approaches. The authors categorize these techniques into seven primary methods, distinguishing between Architecture-Level ensembles (modifying the model structure or parameters) and Model-Level ensembles (combining outputs or routing decisions without necessarily merging weights).

A. Architecture-Level Ensembles

Weight Merging: Combines parameters from multiple trained LLMs into a single model.
- Techniques: Simple averaging, Task Arithmetic (using task vectors defined as the difference between fine-tuned and pre-trained weights), TIES (Trimming, Interpolation, and Sign-based selection), and DARE (Drop And RE-scale).
- Goal: Consolidate diverse learned knowledge without additional training.
Knowledge Fusion: Trains a target LLM using a fused probabilistic distribution derived from multiple source LLMs.
- Techniques: Extends knowledge distillation by defining fusion functions over token-level prediction distributions. It addresses token alignment issues (different tokenizers) using dynamic programming or minimum edit distance.
- Goal: Transfer collective knowledge from structurally different models to a target model.
Mixture-of-Experts (MoE): A transformer architecture where a gating network routes input tokens to specific expert sub-networks.
- Techniques: Sparse activation (e.g., selecting top- $n$ experts) allows for scaling model capacity without proportional increases in computational cost.
- Goal: Model multidomain data distributions efficiently.
Reward Ensemble: Aggregates multiple Reward Models (RMs) within Reinforcement Learning from Human Feedback (RLHF).
- Techniques: Uses mean optimization, worst-case optimization, or uncertainty-weighted optimization to combine scores from multiple RMs.
- Goal: Mitigate reward hacking (over-optimization) and improve alignment with human preferences.

B. Model-Level Ensembles

Output Ensemble: Combines responses from multiple LLMs at inference time without retraining.
- Techniques: Majority voting for classification, or using a synthesizing LLM to fuse multiple generated texts (e.g., Mixture-of-Agents).
- Goal: Leverage the strengths of different base models to enhance accuracy and robustness.
Routing: Directs input queries to the most suitable LLM from a pool based on a scoring model.
- Techniques: Uses classifiers (e.g., kNN, transformers) or scoring functions (e.g., BART score) to predict the best model for a specific task, balancing cost and performance.
- Goal: Optimize deployment costs while maintaining high performance.
Cascading: Invokes a sequence of LLMs, starting with a weaker/cheaper model and escalating to stronger models only if necessary.
- Techniques: A decision maker evaluates the initial output (e.g., via agreement scores or consistency checks) against a threshold. If the threshold is not met, a more powerful model is invoked.
- Goal: Achieve near-optimal performance at a fraction of the cost.

C. Evaluation Framework

The paper evaluates these methods using standard metrics for text and code generation:

Text: Accuracy, Win Rate, BLEU, Alignment Scores, Human Evaluation, MMLU, GSM8K, AlpacaEval, MT-Bench.
Code: Pass@k, CodeBLEU, Execution Time, HumanEval, MBPP.

3. Key Contributions and Results

The paper reviews representative works (2023–2024) for each category, highlighting specific performance gains:

Weight Merging: Methods like DARE and TIES allow merging specialized models (e.g., math and code models) without training. Results show that merged models can outperform GPT-3.5 on specific tasks (e.g., Japanese math reasoning) and balance capabilities across domains, though they may still trail GPT-4 in general conversation.
Knowledge Fusion: Models like FusionChat (7B parameters) successfully leverage knowledge from diverse architectures (Mixtral, Solar, OpenChat) to outperform constituent models in reasoning and coding, achieving results competitive with closed-source GPT-3.5.
Mixture-of-Experts: Architectures like Mixtral 8x7B and OLMOE demonstrate that sparse activation can outperform larger dense models (e.g., LLaMA 2 70B) in MMLU and code generation (HumanEval) while using significantly fewer active parameters.
Reward Ensemble: LoRA-based reward ensembles improve alignment scores (e.g., from <4.16 to 4.35 on MT-Bench) and accuracy on helpfulness datasets without proportional computational increases.
Output Ensemble: The Mixture-of-Agents (MoA) approach, which uses a layered architecture of agents, achieved a 65.1% win rate on AlpacaEval2.0 (vs. 57.5% for GPT-4o mini) and 92.0% on HumanEval, outperforming strong commercial models.
Routing: Systems like Routoo and MLC Router demonstrate that routing between open-source models can achieve 75.87% accuracy on MMLU at less than half the cost of GPT-3.5, and approach GPT-4-Turbo performance at half the cost when mixing in closed models.
Cascading: FrugalGPT and Mixture-of-Thought (MoT) cascades reduce costs by up to 80% while maintaining or slightly exceeding the accuracy of GPT-4 on classification and reasoning tasks.

4. Significance and Claims

The paper claims that LLM ensemble techniques offer a practical pathway to overcome the limitations of single-model approaches, providing:

Enhanced Diversity and Quality: By combining multiple agents, ensembles improve the representation of diverse language patterns and reduce biases.
Cost and Resource Efficiency: Techniques like routing, cascading, and weight merging allow for high-performance outputs without the computational burden of training massive new models from scratch.
Flexibility: These methods enable the integration of both open-source and closed-source models, addressing privacy concerns and allowing for modular system design.
Real-World Applicability: The survey provides a foundation for selecting appropriate ensemble strategies for specific tasks (e.g., reasoning, code generation, instruction following) across sectors like medicine, finance, and education.

The authors conclude that while challenges remain—particularly in adapting these methods specifically for code generation and extending them to multimodal LLMs—ensemble learning represents a crucial step toward democratizing advanced AI capabilities and optimizing decision-making processes in industry.