🤖 machine learning

FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data

The paper introduces FusionFactory, a systematic framework and accompanying LLMFusionBench benchmark that leverages multi-LLM log data to fuse complementary capabilities across query, thought, and model levels, consistently outperforming individual models across diverse tasks while accommodating real-world serving constraints.

Original authors: Tao Feng, Haozhen Zhang, Zijie Lei, Pengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jiaxuan You

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tao Feng, Haozhen Zhang, Zijie Lei, Pengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jiaxuan You

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of conversations between people and many different AI assistants. Some AIs are great at math, others are wizards at coding, and some are encyclopedias of general facts. Usually, when you ask a question, you have to pick just one AI to answer. But what if you could look at all those past conversations and combine the best parts of every AI into one super-smart system?

That's exactly what this paper, FusionFactory, is about. The authors built a giant toolkit to figure out the best way to mix and match these different AI brains using their "log data" (the records of what they said and how well they did).

Here is the simple breakdown of their discovery:

1. The Problem: Too Many Choices, One Question

Think of the AI world like a restaurant with 20 different chefs.

Chef A makes the best pizza.
Chef B makes the best soup.
Chef C is a master at desserts.

If you order a meal, you usually have to pick just one chef to cook your whole dinner. But the authors noticed that in the real world, companies often ask all the chefs to cook the same dish and keep a record of who did what. They asked: "Can we use these records to create a 'Super Chef' that knows how to make the perfect pizza, soup, and dessert all at once?"

2. The Solution: FusionFactory (The Three Ways to Mix)

The authors created a framework called FusionFactory that tries three different ways to combine these chefs, depending on when you mix them.

Level 1: The "Smart Waiter" (Query-Level Fusion)

How it works: Before you even order, a smart waiter looks at your request. If you ask for a math problem, the waiter immediately sends it to the Math Chef. If you ask for a joke, they send it to the Comedy Chef.
The Analogy: It's like a traffic cop directing cars to the fastest lane.
The Result: This is the cheapest and fastest method. It doesn't change the chefs; it just picks the right one for the job. It saves money because you aren't wasting time asking the wrong chef to try.

Level 2: The "Recipe Book" (Thought-Level Fusion)

How it works: Instead of just picking one chef, this method looks at the best recipes the chefs used in the past. If 10 chefs solved a hard math problem, the system reads their steps, summarizes the "perfect way to think about this," and gives that summary to the chef you are using.
The Analogy: Imagine you are cooking, and before you start, someone hands you a sticky note that says, "Here is the secret trick the best chefs used to solve this specific problem." You then cook your dish using that trick.
The Result: This turned out to be the winner. It gave the biggest boost in performance. It's like giving the chef a cheat sheet of the best thinking patterns without having to retrain them.

Level 3: The "Apprentice Chef" (Model-Level Fusion)

How it works: This is the heavy lifting. The system takes all the best answers from the 20 chefs and forces one specific chef to study them until they memorize the lessons. It's like a long, expensive training camp.
The Analogy: You take a junior chef and make them memorize the best dishes of all the masters so they become a master themselves.
The Result: This was the least effective. The "junior chef" (the model) got confused trying to learn too many different styles at once and actually performed worse than just picking the right chef or giving them a cheat sheet.

3. The Big Discovery: "One Size Does Not Fit All"

The paper tested this on 14 different types of tasks (like math, coding, reading, and trivia) using 20 different AI models.

The "Cheat Sheet" (Thought-Level) is the MVP: For most tasks, giving the AI a summary of how to think (the "Recipe Book") worked better than anything else. It was flexible and powerful.
The "Smart Waiter" is the Budget King: If you are worried about cost, just letting a smart router pick the right AI is a great deal. It's almost as good as the best single AI but costs a fraction of the price.
The "Training Camp" (Model-Level) is Tricky: Trying to merge all the knowledge into one single AI model often failed. It's hard to teach one brain to be good at everything when the "lessons" come from 20 different personalities.

4. Why This Matters

The authors argue that we don't need to invent new, complex AI models to get better results. We already have a goldmine of data: the logs of what different AIs have said in the past.

If you have money but need speed: Use the Smart Waiter (Query-Level).
If you want the absolute best answers: Use the Recipe Book (Thought-Level).
If you want to build a single, permanent AI: Be careful, because the Training Camp (Model-Level) might not be worth the effort yet.

In short, the paper says: Don't just pick one AI. Look at the records of how they all think, and use those records to guide your current AI. It's like giving your AI a library of wisdom to read before it answers, rather than trying to force it to memorize the whole library.

Technical Summary: FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data

Problem Statement

The rapid diversification of Large Language Models (LLMs) has led to a landscape where different models excel at distinct tasks (e.g., code generation vs. factual retrieval). Consequently, practitioners increasingly employ multiple LLMs in workflows, generating valuable multi-LLM log data. This data consists of structured serving records containing aligned query-response pairs from multiple models, along with metadata such as model identity, response mode (direct vs. reasoning-augmented), inference cost, and quality feedback.

Despite the availability of this data, existing methods for integrating multiple LLMs typically address only a single stage of the pipeline in isolation:

Query-level: Routing methods select the best model for a specific query but do not fuse capabilities beyond selection.
Thought-level: Prompting strategies aggregate reasoning but often lack systematic evaluation against other fusion stages.
Model-level: Distillation and model merging transfer knowledge into a single model but often ignore the specific constraints of API-based serving or the nuances of log data.

A critical gap exists in understanding how the same structured multi-LLM log data can be leveraged across different intervention points (routing, prompting, and training) to meet varied user needs, particularly in scenarios where model weights are inaccessible (API-based serving).

Methodology

The authors propose FusionFactory, a systematic framework for fusing LLM capabilities, supported by LLMFusionBench, a large-scale benchmark designed to evaluate fusion strategies across three distinct stages.

1. LLMFusionBench Construction

To enable cross-stage comparison, the authors constructed a benchmark comprising:

Scope: 14 tasks across 6 domains (Math, Code, Commonsense Reasoning, World Knowledge, Reading Comprehension, Popular Knowledge).
Data: Responses from 20 open-source LLMs ranging from 8B to 671B parameters, totaling 103M tokens.
Structure: Unlike standard datasets, each entry is a structured log tuple $(q, t, m, s, r, c, y, j)$ , capturing the query, task, model identity, response mode, generated response, inference cost, task-specific performance, and an LLM-judge score.
Dual Modes: For each query, models generate both direct responses and reasoning-augmented (Chain-of-Thought) responses to capture diverse reasoning patterns.

2. The FusionFactory Framework

FusionFactory operates at three levels, corresponding to early, mid, and late fusion stages in the LLM pipeline:

A. Query-Level Fusion (Early Fusion)

Goal: Select the optimal LLM and response mode for a specific query before generation.
Mechanism: A router learns a policy $f_\phi$ to map $(query, task)$ to an action $a = (model, mode)$.
Objective: Maximize a reward function balancing performance ( $P$ ), cost ( $C$ ), and LLM-judge quality ( $J$ ):
$Reward = \alpha \cdot P - \beta \cdot C + \gamma \cdot J$
Implementation: Evaluated using various routing algorithms (KNN, SVM, MLP, BERT, GraphRouter) under different reward weighting scenarios (Performance First, Balance, Cost First, LLM Judge).

B. Thought-Level Fusion (Mid Fusion)

Goal: Enhance generation by injecting reusable reasoning patterns without modifying model weights.
Mechanism:
1. Template Extraction: An LLM summarizer distills abstract "thought templates" from the top- $k$ responses of historical queries (ranked by performance or judge score).
2. Retrieval: For a new query, similar historical queries are retrieved via embedding similarity.
3. Prompting: The corresponding thought templates are used as few-shot demonstrations to guide the target LLM.
Strategy: Evaluated using Performance-based, LLM-Judge-based, and Hybrid selection strategies for the top- $k$ responses.

C. Model-Level Fusion (Late Fusion)

Goal: Consolidate complementary capabilities into a single base model via supervised fine-tuning (SFT).
Mechanism: High-quality responses from the multi-LLM logs (selected by performance or judge score) serve as training data for a base model.
Strategy: Compared Top- $K$ SFT (using selected multi-LLM responses) against Top- $K$ Label-only SFT (using only ground truths) to isolate the benefit of multi-LLM knowledge.

Key Results

Experiments were conducted on the full test set of LLMFusionBench, comparing the three fusion levels against the best individual LLM and zero-shot baselines.

Thought-Level Fusion Achieves Best Performance:
- Thought-level fusion (specifically the Hybrid strategy with full-scale models) achieved the highest average performance (0.615), outperforming the best individual LLM (0.600) and other fusion baselines like LLM-Blender (0.603) and FuseLLM-adapted (0.557).
- Gains were most pronounced in reasoning-intensive tasks (Math and Code), where the retrieval of diverse reasoning traces provided significant value.
- The performance gains were robust even when controlling for near-duplicate retrieval, suggesting the benefit comes from reusable reasoning patterns rather than simple data overlap.
Query-Level Fusion Offers Best Cost-Efficiency:
- Query-level fusion (via GraphRouter) provided a favorable trade-off between performance and cost.
- Under cost-aware routing, it achieved performance comparable to the best single LLM but at approximately 10x lower inference cost (0.0184 vs. 0.1850 in normalized units).
- This confirms its utility for API-based serving where minimizing token usage is critical.
Model-Level Fusion Shows Limited Generalization:
- Model-level fusion (SFT) yielded modest gains and, in some domains (Code and World Knowledge), performed worse than zero-shot baselines.
- The authors attribute this to overfitting on the specific response styles of the logged data and the difficulty of abstracting heterogeneous task behaviors into a single parameter set.
- Notably, multi-LLM SFT outperformed Label-only SFT, confirming that multi-LLM logs contain valuable knowledge beyond ground truths, but the consolidation process introduces generalization challenges.
Domain-Specific Variations:
- Fusion provided marginal improvements in World Knowledge and Math domains, which rely heavily on strict factual accuracy and logical consistency, suggesting these areas are less amenable to the current fusion heuristics.
- Reasoning-heavy tasks (Math, Code) benefited most from thought-level fusion, while knowledge-intensive tasks showed mixed results.

Significance and Claims

The paper claims that FusionFactory and LLMFusionBench provide a unified, stage-aware perspective on LLM fusion that has been previously lacking.

Beyond API Compatibility: While prior work focused on API-only routing, this work demonstrates that the same structured multi-LLM log data can be systematically evaluated across routing, prompting, and training stages.
Decision Framework: The "fusion-stage matrix" reveals that there is no single optimal fusion strategy; the choice depends on deployment constraints (e.g., cost vs. accuracy) and task requirements (e.g., reasoning vs. recall).
- Use Query-level for cost-sensitive, high-throughput scenarios.
- Use Thought-level for maximizing reasoning capabilities without model retraining.
- Use Model-level only when a single deployable model is strictly required and offline training is feasible, acknowledging its generalization limits.
Practical Foundation: The work validates multi-LLM log data as a practical, scalable supervision source for fusing diverse LLM capabilities, offering a blueprint for future systems that must operate in heterogeneous, real-world serving environments.

The authors conclude that while thought-level fusion currently offers the strongest performance gains, the systematic comparison enabled by their framework is essential for understanding the trade-offs between flexibility, cost, and generalization in multi-LLM systems.