🤖 AI

When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models

This paper argues that the performance gains of multi-model LLM systems like routing and voting are fundamentally capped by the "co-failure rate" (the frequency with which all models fail simultaneously), a metric that standard correlation measures miss and which often limits ensemble accuracy to that of the single best model unless a strong query-level routing signal is present.

Original authors: Josef Chen

Published 2026-06-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Josef Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why "More Voices" Doesn't Always Mean "Better Answers"

Imagine you are a manager trying to solve a difficult puzzle. You have a team of 67 different experts (AI models) to choose from. The common advice in the tech world is: "Don't just pick the smartest person; ask a group and vote on the answer. If they disagree, the group is usually smarter than any single person."

This paper argues that this advice is often wrong. Sometimes, asking a group of experts actually gives you the same answer as asking just the single best expert, but at a much higher cost. In fact, on hard problems, the whole group might fail together, and no amount of voting can fix that.

The Two Main Problems

The authors found that combining models hits a "glass ceiling" (a limit you can't break through) for two main reasons.

1. The "All-Wrong" Ceiling (The Common Blind Spot)

Imagine you ask 67 experts a very tricky math question.

The Old Belief: If the experts are diverse, they will make different mistakes. If Expert A is wrong, Expert B might be right.
The Reality: On hard, open-ended questions (like complex math or coding), all 67 experts often get the exact same question wrong at the same time. They share a "blind spot."

The paper calls this $\beta$ (Beta): the rate at which every single model fails on the same question.

The Rule: No matter how you combine them (voting, routing, or cascading), your accuracy can never be higher than 100% minus $\beta$ .
The Analogy: Imagine a group of 67 people trying to find a hidden key in a room. If they all look in the same wrong corner (because they were all taught the same wrong thing), asking all of them won't help. You will never find the key, no matter how many people you ask.

2. The "Correlation Trap" (The Wrong Metric)

Currently, companies decide whether to use a group of models by looking at $\rho$ (Rho), which measures how often two models make the same mistake.

The Trap: The paper proves that $\rho$ is a terrible predictor of the "All-Wrong" problem. You can have two models that seem very different (low correlation) but still fail together on the hardest questions.
The Analogy: It's like checking if two weather forecasters agree on whether it will rain tomorrow. They might disagree 90% of the time. But if a massive hurricane hits (a hard problem), they might both get it wrong. Checking their daily agreement doesn't tell you if they will both fail when the big storm comes.

The Two Regimes: When to Combine, When to Stop

The paper divides tasks into two distinct "worlds" or regimes:

World A: The "Ceiling-Bound" Tasks (Open-Ended Math & Code)

What happens: These are hard problems with no multiple-choice options.
The Result: The "All-Wrong" rate ( $\beta$ ) is high. The experts all fail together.
The Lesson: Combining models here is useless. You cannot beat the single best model because the whole group is stuck in the same blind spot. The paper found that on these tasks, the "All-Wrong" rate was about 2.5 times higher than what standard math predicted.

World B: The "Realizability-Bound" Tasks (Science & Multiple Choice)

What happens: These are tasks where the answer is clear or there are options to choose from (like a multiple-choice test).
The Result: The "All-Wrong" rate is near zero. The experts rarely fail together.
The Lesson: Here, there is room for improvement. The experts disagree on the answers, so a smart system could theoretically pick the right one. However, the paper found that current "router" systems (AI that decides which model to use) are too dumb to actually find these differences. They miss the opportunity.

The "Format" Surprise

The authors did a cool experiment with science questions (GPQA).

When asked as Multiple Choice, the models rarely all got it wrong.
When asked as Free Response (stripping away the options), the models suddenly started failing together at a high rate.
The Takeaway: It's not the topic (science) that causes the failure; it's the format. Open-ended questions trigger the "All-Wrong" blind spot.

The Economic Lesson: Diversity vs. Quality

The paper also tested a common idea: "If you mix a smart model with a dumb but diverse model, you get a better result."

The Finding: This is usually false. If the "dumb" model is too different, it often votes for the wrong answer, dragging the smart model down.
The Exception: Diversity only helps if the models are of equal quality. If you mix a group of models that are all equally smart but make different types of mistakes, then combining them works. But if you mix a genius with a novice, the novice usually hurts the team.

Summary: What Should You Do?

Stop trusting the "Pairwise Correlation" number. It tells you nothing about whether the whole group will fail together.
Check for the "All-Wrong" rate ( $\beta$ ). Before you build a complex system of multiple models, check if they all fail on the same hard questions. If they do, you have hit a ceiling. No amount of voting will break it.
Don't just add more models. Adding more models to a group that already shares a blind spot is like adding more people to a search party that is looking in the wrong room. It doesn't help.
Focus on the "Format." If you are asking open-ended questions, expect the models to fail together. If you can turn them into multiple-choice questions, you might unlock the ability to combine them effectively.

The Bottom Line: On today's hardest open-ended tasks, the "best" models are so similar that they fail together. Combining them rarely beats the single best model unless you have a very specific signal to know which model is right for which question. The magic isn't in the number of models; it's in finding models that fail in different ways.

Technical Summary: A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents

Problem Statement

The field of Large Language Model (LLM) orchestration has shifted from selecting a single "best" model to managing heterogeneous pools of models via routing, voting, cascades, and Mixture-of-Agents (MoA). Practitioners currently rely on pairwise error correlation ( $\rho$ ) as the primary diagnostic to determine if diversity will yield accuracy gains. The prevailing assumption is that low $\rho$ implies high diversity and thus high potential for ensemble improvement.

This paper argues that $\rho$ is the wrong diagnostic. It posits that the achievable accuracy of any policy whose output is a member model's answer (e.g., a router, majority vote, or cascade) is bounded not by pairwise correlations, but by $\beta$ , the rate at which all models in the pool fail on the same query. The central problem is that $\rho$ cannot identify $\beta$ ; distinct joint error distributions can share identical marginals and pairwise correlations yet possess vastly different co-failure rates. Consequently, relying on $\rho$ leads to a systematic underestimation of the "co-failure tail," potentially causing buyers to overestimate the value of orchestration or misjudge the ceiling of achievable accuracy.

Methodology

Theoretical Framework

The authors formalize orchestration as a budget-constrained allocation problem.

The Ceiling Theorem (Prop. 1): For any selection policy $\pi$ outputting a member's answer, accuracy is bounded by $1 - \beta$ , where $\beta = P(\text{all } m \text{ models wrong})$ . The maximum possible gain over the single-best model is $\Delta_{ceil} = (1 - \beta) - \bar{q}_{best}$ .
Non-Identification (Prop. 3): For $m \ge 3$ , the co-failure rate $\beta$ is not a function of pairwise error laws. The authors prove that joint distributions exist with identical one- and two-dimensional marginals (and thus identical $\rho$ ) but different $\beta$ .
The Underpricing Mechanism (Prop. 2): The authors model errors as a mixture of a common shock (all models fail with probability $\pi$ ) and independent errors. They demonstrate that a single-factor Gaussian copula, calibrated to pairwise $\rho$ , predicts a co-failure rate $\beta_{sf}$ that converges to 0 as pool size $m$ increases, whereas the true $\beta$ converges to $\pi > 0$ . Thus, the underpricing ratio $\beta / \beta_{sf}$ diverges with pool size.
Finite-Sample Certificate: Using Clopper–Pearson intervals on a held-out query set, the authors derive a pre-deployment "$0 certificate." If the lower bound of $\beta$ implies that $1 - \beta_{lo}$ is close to the current single-best accuracy, no policy can pay for its orchestration overhead.

Experimental Setup

The study evaluates 67 frontier models from 21 providers (including GPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro, and others) across three distinct regimes:

Saturated Mix: Standard benchmarks (GSM8K, MMLU, ARC-Challenge, MATH-500).
Hard Single-Domain: MMLU-Pro and MATH-Hard (Level-5).
Structurally Disjoint Domains:
- Open-ended mathematics (MATH-500, MATH-Hard).
- Execution-graded code (competitive programming problems with stress tests).
- Graduate-level science (GPQA-Diamond), tested in both multiple-choice and free-response formats.

Grading is strictly programmatic (exact match, integer extraction, or execution-based) to avoid LLM judge bias, except for a specific free-response GPQA experiment graded by a 5-judge panel. Costs are metered against real API usage.

Key Results

1. The Co-Failure Tail and the $\rho$ Trap

Across open-ended domains (math and code), the empirical all-wrong rate $\beta$ is significantly higher than predicted by pairwise correlations.

MATH-500 (67 models): Empirical $\beta = 0.052$ (17/330 queries). A correctly tetrachoric-calibrated single-factor Gaussian copula predicts $\beta_{sf} = 0.021$ . The empirical tail is 2.5 times fatter than the copula prediction.
Code Contests: Empirical $\beta = 0.079$ . The single-factor model underprices this by 3.1 times.
Pool Size Effect: Resampling random subsets of models shows the underpricing ratio grows monotonically with pool size (from 1.0 at $k=2$ to a median of 2.5 at $k=67$ ).
Multiple-Choice vs. Free-Response: In multiple-choice GPQA, $\beta \approx 0$ . However, when the same questions are asked as free-response (stripping options), $\beta$ jumps to 0.127. This confirms the co-failure regime is driven by task format (open-endedness) rather than subject matter.

2. The Two Regimes of Orchestration

The paper identifies two distinct regimes where the "ceiling" behaves differently:

Ceiling-Bound (Open-Ended Math/Code): Here, $\beta > 0$ is non-negligible. The ceiling $1-\beta$ is low, and the "oracle gain" (theoretical max improvement) is small. The primary limitation is the co-failure tail, which pairwise $\rho$ fails to detect.
Realizability-Bound (Science/Multiple-Choice): Here, $\beta \approx 0$ . The ceiling is effectively open, and the oracle gain is large (e.g., $G=0.15$ in GPQA). However, this gain represents resolvable disagreement. Current learned routers fail to capture this gain because the query-level signal is insufficient to distinguish which model will be correct when the frontier disagrees.

3. Performance of Orchestration Strategies

Learned Routers: On the 15-model saturated mix, a learned router (TF-IDF + domain) captured only 9% of the oracle gain (statistically indistinguishable from zero). A stronger LLM-as-router captured 0%.
Naive Diversity: Unweighted majority voting on heterogeneous triplets often yields negative gains compared to the single best member, as weaker diverse models outvote the strong one.
Matched-Quality Ensembles: When comparing a high-correlation Self-MoA (multiple samples of the best model) against a low-correlation heterogeneous ensemble of matched quality, the heterogeneous ensemble wins (avg. gain +0.027). This supports the diversification theorem but only when member quality is controlled.
Cascades: Cascade performance collapses to random mixing as the verifier's AUC approaches 0.5, confirming theoretical bounds.

Significance and Claims

The paper claims to redefine the economics of LLM orchestration by shifting the focus from pairwise correlation ( $\rho$ ) to co-failure rate ( $\beta$ ).

The Diagnostic Shift: $\rho$ is blind to the "all-wrong" tail. A buyer cannot determine if orchestration is viable by looking at $\rho$ alone; they must estimate $\beta$ directly.
The $0 Certificate: A single graded query set can provide a statistical lower bound on $\beta$ . If $1 - \beta_{lo}$ is close to the current best model's accuracy, the paper claims it is mathematically impossible for any routing or voting policy to justify its cost.
The Limit of Scale: Simply adding more models to a pool does not guarantee better performance. If the pool shares a "common-mode atom" (a shared failure mode), the co-failure rate $\beta$ remains positive, capping the utility of the entire pool regardless of size.
Format Dependence: The "ceiling-bound" regime is a function of task format (open-ended vs. multiple-choice), not just domain difficulty. Open-ended tasks inherently carry a co-failure tail that limits the efficacy of ensemble methods.

The authors conclude that for current frontier models on verifiable, open-ended tasks, the "lever" for improvement is not adding more models or relying on naive diversity, but rather identifying failure-mode heterogeneity and managing market churn (the option value of breadth). The field's reliance on $\rho$ obscures the true ceiling set by $\beta$ .