🤖 AI

Agreement Before Diversity: Verification-First Complementarity for Heterogeneous Language-Model Coordination

The paper proposes Agreement-Before-Diversity (ABD), a verification-first coordination method for heterogeneous language models that uses a frozen, label-free decision rule to retain anchor answers only when corroborated by trusted samples, thereby achieving state-of-the-art performance on LiveCodeBench and GPQA-Diamond while providing exact theoretical identities to audit the trade-offs between accuracy gains and inference costs.

Original authors: Ruitong Li, Binjie Guo, Aisheng Mo, Guowei Su, Jie Li, Ru Zhang

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruitong Li, Binjie Guo, Aisheng Mo, Guowei Su, Jie Li, Ru Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky puzzle, and instead of asking just one person for the answer, you ask a whole team of experts. Some are math wizards, some are creative writers, and some are logic masters. This is the world of Large Language Models (LMs) working together. In science, this is called an "ensemble." The big idea is simple: if you have more people trying, you have a better chance of getting the right answer. It's like having ten people guess the number of jellybeans in a jar; the average of their guesses is usually closer to the truth than just one person's guess.

However, there's a catch. Just because you have ten guesses doesn't mean you know which one to pick. Sometimes, the most confident-sounding expert is actually wrong, and the quiet one is right. Worse, if you ask a "super-expert" to combine all the answers, they might accidentally talk over the correct answer with a fancy but wrong one. The big question scientists have been asking is: How do we know when to trust the new, fancy answer and when to stick with the one we already have? We need a rule that says, "Stop! This new answer is good enough to replace the old one," without just guessing.

This is where a new method called Agreement-Before-Diversity (ABD) comes in. Think of it as a strict but fair referee for a team of AI experts. Instead of blindly letting a fancy new answer overwrite an old one, ABD sets up a simple "safety check." Here's how it works:

The Anchor: First, the team picks one "anchor" answer (the current best guess).
The Safety Check: Before they even look at the fancy new answers, two other trusted experts from the same team check the anchor. If these two agree with the anchor perfectly, the referee says, "Great! We have a consensus. Keep the anchor. Don't waste time or money on anything else."
The Switch: But if those two experts disagree with the anchor, the referee says, "Okay, the anchor is shaky. Now, go ahead and bring in the whole team of diverse experts to synthesize a brand new, fancy answer."

The paper proves that this simple rule is incredibly powerful because it separates diversity (having many options) from authority (the right to change the answer). The authors found that this method doesn't just guess; it uses math to show exactly why it works. They discovered two "magic formulas" (exact identities) that explain the results:

The method beats a system that always makes a new fancy answer only when the anchor is actually better than the fancy answer on the specific cases where they agreed.
The method beats a system that never changes the answer only when the new fancy answer fixes a mistake the anchor made, without accidentally breaking a correct answer.

In real-world tests, this referee system was amazing. On a coding challenge called LiveCodeBench, the ABD method got 59.43% correct, beating the standard methods which only got around 52%. On a tough science quiz called GPQA-Diamond, it hit 75.00%, again beating the others.

What's really cool is that the paper shows why it worked in each case. On the coding test, the "safety check" rarely passed (only 1.71% of the time) because computer code is so specific that two random tries rarely match perfectly. So, the system mostly just let the fancy team take over, which turned out to be the right move. But on the science quiz, the safety check passed often (73.33% of the time), and because the "anchor" was usually right, the system saved a ton of effort by sticking with the simple answer instead of overthinking it.

The paper also explicitly rules out some common ideas. It shows that just having a "diverse" team of different AI models isn't enough on its own; without a strict rule on when to switch, you might just get more wrong answers. It also proves that you don't need to know the "correct" answer beforehand to make this decision; the rule works just by looking at whether the experts agree with each other.

In short, Agreement-Before-Diversity teaches us that having more options is great, but you need a smart gatekeeper to decide when to use them. It turns the chaotic process of asking many AIs for help into a precise, auditable, and highly effective strategy. It's not about having the smartest AI; it's about having the smartest rule for using them.

Technical Summary: Agreement Before Diversity (ABD)

Problem Statement

Heterogeneous language-model ensembles expand the space of candidate responses, offering the potential for higher accuracy through diversity. However, existing approaches lack a principled criterion for determining when a newly generated answer should supersede an already supported one. Current methods often conflate candidate headroom (the raw potential of diverse outputs) with replacement authority (the justification for changing the current answer). A diverse error remains an error, and a fluent synthesizer can overwrite a correct answer just as easily as it repairs a wrong one. The core challenge is distinguishing between having more options and knowing which option to trust without relying on unverifiable assumptions like sample independence or calibrated confidence scores.

Methodology: Agreement-Before-Diversity (ABD)

The authors propose Agreement-Before-Diversity (ABD), a frozen, label-free decision rule that decouples candidate generation from replacement authority. The method operates on a specific item $x$ as follows:

Anchor and Probes: An anchor model produces an answer $y_0$ (at temperature 0). Two additional trusted samples, $y_1$ and $y_2$ , are generated from the same model (at temperature 0.7).
Verification Event ( $A$ ): A frozen equivalence relation $E$ $E$ (e.g., normalized string equality for code, exact option matching for multiple choice) compares the parsed answers.
- If $y_0, y_1,$ and $y_2$ are all non-null and $E(y_0, y_1)$ and $E(y_0, y_2)$ hold, the agreement event $A(x) = 1$ .
- Otherwise, $A(x) = 0$ .
Decision Branching:
- If $A(x) = 1$ (Protected): The anchor $y_0$ is retained. Total cost: 3 inference calls.
- If $A(x) = 0$ (Escalated): The system synthesizes a new answer $s$ using a heterogeneous pool of four peer models and a resolver. Total cost: 8 inference calls (3 probes + 4 peers + 1 resolver).

The equivalence relation $E$ is deliberately conservative and label-free; it does not certify correctness but serves solely as a gate to decide whether diversity is permitted to override the anchor.

Key Contributions and Theoretical Framework

The paper's primary contribution is not a tuned coordinator but a decision principle with exact accounting. The authors derive two exact identities that decompose the performance of ABD relative to two extreme policies, requiring no assumptions of independence or calibrated confidence.

Identity 1 (vs. Unconditional Synthesis):
$E[U - Q] = q(a - b)$
The accuracy gap between ABD and always synthesizing is determined by the agreement coverage ( $q$ ) multiplied by the anchor's advantage ( $a - b$ ) on the protected subset. This identity shows that ABD only outperforms unconditional synthesis if the anchor is more accurate than the synthesis on the items where agreement occurs.
Identity 2 (vs. Never Synthesizing):
$E[U - Z] = (1 - q)(r - d)$
The accuracy gap between ABD and always keeping the anchor is determined by the escalation rate ( $1-q$ ) multiplied by the net authorized recovery minus authorized destruction ( $r - d$ ) on the escalated subset. Here, $r$ represents cases where synthesis corrects a wrong anchor, and $d$ represents cases where synthesis corrupts a correct anchor.
Cost Identity:
$E[C] = 8 - 5q$
The expected inference cost is a linear function of coverage. ABD is cheaper in terms of call count than always-on synthesis (which costs 6 calls) if and only if $q > 0.4$ .

These identities localize every aggregate difference to an enumerable, protected stratum, making the system's behavior fully auditable.

Experimental Results

The method was evaluated under strict, blind conditions on two distinct benchmarks: LiveCodeBench-v6 (175 tasks, executable tests) and GPQA-Diamond (180 untouched items, exact option scoring).

LiveCodeBench-v6: ABD achieved 59.43% accuracy, outperforming Single9 (52.57%) and HAC (52.00%). It tied with Heterogeneous MoA (Always Synthesis) and exceeded Homogeneous MoA.
- Mechanism: The agreement coverage ( $q$ ) was extremely low (1.71%) because independently generated code rarely matches exactly. Consequently, the gate was nearly inert, and the system effectively reduced to heterogeneous synthesis. The theoretical bound predicted a maximum contribution of 1.71 points; the observed contribution was exactly zero, confirming the theory.
GPQA-Diamond (Untouched): ABD achieved 75.00% accuracy, outperforming Single9, HAC, and Heterogeneous MoA (72.78%).
- Mechanism: Here, coverage was high (73.33%). The gain over Always Synthesis (2.78 points) was entirely explained by the preservation of correct anchor answers on the protected subset ( $q(a-b)$ ).
Perturbation Study: When the anchor model was replaced with a weaker one (frozen perturbation), coverage dropped to 39.44%, but the anchor's advantage on the protected set increased significantly. ABD still outperformed baselines, demonstrating that the method adapts to anchor quality without retraining.

Component Ablations:

Removing the gate (Always Synthesis) showed that diversity alone does not guarantee improvement; on LiveCodeBench, the gate removal had 0.00 impact, while on GPQA, it caused a drop.
Removing proposer heterogeneity (Homogeneous MoA) showed that diversity is not universally superior; it helped on code but hurt on GPQA compared to the heterogeneous version.

Significance and Claims

The paper claims that ABD provides a principled framework for managing the tension between stability and exploration in language-model ensembles.

Auditable Decision Making: Unlike heuristic coordinators, ABD offers exact identities that account for every aggregate difference. Practitioners can measure coverage ( $q$ ), protected-set advantage ( $a-b$ ), and recovery/destruction rates ( $r-d$ ) to diagnose system behavior without needing labeled validation sets.
Separation of Potential and Authority: The results demonstrate that diversity supplies potential (headroom), but verification structure supplies the authority to realize it. A diverse pool is only beneficial if the gate correctly identifies when to trust the anchor and when to escalate.
Modest Scope: The authors explicitly state that the evidence is scoped to code generation and expert multiple-choice questions. They do not claim ABD is a universal solution for all tasks but rather a framework that highlights the necessity of domain-aware verification signals (e.g., execution signatures for code, provenance for reasoning) to define the equivalence relation $E$ .
No "Correctness Oracle": The method does not claim to know the ground truth. It only grants permission to preserve an answer when supported by agreement, or to replace it when that support fails.

In conclusion, ABD transforms the coordination problem from a black-box optimization task into a transparent, measurable process where the trade-offs between accuracy, cost, and diversity are explicitly quantified.