🤖 AI

Power and Limitations of Aggregation in Compound AI Systems

This paper investigates the power and limitations of aggregating responses from multiple identical models in compound AI systems, proving that such aggregation expands the set of elicitable outputs only through three specific mechanisms—feasibility expansion, support expansion, and binding set contraction—and providing empirical validation for these findings in large language model tasks.

Original authors: Nivasini Ananthakrishnan, Meena Jagadeesan

Published 2026-07-09

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Nivasini Ananthakrishnan, Meena Jagadeesan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot friend who can write stories, solve math problems, or list research papers. You ask it one question, and it gives you one answer. But what if you asked three copies of that same robot the same question, gave them slightly different instructions, and then mashed their answers together? Would the final mash-up be smarter than any single robot could be on its own?

This is the big question researchers Nivasini Ananthakrishnan and Meena Jagadeesan tackled in their new paper. They didn't just guess; they built a mathematical "principal-agent" framework (think of it as a strict rulebook for how a boss gives tasks to workers) to figure out exactly when and why combining answers works, and when it's just a waste of time.

The Three Magic Tricks of Aggregation

The authors discovered that simply asking the same robot three times doesn't automatically make it smarter. In fact, they proved that for a group of robots to produce an answer that no single robot could ever give on its own, the group must perform one of three specific "magic tricks." If the group doesn't do at least one of these tricks, the aggregation is useless, no matter how clever your instructions (prompts) are.

Here are the three tricks, explained with a story about a Paper Reference Generator:

1. The "Impossible Mix" Trick (Feasibility Expansion)
Imagine a robot that has a glitch: it can't talk about "Blockchain" without accidentally talking about "Cryptography" at the same time. It's like a robot that can't say "Apple" without also saying "Pie."

The Single Robot: If you ask it for a list about Blockchain, it must include Cryptography. It can't help it.
The Group: You ask Robot A for a list about "Blockchain but not Distributed Systems," and Robot B for "Cryptography but not Blockchain." Robot A gives you a list heavy on Blockchain; Robot B gives you a list heavy on Cryptography. When you combine (aggregate) their lists, you might end up with a perfect list that has only Blockchain and no Cryptography.
The Magic: The group created an output (Pure Blockchain) that was impossible for any single robot to produce alone because of their internal "glitches" (constraints). The paper calls this Feasibility Expansion.

2. The "Too Many Hobbies" Trick (Support Expansion)
Imagine you want a robot to write a story that is equally about "Space" and "Medicine."

The Single Robot: The robot is bad at multitasking. If you tell it to focus on Space, it ignores Medicine. If you tell it to focus on Medicine, it ignores Space. It can't hit both targets perfectly at once.
The Group: You ask Robot A to focus only on Space, and Robot B to focus only on Medicine. Robot A gives a great Space story; Robot B gives a great Medicine story. When you blend them, you get a story that is perfectly balanced between both.
The Magic: The group created an output that covers a wider range of topics (a "richer support") than any single robot could manage alone. The paper calls this Support Expansion.

3. The "Loose Constraint" Trick (Binding Set Contraction)
Imagine a robot that is forced to follow a rule: "If you mention 'Deep Learning,' you must also mention 'Neural Networks'."

The Single Robot: If you ask for a list about Deep Learning, the robot is "stuck" (bound) to the rule. It can't give you Deep Learning without Neural Networks.
The Group: You ask Robot A for a list that is 90% Deep Learning and 10% Neural Networks. You ask Robot B for a list that is 90% Neural Networks and 10% Deep Learning. When you combine them, the "must include" rule might cancel out in a way that lets you get a list that is purely Deep Learning, or at least has much less Neural Network fluff than the single robot could ever produce.
The Magic: The group found a way to slip out of the "tightest" constraints that were holding the single robots back. The paper calls this Binding Set Contraction.

The Big "No" (What Aggregation Can't Do)

Here is the most important part: The paper explicitly rules out the idea that just having more robots or more complex instructions guarantees a better result.

The authors proved mathematically that if your aggregation method (how you mix the answers) doesn't perform at least one of those three tricks above, it offers zero power. It doesn't matter if you are a genius at writing prompts or if the robot is very advanced. If the mixing process doesn't expand the "feasible" area, expand the "support" (topics), or contract the "binding" constraints, the group is just as limited as a single robot.

In fact, they showed that if you try to mix answers in a way that doesn't use these tricks, you are stuck. You can't get an answer that the single robot couldn't give you, no matter how hard you try.

How Sure Are They?

The authors didn't just make these up; they proved it.

The Theory: They used rigorous math to prove that these three mechanisms are necessary. This means if you see a group of AI agents doing something amazing that a single agent couldn't, it must be because they are doing one of these three things. They also proved that a "strengthened" version of these rules is sufficient, meaning if you can show these rules are happening, you are guaranteed to get a better result.
The Simulation: To make sure their math wasn't just a pretty theory, they ran a simulation (a computer experiment) using real Large Language Models (specifically GPT-4o-mini).
- They set up a "toy" task where the AI had to generate lists of research paper titles.
- They tested the three tricks. For Support Expansion, they found that a single model couldn't balance "Complexity Theory" and "Macroeconomics" perfectly, but the group could.
- For Feasibility Expansion, they showed the group could create a list about "Blockchain" that avoided "Distributed Systems," something a single model failed to do.
- For Binding Set Contraction, they showed the group could isolate "Deep Learning" better than a single model could.

In these simulations, the distance between what the group produced and what a single model could produce was clear. For example, in the Support Expansion test, the difference (measured in $\ell_1$ distance) was between 0.63 and 1.02. In the Feasibility Expansion test, the gap was between 0.07 and 0.39. These numbers aren't zero; they prove the group did something the single robot couldn't.

The Takeaway for Curious Minds

So, does asking five robots to work together always make a super-robot? No. The paper suggests that it only works if you are clever about how you combine their answers. You have to use the combination to break a rule the single robot was stuck with, or to cover more ground than one robot could.

If you just mash their answers together randomly, or if you ask them to do the exact same thing, you aren't gaining anything new. But if you use the right "mixing" strategy to exploit the robots' weaknesses (like their inability to multitask or their forced side-effects), you can unlock a whole new world of answers that were previously impossible.

The authors conclude that this gives us a roadmap for building "Compound AI Systems." Instead of just hoping that more models equal better results, we now know exactly what kind of "mixing" we need to do to actually make the system smarter. It's not magic; it's math.

Technical Summary: Power and Limitations of Aggregation in Compound AI Systems

Problem Statement

Compound AI systems, which leverage multiple AI components rather than a single model in isolation, have emerged as a powerful paradigm for tackling complex tasks. A common strategy involves querying multiple copies of the same Large Language Model (LLM) with different prompts or tools and aggregating their responses. While empirical success suggests aggregation improves performance, the theoretical conditions under which aggregating homogeneous models unlocks a strictly greater set of outputs than querying a single model remain unclear.

This paper investigates the power and limitations of such aggregation. Specifically, it asks: When does aggregating outputs from multiple copies of the same model allow a system designer to elicit outputs that are impossible to elicit from a single model, given constraints on prompt engineering (reward specification) and model capabilities?

Methodology

The authors formalize this problem using a stylized principal-agent framework, extending the work of Kleinberg and Raghavan [2020].

Theoretical Framework

Agents and Outputs: The system consists of $K$ agents (representing identical LLM copies) and a single principal (the system designer). Outputs are represented as $M$ -dimensional non-negative vectors $x \in \mathbb{R}^M_{\ge 0}$ .
Model Capabilities (Conic Constraints): Model limitations are modeled as conic constraints $C$ on the output space. The feasible set is defined as $B_{feasible}(C) = \{x \in \mathbb{R}^M_{\ge 0} \mid Cx \le 0\}$ . This captures scenarios where a model cannot produce certain desirable outputs without incurring undesirable side effects (e.g., hallucinations or safety violations).
Prompt Engineering Limitations (Coarser Features): The system designer cannot specify rewards directly over the $M$ -dimensional output space. Instead, rewards operate over a coarser $N$ -dimensional feature space defined by a feature weight matrix $\alpha$ . The feature mapping $F(x)$ aggregates output dimensions, capturing the limitation that prompts may be underspecified or that models may misinterpret fine-grained distinctions.
Agent Optimization: Each agent $k$ maximizes a monotone reward function $R^{(k)}$ subject to its budget $E^{(k)}$ and the conic constraints $C$ .
Elicitability-Expansion: An aggregation operation $x^{(1)}, \dots, x^{(K)} \to x^{(A)}$ is defined as elicitability-expanding if the constituent outputs $x^{(k)}$ are individually elicitable by some reward/budget pair, but the aggregated output $x^{(A)}$ is not elicitable by any single reward/budget pair.

Aggregation Rules

The paper analyzes general aggregation rules but focuses on two natural types:

Intersection Aggregation: Coordinate-wise minimum ( $x^{(1)} \wedge \dots \wedge x^{(K)}$ ), analogous to debate protocols or consensus filtering.
Addition Aggregation: Weighted sum ( $\sum w_k x^{(k)}$ ), analogous to synthesizing specialized subtasks.

Key Contributions

1. Identification of Three Natural Mechanisms

The authors identify three mechanisms through which aggregation can expand the set of elicitable outputs:

Feasibility Expansion: Aggregation produces an output $x^{(A)}$ that lies outside the feasible set of any individual agent ( $Cx^{(A)} \not\le 0$ ), effectively bypassing model capability constraints.
Support Expansion: Aggregation combines outputs with smaller supports (fewer non-zero dimensions) into an output with a richer support, overcoming limitations in reward specification where specific feature combinations are hard to elicit directly.
Binding Set Contraction: Aggregation combines outputs where constraints are binding (active) into an output where those constraints are no longer binding (interior), leveraging the geometry of the feasible set to access new directions.

2. Necessary and Sufficient Conditions

The paper provides a rigorous characterization of when aggregation is powerful:

Necessity: Theorem 3.7 proves that for an aggregation to be elicitability-expanding, it must implement at least one of the three mechanisms (feasibility expansion, or support/binding-set expansion for every agent). If none are implemented, aggregation offers no power regardless of prompt engineering limitations.
Insufficiency of Basic Mechanisms: The authors demonstrate that merely implementing these mechanisms is not sufficient to guarantee elicitability expansion.
Strengthened Characterization: Theorems 4.3 and 4.4 provide the necessary and sufficient conditions for elicitability expansion. These conditions require a "strengthened" version of the mechanisms, specifically involving a "power-characterizing condition" based on feasible, budget-reducing directions. This condition ensures that the aggregated output is robustly distinct from the individual outputs in a way that prevents any single reward function from eliciting it.

3. Empirical Illustration

The authors empirically validate their theoretical findings using GPT-4o-mini (and other models) in a toy reference-generation task.

Task: Generating lists of paper references reflecting specific topic balances.
Setup: The system designer prompts models with high-level topics (coarser features) and aggregates the resulting lists via union (addition) or intersection.
Results: The experiments successfully constructed aggregation operations that implemented each of the three mechanisms. In each case, the aggregated output was significantly closer to the target than the closest output elicitable by a single model, confirming that aggregation can overcome both prompt engineering and model capability limitations in practice.

Results and Significance

The paper concludes that the power of compound AI systems relying on aggregation is not automatic; it is fundamentally constrained by the geometry of the output space and the limitations of the reward specification.

Theoretical Insight: The work fully characterizes when aggregation expands the set of elicitable outputs. It establishes that aggregation is only beneficial if it exploits specific geometric properties (feasibility, support, or binding sets) that single-agent prompting cannot access.
Practical Implication: System designers can use these mechanisms to determine if a compound system will outperform a single model. For instance, if models have no capability limitations (no conic constraints), intersection aggregation may offer no power, whereas addition aggregation might still provide benefits through support expansion.
Robustness: The empirical results suggest these theoretical mechanisms hold even when moving beyond the paper's linear assumptions (e.g., to stochastic LLMs with non-zero temperature).

The paper positions itself as a step toward understanding the fundamental limits and capabilities of aggregating homogeneous models, offering a conceptual framework for designing compound AI systems that effectively overcome the dual limitations of prompt engineering and model capabilities.