🤖 AI

Locally Coherent, Globally Incoherent: Bounding Compositional Incoherence in Multi-Component LLM Agents

This paper formalizes and quantifies the "locally coherent, globally incoherent" failure mode in multi-component LLM agents using a runtime-computable compositional residual, demonstrating that local coherence often fails to ensure global probability consistency and that current LLM-side mitigations are ineffective while deterministic projection and sequential monitoring offer viable solutions.

Original authors: Anany Kotawala

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anany Kotawala

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too Many Chefs" Dilemma

Imagine you are running a restaurant where you have hired four different expert chefs to plan a single, massive banquet.

Chef A is only told to plan the Appetizers.
Chef B is only told to plan the Main Course.
Chef C is only told to plan the Dessert.
Chef D is only told to plan the Drinks.

Each chef is a genius. They are perfectly confident in their own section. Chef A says, "I'm 80% sure we'll need 50 appetizers." Chef B says, "I'm 80% sure we'll need 50 main courses." And so on.

The Problem: The restaurant owner (the "Agent") just adds up all the numbers. But because the chefs didn't talk to each other, they might accidentally plan for 200 people when the restaurant only seats 100. Or, they might plan for 100% chance of appetizers and 100% chance of main courses, even though the menu only allows for one of those to happen per table.

In the world of AI, this is called "Locally Coherent, Globally Incoherent."

Locally Coherent: Each AI model (chef) is doing a good job on its own tiny task.
Globally Incoherent: When you put their answers together, the total math doesn't make sense. The probabilities add up to more than 100% (or less), creating a logical mess.

The Paper's Main Discovery

The authors found that even if every single AI model is perfectly calibrated and logical on its own, combining them often breaks the math.

They call this broken math a "Compositional Residual" ( $\epsilon^\star$ ). Think of this as a "Garbage Meter."

If the meter reads 0, the combined answer is perfect.
If the meter reads high, the combined answer is logically broken (e.g., claiming there is a 150% chance of rain).

They tested this on nearly 2,000 different scenarios (like predicting election winners, sports outcomes, or tech IPOs) using four different AI models. They found that 33% to 94% of the time, the combined answer was broken, even though the individual AI parts were working fine.

Why Does This Happen?

The paper explains that the AI models are "blind" to the big picture.

The Blind Specialist: When you ask an AI, "What is the chance the US wins the World Cup?", it answers based on its training.
The Missing Context: It doesn't know that you also asked another AI, "What is the chance Brazil wins?" and "What is the chance Germany wins?"
The Result: The US AI says 60%, Brazil says 60%, and Germany says 60%. The total is 180%. This is impossible. The AI didn't know it was part of a team where the total must equal 100%.

The Solutions They Tried (and Failed)

The researchers tried to fix this using "soft" methods, which are like giving the chefs better instructions:

Retrieval: Letting the chefs look up facts on Google. (Failed: They still couldn't agree on the total).
Better Prompts: Telling the chefs, "Remember, you are part of a team, and your answers must add up to 1." (Failed: They still made mistakes).
The "Boss" AI: Having a smart AI manager listen to all the chefs and rewrite the answers. (Failed: The manager often made things worse or didn't fix the math).

The Conclusion: You cannot fix this just by talking to the AI better. The AI models are too isolated by design.

The Real Fix: The "Mathematical Corrector"

Since you can't fix the chefs, you need a Mathematical Corrector that steps in after the chefs are done.

The authors propose a method called Hierarchical Boyle–Dykstra Projection.

The Analogy: Imagine the chefs hand you a list of ingredients that weighs 200kg, but your truck can only carry 100kg. Instead of asking the chefs to rewrite their lists (which they do poorly), you simply use a machine to trim the excess weight off the top of the pile until it fits exactly in the truck, while keeping the relative proportions of the ingredients as close to the original as possible.
How it works: The system takes the broken, over-100% answer and uses a strict mathematical formula to "project" it onto a valid, logical shape (a "polytope"). It forces the numbers to obey the laws of probability without needing the AI to "think" about it.

This method is deterministic (it always works the same way) and fast. It turns a broken, 150% probability answer into a perfect 100% answer instantly.

Why Should You Care? (The "Regret" Factor)

The paper shows that this isn't just a theoretical math problem; it costs real money in decision-making.

If you are a bettor or a decision-maker using these AI agents, a broken answer leads to regret. You might bet on a horse that the AI says has a 60% chance, but because the AI's math was broken, that 60% was actually a lie.
By using their "Mathematical Corrector," they showed that you can save money (measured in "nats," a unit of information gain) because you stop making bets based on impossible math.

Summary in One Sentence

When you ask multiple AI models to solve parts of a big puzzle, they often give answers that don't add up; the best way to fix this isn't to ask them to try harder, but to use a mathematical "trimming tool" to force their answers to make sense before you use them.

Technical Summary: Locally Coherent, Globally Incoherent

1. Problem Statement

Multi-component Large Language Model (LLM) agents often decompose complex joint problems into sub-tasks handled by specialist sub-agents or tools. While individual components may be locally coherent (internally consistent and calibrated on their specific sub-questions), the aggregation of their outputs can violate basic probability axioms. This phenomenon, termed "locally coherent, globally incoherent," results in a composed belief state that assigns a total probability mass greater than 1 (or violates logical constraints like $P(A) + P(\neg A) = 1$ ).

Existing evaluation metrics (per-question accuracy, calibration, conformal prediction) operate on individual model outputs and fail to detect cross-component logical violations. Consequently, multi-component agents can produce "Dutch-book" exposures—combinations of bets that guarantee a loss for the system—strictly due to the incoherence introduced during composition, even when every component is individually sound.

2. Methodology

Theoretical Framework

The paper formalizes the failure mode using convex geometry and probability theory:

Compositional Residual ( $\varepsilon^\star$ ): Defined as the $L_2$ distance between the composed quote (after local repairs) and the joint coherent polytope ( $M_\star$ ). This polytope represents the set of all probability vectors satisfying the joint logical constraints (e.g., partitions, negations, conjunctions).
Product-Structure Dichotomy (Theorem 3.3): The paper establishes a necessary and sufficient condition for local coherence to imply global coherence. Under "owner-selected aggregation" (where each component owns a distinct subset of coordinates), global coherence is guaranteed if and only if the joint polytope factorizes as a Cartesian product of local polytopes ( $M_\star = M_\boxtimes$ ). If cross-component coupling constraints exist (e.g., a partition summing to 1), local coherence is insufficient, and $\varepsilon^\star > 0$ is possible.
Magnitude Prediction (Corollary 3.9): The expected squared residual is predicted by a Rayleigh-quotient form based on the covariance of the specialist panel and the normal vector of the binding constraint. This allows for a closed-form prediction of incoherence magnitude before routing occurs.
Exposure Bound (Corollary 3.5): Under the Fundamental Theorem of Asset Pricing (FTAP), the system-level Dutch-book exposure is bounded by $\sqrt{m^\star} \varepsilon^\star$ , where $m^\star$ is the number of joint coordinates.

Algorithmic Repair

Hierarchical Boyle–Dykstra Projection: To repair incoherent compositions deterministically, the authors propose a cyclic projection algorithm. This iteratively projects the composed quote onto local polytopes and the coupling set (constraints) until convergence to the joint coherent polytope.
Sequential Monitoring: An anytime-valid e-process is introduced to monitor streams of compositional residuals over time, allowing operators to detect persistent incoherence without fixing a stopping horizon in advance.

Experimental Setup

The authors evaluated 1,876 ensemble cliques across four logical relation classes: Negation, Conjunction, Disjunction, and Partition.

Models: A mid-tier panel (Claude-Haiku-4.5, GPT-5.4-mini/nano, Llama-3.3-70b) and a frontier panel (including GPT-5.5, Claude-Opus-4.7, DeepSeek-V3.2).
Benchmarks: Paleka (cross-question consistency) and Polymarket (partition events).
Protocols: Random assignment of questions to specialists, structured routing, and planner-discretion harnesses.

3. Key Results

Empirical Findings

Prevalence of Incoherence: $\varepsilon^\star > 0$ was observed on 33–94% of cliques, depending on the relation class. Partition tasks showed the highest failure rate (94%), followed by negation (66%), disjunction (43%), and conjunction (33%).
Magnitude Prediction: The Rayleigh-quotient prediction matched observed residuals within 7% for three of the four relation classes. The conjunction class under-predicted slightly (0.83×) due to the panel mean lying in the interior of the Fréchet halfspace, a regime predicted by the theory.
Impact on Decision Making: On 1,770 resolved bets, naive composition incurred a regret of +0.115 nats per bet under a proportional allocation rule. Hierarchical JCD repair reduced this to near-zero, matching the performance of a single-LLM oracle.
Failure of LLM-Side Mitigations:
- Retrieval: Augmenting specialists with search tools failed to reduce $\varepsilon^\star$ ; in 20/30 cases, it increased incoherence.
- Prompt Engineering: Partition-aware prompting reduced residuals but left 53% of partitions with $\varepsilon^\star > 0.05$ .
- Aggregator LLMs: Delegating coherence to an aggregator LLM reduced the mean residual but still resulted in 15% of partitions exceeding the 0.05 threshold and 7% regressing relative to naive composition.
- Frontier Models: Even with a frontier-only panel, $\varepsilon^\star > 0$ persisted on 97.8% of partition bets, though the magnitude decreased by 39%.

Mechanism Validation

Coupling Visibility: An experiment where specialists were explicitly shown the partition constraints and peer quotes ("INFORMED") reduced $\varepsilon^\star$ significantly compared to isolated prompts ("BLIND"), confirming that the failure arises because specialists lack visibility into cross-component constraints.
Routing Topology: Structured routing (assigning specific models to specific relation types) did not eliminate incoherence, though it altered the distribution of residuals.

4. Contributions

Formalization of a Failure Mode: The paper defines and characterizes the "locally coherent, globally incoherent" regime, proving that per-component calibration is insufficient for system-level coherence under coupling.
Runtime Certificate ( $\varepsilon^\star$ ): Introduces a distribution-free, computable metric for system-level incoherence that requires only the composed output and declared constraints.
Theoretical Dichotomy: Provides a precise condition (Theorem 3.3) for when local coherence suffices, distinguishing between product-structure factorization and coupled constraints.
Constructive Repair: Demonstrates that hierarchical Boyle–Dykstra projection can deterministically eliminate $\varepsilon^\star$ at the cost of a single quadratic programming (QP) solve, outperforming all prompt-based or LLM-aggregator alternatives.
Empirical Benchmark: Establishes a decomposable benchmark where the hardness ordering (Partition > Negation > Disjunction > Conjunction) aligns with theoretical predictions regarding polytope restrictiveness.

5. Significance and Claims

The paper claims that system-level coherence is a distinct property from per-component calibration that must be explicitly managed in multi-agent systems. The primary contribution is the operational reframing of $\varepsilon^\star$ as a runtime certificate that can detect and repair Dutch-book exposures that no individual component can identify.

The authors emphasize that:

Prompt engineering and retrieval are insufficient to solve this problem; the failure is structural to the composition of isolated probabilistic forecasts.
Geometric repair is necessary and sufficient to restore coherence when the coupling set is explicit.
The failure persists across model scales, indicating that increasing model capability does not inherently solve compositional incoherence.

The work positions $\varepsilon^\star$ as a critical guardrail for high-stakes applications (forecasting, decision support) where logical consistency is paramount, offering a deterministic fix that complements existing uncertainty quantification methods like conformal prediction.