💬 NLP

ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces

This paper introduces ACAR, a measurement framework that routes tasks across multi-model ensembles using self-consistency variance to achieve accuracy surpassing two-model baselines while avoiding full ensembling on over half of tasks, alongside documenting critical limitations regarding retrieval augmentation, agreement-on-wrong failure modes, and the inadequacy of proxy signals for attribution.

Original authors: Ramchand Kumaresan

Published 2026-02-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ramchand Kumaresan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a high-end consulting firm. You have three brilliant experts: Claude, GPT, and Gemini.

Sometimes, your clients ask simple questions like, "What's the capital of France?" You don't need to call a meeting with all three experts for that; one is enough. But other times, clients ask incredibly hard questions like, "How do I build a self-driving car that never crashes?" For those, you need all three experts to brainstorm together to get the right answer.

The problem? Calling all three experts for every question is expensive and slow. Calling just one expert for every question is cheap but risky because they might make a mistake.

This paper introduces a smart system called ACAR (Adaptive Complexity Routing) that acts like a super-efficient project manager to decide when to call one expert and when to call the whole team.

Here is how it works, broken down into simple concepts:

1. The "Vibe Check" (The Routing Mechanism)

Before asking the expensive experts, the system asks a fast, cheap "probe" model (Gemini) to answer the question three times in a row.

Scenario A (The Easy Task): The probe gives the same answer three times.
- The Manager's thought: "Okay, everyone agrees. This is easy. Let's just send the final answer and save money."
- Result: Only one model is used.
Scenario B (The Tricky Task): The probe gives two different answers (e.g., two say "Yes," one says "No").
- The Manager's thought: "Hmm, there's some disagreement. Let's bring in two experts to double-check."
- Result: Two models are used.
Scenario C (The Hard Task): The probe gives three completely different answers.
- The Manager's thought: "Total chaos! This is a nightmare. We need the full team of three experts to solve this."
- Result: All three models are used.

The Magic: By doing this "vibe check" first, the system saves money on easy tasks while still getting high-quality answers for hard ones.

2. The "Audit Trail" (Why This Paper is Special)

Most AI research is like a black box: you put data in, get results out, and hope it works. This paper is different. The researchers built a system called TEAMLLM that is like a glass-walled factory.

Every single decision, every dollar spent, and every time the models agreed or disagreed is recorded in an unchangeable log. It's like a flight recorder on a plane. If something goes wrong, you can look back and see exactly why. This makes the results trustworthy and reproducible.

3. The Surprising "Oops" Moments (What Didn't Work)

The researchers tried a few things they thought would help, but they actually made things worse. These are the "negative results" that are just as important as the successes:

The "Bad Reference" Problem: They tried to give the models a "cheat sheet" of past similar questions (Retrieval Augmentation).
- Analogy: Imagine asking a chef to cook a steak, but you hand them a recipe for a cake because they look similar on the page.
- Result: It confused the models and made them worse. It turns out, just having "more info" isn't helpful if the info doesn't actually match the problem.
The "Groupthink" Trap: Sometimes, all three experts agree on the wrong answer.
- Analogy: If three people in a room all confidently say, "The sky is green," and you trust their agreement, you are still wrong.
- Result: The system couldn't fix this. If the initial "vibe check" agrees on a wrong answer, the system thinks the task is easy and stops there. This is a hard limit on how smart the system can get.

4. The Bottom Line

The ACAR system is a smart way to balance cost and quality.

Without it: You either pay a fortune to use all experts all the time, or you risk bad answers by using only one.
With it: You get answers that are better than using just two experts, but you spend less money because you skip the expensive "full team meeting" for the easy stuff.

In short: This paper teaches us that you don't need a complex AI brain to decide which AI to use. Sometimes, a simple "let's ask three times and see if they agree" rule is the best way to save money and get good results. It also warns us that blindly trusting "more data" or "group agreement" can sometimes lead us astray.

1. Problem Statement

The paper addresses the fundamental trade-off in deploying Large Language Models (LLMs) between quality and cost.

The Challenge: A single model may fail on complex tasks, while running multiple models in parallel (ensembling) improves reliability but multiplies inference costs.
The Gap: Existing solutions are insufficient:
- Learned Routers: Require training data, suffer from distribution shift, lack interpretability, and cannot handle credit assignment in collaborations.
- Observability Platforms: Provide post-hoc analysis but do not influence routing decisions in real-time.
The Goal: Develop a principled, adaptive routing mechanism that allocates compute based on task difficulty before execution, without relying on learned classifiers, while ensuring full auditability and reproducibility.

2. Methodology

A. The Execution Substrate: TEAMLLM

The authors built TEAMLLM, a deterministic execution substrate designed for auditable multi-model experiments. It enforces three invariants:

Deterministic Execution: Every run captures seeds, prompt hashes, and environment fingerprints to ensure identical re-execution.
Immutable Artifacts: All responses and decision traces are append-only, preventing modification of historical data.
Forward-Only State Machine: Runs progress through defined states (Pending → Executing → Verifying → Completed) with no rollback.

B. The Core Mechanism: ACAR (Adaptive Complexity & Attribution Routing)

ACAR uses Self-Consistency Variance ( $\sigma$ ) as a proxy for task difficulty. Instead of training a classifier, it uses a heuristic based on the agreement of a fast "probe" model.

Probe Phase: A fast model (Gemini 2.0 Flash) generates $N=3$ independent samples for a given task.
Variance Calculation: The system extracts canonical answers ( $a_1, a_2, a_3$ $a_{1}, a_{2}, a_{3}$ ) and computes $\sigma$ $σ$ :
- $\sigma = 0.0$ : All 3 answers agree (Task is likely easy).
- $\sigma = 0.5$ : 2 out of 3 answers agree (Task is ambiguous).
- $\sigma = 1.0$ : All 3 answers differ (Task is hard/complex).
Routing Logic:
- $\sigma = 0.0$ (Single-Agent): Routes to a single high-quality model.
- $\sigma = 0.5$ (Arena-Lite): Routes to a 2-model ensemble for verification.
- $\sigma = 1.0$ (Full-Arena): Routes to a 3-model ensemble with a judge to select the best answer.

C. Experimental Setup

Models: Claude Sonnet 4, GPT-4o, and Gemini 2.0 Flash (used as the probe).
Benchmarks: 1,510 tasks across MathArena, Reasoning Gym, LiveCodeBench, and SuperGPQA.
Configurations: Compared Single-Model, 2-Model Ensemble, 3-Model Ensemble, and two ACAR variants (with and without retrieval augmentation).

3. Key Contributions

ACAR Framework: An adaptive routing mechanism that achieves 55.6% accuracy, outperforming a static 2-model baseline (54.4%) while avoiding full ensembling on 54.2% of tasks.
Negative Results & Falsifiable Baselines: The paper explicitly documents what fails, providing critical insights for the community:
- Retrieval Augmentation: Naive retrieval (without strict semantic alignment) decreased accuracy by 3.4 percentage points due to noise injection.
- Attribution Proxies: Estimates based on response similarity or entropy showed weak correlation with ground-truth counterfactuals; true attribution requires explicit counterfactual computation.
- Agreement-But-Wrong: When models unanimously agree on a wrong answer ( $\sigma=0$ ), no downstream ensemble can recover. This creates an irreducible accuracy ceiling (8pp below full ensembling).
TEAMLLM Infrastructure: Release of a deterministic substrate with 7,550+ auditable runs, enabling reproducible multi-model research.

4. Key Results

Performance vs. Cost: ACAR-U (without retrieval) achieved 55.6% accuracy at a cost of $20.34, outperforming the static 2-model ensemble (54.4% accuracy, $20.64) and the single-model baseline (45.4% accuracy, $17.04).
Efficiency: By routing 32.9% of tasks to single-agent mode and 21.3% to "arena-lite," ACAR avoids the expensive full-ensemble mode for the majority of tasks.
Retrieval Failure: The variant with retrieval (ACAR-UJ) dropped to 52.4% accuracy. Analysis showed a median retrieval similarity of only 0.167, indicating that high hit rates do not equate to semantic relevance.
Latency: ACAR-U achieved intermediate latency by routing easy tasks to faster single-model execution, avoiding the coordination overhead of full ensembles.

5. Significance and Implications

Shift from Optimization to Measurement: Unlike prior work focused on maximizing routing accuracy via learned models, ACAR prioritizes auditability, interpretability, and reproducibility. It treats cost-aware routing as a measurement problem.
Redefining Retrieval Utility: The paper provides strong evidence that "more context" is not automatically beneficial. Retrieval augmentation requires high semantic alignment (similarity > 0.7); otherwise, it introduces harmful noise.
Limits of Self-Consistency: The study identifies a fundamental theoretical bound: systems relying on self-consistency for difficulty estimation cannot recover from "unanimous incorrect agreement."
Attribution Reality: It demonstrates that inferring model contribution from observational data (proxies) is unreliable; rigorous attribution requires explicit counterfactual computation.

In summary, ACAR provides a robust, model-agnostic framework for adaptive inference that balances cost and quality, while simultaneously offering a critical "reality check" on common assumptions regarding retrieval and attribution in multi-model systems.