💬 NLP

More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding

This paper demonstrates that adding more scaffolding components to LLM agents often leads to destructive cross-component interference, proving that task-specific subset selection outperforms the conventional "all-in" approach and that greedy selection methods are unreliable due to frequent submodularity violations.

Original authors: Ming Liu

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ming Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate "super-brain" for a computer program (an AI agent) to solve difficult puzzles. The standard advice from experts has always been: "More is better." The idea is that if you give the AI a planner, a memory bank, a set of tools, a way to reason step-by-step, and a way to check its own work, it will perform perfectly.

This paper argues that this advice is often wrong. In fact, piling on too many features can actually make the AI dumber.

Here is the breakdown of what the researchers found, using simple analogies.

1. The "Too Many Cooks" Problem

The researchers tested every possible combination of five common AI features (Planning, Tools, Memory, Reasoning, and Self-Reflection) on two different types of puzzles:

HotpotQA: Like a trivia game where you have to search through many documents to find an answer.
GSM8K: Like a math homework assignment.

They found that for the smaller AI models (the "8B" version), adding extra features to the most basic, effective setup actually hurt the performance.

The Analogy: Imagine you are trying to find a specific needle in a haystack.
- The Best Setup: You just use a magnet (the "Tool"). It works great.
- The "All-In" Setup: You give the person holding the magnet a map (Planning), a notebook (Memory), a flashlight (Reasoning), and a coach yelling instructions (Reflection).
- The Result: The person gets confused. The map distracts them from the magnet; the coach's voice drowns out their own thoughts. They drop the needle. The simple magnet alone was 32% better than the whole fancy kit.

2. It Depends on the Task and the "Brain Size"

The paper shows that the "best" number of features changes depending on what the AI is doing and how smart the AI is.

Task Dependence:
- For the Trivia Game, the best setup was just one feature: The Tool. Adding anything else made it worse.
- For the Math Game, the best setup was three features: Tool + Reasoning + Reflection. Here, the "coach" and the "step-by-step thinking" actually helped. But adding a "Planner" or "Memory" still made it worse.
Brain Size (Model Scale):
- Small AI (8B): Very sensitive. Adding extra features causes a lot of interference. The gap between the simple setup and the "All-In" setup was huge (32%).
- Medium AI (70B): Stronger. It can handle more features, but the "All-In" setup is still not the best. The gap shrank to 19%.
- Very Smart AI (Claude Haiku): So capable that adding extra features doesn't help or hurt much. It's like a genius who can ignore distractions. The gap disappeared, but the simple setup was still just as good as the complex one.

3. Why Does This Happen? (The "Crowded Room" Effect)

The researchers call this Cross-Component Interference (CCI).

The Analogy: Imagine a small, crowded room where a team is trying to solve a problem.
- If you add a "Planner" who keeps shouting about the schedule, the "Tool User" can't hear the instructions.
- If you add a "Memory" person who keeps reminding everyone of old facts, the "Reasoner" gets stuck in the past.
- The components are fighting for the AI's attention (its "context window"). Instead of working together, they trip over each other.

4. You Can't Just "Add and Check"

A common way to build these systems is "greedy selection": Start with nothing, add one feature, see if it helps. If it does, keep it. Add another. If it helps, keep it. Stop when it stops helping.

The paper says this method is unreliable.

The Analogy: Imagine you are building a sandwich.
- Bread + Ham = Good.
- Bread + Ham + Cheese = Better.
- Bread + Ham + Cheese + Pickles = Worse (because the pickle makes the ham soggy).
- But, Bread + Ham + Cheese + Pickles + Mustard = Amazing again!
- If you stopped at the "Worse" stage because the pickle hurt, you would never find the amazing sandwich. The paper found that sometimes a feature is bad alone but great when combined with specific others. You have to test the whole group, not just add one by one.

5. The Main Takeaway

The researchers ran over 32,000 tests to prove this. Their conclusion is simple:

Don't assume that a "fully loaded" AI agent is the best one.

For many tasks, a simple agent with just the right tool is better than a complex agent with a planner, memory, and reflection.
The "best" setup depends entirely on the specific job and the specific AI model you are using.
If you just throw everything at the wall, you are likely creating noise that confuses the AI, rather than helping it.

In short: Sometimes, the simplest tool is the most powerful one. Adding more parts doesn't always make the machine run faster; sometimes, it just makes it stumble.

Technical Summary: More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffolding

Problem Statement

The standard paradigm for constructing Large Language Model (LLM) agents involves "scaffolding": augmenting a base model with a stack of functional components such as planning modules, tool interfaces, memory, self-reflection, and retrieval. The prevailing assumption in the field is that adding more components improves performance ("more is better"). This paper challenges that assumption by investigating Cross-Component Interference (CCI): the phenomenon where combining scaffolding components results in negative marginal returns, causing the full "All-In" system to underperform specific subsets of those components.

Methodology

The authors conducted a systematic, full-factorial empirical study to map the interaction landscape of agent components.

Experimental Design: A full factorial design over all $2^5 = 32$ $2^{5} = 32$ subsets of five canonical scaffolding components:
1. Planning (P): Decomposing tasks into sub-goals.
2. Tool Use (T): Function-calling interfaces.
3. Memory (M): Structured working memory.
4. Structured Reasoning (SR): Chain-of-thought formatting.
5. Reflection (R): Self-evaluation prompts.
Benchmarks & Models: Experiments were conducted on two benchmarks: HotpotQA (multi-hop reasoning, retrieval-focused) and GSM8K (math reasoning). The study spanned three model families and five scales:
- Llama-3.1 (8B, 70B)
- Qwen2.5 (3B, 7B)
- Claude Haiku 4.5 (via AWS Bedrock)
Scale: The study involved 118 controlled configurations and over 32,000 evaluations. Key configurations were validated across 10 random seeds (16,000 evaluations on HotpotQA/8B).
Statistical Analysis: The authors employed paired t-tests, Wilcoxon signed-rank tests, Bayesian analysis (JZS priors), and effect size measurements (Cohen's $d_z$ ). They utilized exact Shapley value decomposition to attribute value to components and tested for submodularity violations. Regression models (main-effects vs. pairwise interaction) were fitted to characterize the interaction structure.

Key Results

1. Prevalence of Cross-Component Interference (CCI)

In every setting tested, the best proper subset of components matched or exceeded the performance of the full "All-In" system.

HotpotQA (Llama-3.1-8B): The single-tool agent (Tool Use only) significantly outperformed the 5-component All-In system by 32% (F1 0.233 vs. 0.177; $p=0.023$ ). Adding any component to the Tool Use baseline degraded performance in 5 out of 6 expansions.
GSM8K (Llama-3.1-8B): The optimal configuration required three components (Tool Use + Structured Reasoning + Reflection), achieving 79% higher accuracy than the All-In system (0.43 vs. 0.24).
Optimal Count ( $k^*$ ): The optimal number of components is task-dependent ( $k^*=1$ for HotpotQA, $k^*=3$ for GSM8K) and scale-sensitive.

2. Scale and Model Dependence

CCI severity follows a capability gradient:

8B Models: CCI is strong and pervasive.
70B Models: CCI is attenuated but persists. On HotpotQA, the gap between the best subset and All-In narrowed from 32% (8B) to 19% (70B). Notably, the direction of interference reversed for some components (e.g., adding Reflection helped at 70B but hurt at 8B), yet the All-In system still did not outperform the best subset.
Claude Haiku 4.5: CCI saturated to within-noise levels. While the All-In system was not worse than simpler subsets, the additional components beyond Tool Use contributed virtually nothing (0.2% gain), indicating a "plateau" where complexity yields zero marginal return.

3. Structural Analysis and Interaction Models

Main-Effects Dominance: A 6-parameter main-effects regression model explained 91.6% of the variance ( $R^2=0.916$ ), decisively outperforming a 16-parameter pairwise interaction model ( $\Delta BIC = 25.3$ ). This suggests CCI is largely additive; components impose a relatively constant marginal cost or benefit regardless of context.
Shapley Values: Tool Use captured 70% of the total scaffold value. Conversely, Planning had a significantly negative Shapley value ( $\phi = -0.029$ ), and Memory was directionally negative on HotpotQA.
Non-Submodularity: The study found 183 violations of submodularity out of 325 testable triples (56.3%). The median submodularity ratio was $\gamma_{med} = 0.52$ . Crucially, 90% of top violations exhibited sign-flipping: components that were harmful in isolation became beneficial in specific combinations (e.g., Structured Reasoning hurt Tool Use alone but helped when combined with Planning and Reflection). This renders greedy selection strategies empirically unreliable.

4. Robustness

Cross-Family Replication: The qualitative pattern (Tool Use dominates, All-In is suboptimal) replicated across Qwen2.5 and Llama families.
Template Robustness: The core finding held across three semantically equivalent prompt template variants, though fine-grained configuration rankings were sensitive to wording.
Length Control: Real components outperformed length-matched meaningless padding by 6–9×, confirming CCI is driven by semantic content, not token count.

Significance and Claims

The paper claims that the assumption "more is better" is systematically wrong for prompt-based scaffolding. The authors argue that:

Practitioner Defaults are Flawed: The default of deploying maximally-equipped agents leads to performance degradation due to interference.
Task-Specific Selection is Required: Optimal agent design requires selecting task-specific subsets informed by interaction-aware analysis rather than stacking all available modules.
Greedy Selection Fails: Due to sign-flipping and non-submodularity, adding components one-by-one until marginal returns turn negative is an unreliable strategy for finding the optimal agent configuration.
Mechanism: The paper identifies CCI as a form of "context-window crowding" (an inference-time analogue to gradient conflict in multi-task learning), where components compete for attention or dilute signals, though the exact causal mechanism remains an open question.

The study concludes that while larger models (70B+) and highly capable closed-source models (Claude Haiku) can tolerate wider component combinations, the simplest adequate scaffold consistently matches or beats the full agent across all tested scales.