More Agents Improve Math Problem Solving but Adversarial Robustness Gap Persists

1. Problem Statement

While Large Language Models (LLMs) have shown remarkable progress in mathematical reasoning, their reliability in real-world deployments is threatened by input noise. Previous research has established two key facts:

1. Multi-agent collaboration (e.g., sampling and voting) significantly boosts reasoning accuracy on clean inputs.
2. Adversarial perturbations (typos, punctuation noise) severely degrade the performance of single LLMs.

The central research question of this paper is: Does increasing the number of collaborating agents (via a sampling-and-voting framework) improve robustness against adversarial inputs, or does the performance gap between clean and noisy conditions persist?

2. Methodology

2.1 Experimental Framework: Agent Forest

The authors utilize a unified sampling-and-voting framework called "Agent Forest."

• Process: For a given question $x$, the same base model $M$ is queried $N$ times with independent random seeds to generate $N$ candidate answers.
• Aggregation: The final answer is determined by majority voting over the canonicalized numeric answers.
• Variable: The number of agents ($n$) is varied across $\{1, 2, 5, 10, 15, 20, 25\}$.
• Models: Six open-source models were tested across various parameter scales:
  • Qwen3 (4B, 14B)
  • Llama 3.1 (8B)
  • Mistral (7B)
  • Gemma 3 (4B, 12B)

2.2 Datasets

Evaluation was conducted on four standard mathematical benchmarks:

• GSM8K: Grade school word problems.
• MATH: Advanced competition-level problems.
• MMLU-Math: Multiple-choice mathematical reasoning.
• MultiArith: Simple multi-step arithmetic.

2.3 Noise Taxonomy

The study introduces a structured taxonomy of adversarial noise to simulate real-world and synthetic errors:

1. Synthetic Punctuation Noise: Random insertion of punctuation symbols (., ,, !, etc.) at three intensities: 10%, 30%, and 50% of the question length.
2. Human-like Typos (WikiTypo): Real-world spelling errors extracted from Wikipedia edit histories, replacing words with authentic typo variants.
3. Adversarial Typo Attack (R2ATA): Iterative character-level substitutions, duplications, and deletions targeting query-critical words (based on the Adversarial Typo Attack framework).

2.4 Metrics

• Accuracy: The fraction of correct answers.
• Attack Success Rate (ASR): The proportion of originally correct predictions (on clean data) that become incorrect after noise is applied. A lower ASR indicates better robustness.

3. Key Contributions

1. Systematic Evaluation of Multi-Agent Robustness: This is the first study to systematically isolate the effect of agent count on adversarial robustness in math reasoning, moving beyond simple accuracy improvements on clean data.
2. Noise Taxonomy for Math: The paper categorizes noise into synthetic (punctuation) and human-like (typos), demonstrating that the type of noise dictates the failure mode.
3. The "Consensus Paradox": The authors identify a critical failure mode where multi-agent systems fail to correct adversarial errors because the noise induces systematic bias rather than random variance. Agents converge on the same erroneous output, rendering majority voting ineffective for robustness.

4. Key Results

4.1 Accuracy vs. Agent Count

• Collaboration Improves Accuracy: Increasing the number of agents reliably improves accuracy across all noise types.
• Diminishing Returns: The largest gains occur between $n=1$ and $n=5$. Beyond $n \approx 10$, performance gains plateau.
• Scale Independence: While larger models (e.g., Qwen3-14B, Gemma3-12B) maintain higher absolute accuracy, smaller models benefit more relatively from collaboration.

4.2 The Robustness Gap (ASR)

• Persistent Gap: Despite accuracy improvements, the Attack Success Rate (ASR) remains nearly constant as the number of agents increases.
• Noise Type Matters:
  • Punctuation Noise: Vulnerability is reduced significantly with more agents (up to $n=5$ or $10$), as the semantic meaning is often preserved.
  • Human Typos (WikiTypo): This remains the dominant bottleneck. Even with 25 agents, ASR stays high (often >10%). The gap between Clean accuracy and WikiTypo accuracy is the largest.
  • R2ATA: Lies between punctuation and WikiTypo in terms of difficulty.

4.3 Dataset Specifics

• MATH & MMLU: These complex datasets show the highest ASR and the least recovery from collaboration, indicating that long, multi-step reasoning chains are highly sensitive to input distortions.
• GSM8K & MultiArith: Easier datasets show lower ASR and greater benefit from increasing agent counts, though the robustness gap never fully closes.

5. Discussion & Analysis

The paper attributes the persistence of the robustness gap to two main factors:

1. Semantic vs. Morphological Noise: Punctuation noise often preserves the morphological root of keywords, allowing attention mechanisms to recover. In contrast, human typos (WikiTypo) fundamentally alter the token embeddings, causing a divergence from the target semantic vector that the model cannot bridge, even with voting.
2. The Consensus Paradox: Ensemble theory assumes errors are independent. However, adversarial noise creates a systematic bias. When the input is perturbed, all agents in the ensemble are likely to fail in the same specific way (e.g., misinterpreting a typo as a different word). Consequently, majority voting replicates the systematic failure rather than mitigating it.

6. Significance and Implications

• Limitation of Scaling Agents: Simply adding more agents (scaling) is not a substitute for improving the base model's inherent robustness to noise.
• Deployment Risks: In safety-critical domains (medical, legal, financial), relying on multi-agent voting without addressing base-model tokenization robustness poses a severe risk, as simple typos can cause catastrophic failures.
• Future Directions: The authors suggest that future work must focus on:
  • Noise-aware sampling and aggregation.
  • Character-robust tokenization or pre-processing guardrails.
  • Training-time augmentation specifically targeting human-like typos.

Conclusion

The paper concludes that while multi-agent collaboration is a powerful tool for boosting accuracy on clean mathematical problems, it fails to close the adversarial robustness gap. The system remains vulnerable to human-like typos regardless of the number of agents, highlighting a fundamental limitation in current LLM architectures regarding non-standard orthography.