Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning

1. Problem Statement

The paper investigates the limits of Prompt Engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition.

• The Task: Determine if an equational law $E_1$ implies another law $E_2$ over all magmas (sets with a single binary operation, no axioms like associativity or commutativity).
• Computational Asymmetry:
  • False Instances: Decidable via finite search. A single finite counterexample (a small magma where $E_1$ holds but $E_2$ fails) proves the implication is false.
  • True Instances: Undecidable in general. Proving $E_1 \implies E_2$ requires showing no counterexample exists across all magmas (including infinite ones).
• The Challenge: Large Language Models (LLMs) must navigate this asymmetry without external symbolic tools. The study explores whether complex "cheatsheets" (prompts containing rules, counterexamples, and heuristics) can overcome the models' inherent inability to perform universal quantification or rigorous proof search.

2. Methodology

The authors conducted a systematic empirical study over five weeks, testing 45+ prompt variants across three models and four dataset splits.

• Models Evaluated:
  • gpt-oss-120b: Primary model (OpenAI's 117B MoE), accessed via DeepInfra.
  • Llama 3.3 70B: Instruction-tuned, accessed via Together AI.
  • Gemma 4 31B: Accessed via Together AI (with reasoning enabled) and OpenRouter (where reasoning was suppressed).
• Datasets:
  • hard1: False-heavy (35% True).
  • hard2: Balanced (50% True), structurally complex.
  • hard3: Near-balanced (49% True), the primary evaluation split ($n=400$).
  • normal: Standard problems to check for regression.
• Prompt Design Dimensions:
  • Content: Counterexample (CE) tables (size-2 and size-3 magmas), singleton-forcing rules (heuristics for True), and block rules (structural classifiers for False).
  • Ordering: The sequence of instructions (e.g., checking for trivial magmas before vs. after searching for counterexamples).
  • Length: Ranging from 0 bytes (baseline) to 4,878 bytes.
• Evaluation Metrics: Accuracy, True Recall, False Recall, and 3-model averages.

3. Key Contributions

1. The Single-Prompt Ceiling (Empirical Saturation Region): The authors identify a performance plateau where adding more complexity to a prompt yields unstable, non-generalizable gains. For gpt-oss-120b, this ceiling lies between 71–79% accuracy on balanced hard splits.
2. Distribution Mismatch Failure Mode: Demonstrated that prompts optimized for a specific distribution (e.g., False-heavy) can catastrophically fail on balanced distributions. A rule that improved False detection on hard1 caused a 51% error rate on True instances in hard2.
3. The "Merge" Limitation: Merging two complementary prompts (one good at True, one at False) does not yield the maximum of both; instead, the model averages the strategies, resulting in performance near the arithmetic mean of the parents.
4. Ordering Effects: The sequence of instructions significantly impacts performance due to attention mechanisms. Placing a "trivial magma check" (a shortcut for True) before the counterexample search outperformed the reverse order by ~7.5 percentage points.
5. Model-Specific Failure Modes:
   • Llama 3.3 70B: Suffers "cognitive load collapse" with prompts >2KB, dropping True recall to 0%.
   • Gemma 4 31B: Suffers from token budget exhaustion; if the reasoning trace exceeds the limit, it defaults to a True verdict.

4. Key Results

Performance on gpt-oss-120b (hard3, $n=400$)

• Baseline (No Cheatsheet): 59.75% accuracy (82.6% True recall, 38.0% False recall). The model is heavily biased toward "True."
• Best Local Result (AN45c): 79.25% accuracy (95.9% True recall, 63.4% False recall).
  • Key Innovation: Reordered instructions to check for trivial magmas first, bypassing the counterexample search for specific structural patterns.
  • Note: This result was achieved with a 2,252-byte prompt.
• Best Robust Result (AN38): 71.8% accuracy. Simpler than AN45c but more robust to distribution shifts.

Cross-Model Generalization

• Complexity vs. Portability: Complex prompts (e.g., AN3c, 4,306 bytes) that boosted gpt-oss-120b performance caused Llama 3.3 70B to collapse to 0% True recall.
• Minimal Effective Prompt (AN19c): A 289-byte prompt with only natural language hints achieved 62% on gpt-oss-120b and 60% on Llama 3.3 70B. It was the only variant to maintain meaningful True recall across all models.

Official Benchmark Validation (Post-Submission)

• The Distribution Gap: While AN45c achieved 79.25% locally on hard3, it scored 55.5% on the official SAIR benchmark (DeepInfra), falling 4.3 percentage points below the baseline.
• AN38 Superiority: The simpler AN38 prompt achieved 65.3% on the official benchmark, a +5.6pp improvement over the baseline.
• Cross-Distribution Trade-off: Analysis of 52 community submissions revealed a "trade-off surface." Prompts optimized for one distribution (e.g., 99% on hard2) often collapsed on the other (e.g., 56% on hard3). Only one submission achieved >65% on both simultaneously.

5. Significance and Implications

• Rethinking Prompt Engineering: The paper argues that for formal reasoning, "Less is More." Elaborate prompts do not teach models new mathematics; they only guide the application of existing knowledge. Beyond a certain complexity, cognitive load causes the model to default to surface heuristics or average conflicting strategies.
• The Router Hypothesis: The authors propose that LLMs in this domain act as heuristic classifiers rather than symbolic provers. They route problems to cached structural patterns (e.g., "variable appears once" $\to$ True) rather than deriving answers compositionally.
• Limitations of Static Prompts: Static text prompts cannot implement conditional routing (e.g., "If structure X, use Strategy A; else Strategy B") because LLMs interpolate between instructions probabilistically rather than executing them conditionally.
• Future Directions: To break the saturation ceiling, the authors suggest:
  1. Ensembles with External Routing: Using a separate classifier to route problems to specialized prompts.
  2. Fine-tuning: Training on the Equational Theories Project graph (22M pairs).
  3. Hybrid Architectures: Combining LLMs with symbolic verifiers (e.g., Mace4) to verify counterexamples.

Conclusion: The study establishes that the "Single-Prompt Ceiling" is a structural constraint of the paradigm, not just a model limitation. The most effective strategy was not adding more rules, but reordering existing logic to align with the model's attention mechanisms, proving that in formal reasoning, minimal, well-structured instructions outperform complex, information-dense ones.