💻 computer science

Does More Deliberation Help? A Comparative Evaluation of Chain-of-Thought Prompting Strategies for Grade-School Math Reasoning in Open-Source Instruction-Tuned Language Models

This paper empirically demonstrates that for small (3.8–8B) open-weight instruction-tuned models solving grade-school math problems, few-shot Chain-of-Thought prompting significantly outperforms zero-shot baselines, while more complex deliberation strategies like self-consistency and tree-of-thought fail to provide statistically reliable accuracy improvements over simpler methods.

Original authors: Arun Kumar

Published 2026-10-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Arun Kumar

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, researchers have long sought a way to make computer programs think more like humans, particularly when solving problems that require a sequence of logical steps. For years, the standard approach involved feeding these systems a direct question and waiting for an answer, but this often led to errors in arithmetic and logic. A breakthrough occurred when scientists realized that if they asked the models to write out their thinking process first—showing their work before stating a conclusion—their accuracy improved dramatically. This technique, known as chain-of-thought prompting, has become a cornerstone for getting machines to reason. However, as these tools have moved from massive, expensive supercomputers to smaller, more affordable versions that individuals and businesses can run on their own devices, a critical question has emerged: do the most complex and resource-heavy reasoning tricks still work, or do they simply add unnecessary cost without improving the result?

A recent study set out to answer this question by putting four different reasoning strategies to the test on three popular, smaller-scale language models. The researchers focused on a benchmark of one hundred grade-school math word problems, a set of puzzles designed to be simple for people but tricky for computers that skip steps. They wanted to see if the models performed better when simply told to think step-by-step, when shown a few examples of how to solve similar problems, when asked to generate multiple possible answers and vote on the best one, or when forced to explore a branching tree of different ideas before settling on a solution. The goal was to determine which method offered the best balance of accuracy and efficiency for the types of models that are becoming common in everyday applications.

The investigation revealed a clear and somewhat surprising hierarchy of effectiveness. The most powerful tool turned out to be the simplest form of showing examples. When the models were given a few worked-out problems to study before attempting the test questions, their accuracy jumped significantly compared to when they were just given a prompt to think carefully. This improvement was consistent across all three models tested, with success rates rising by as much as forty percentage points in some cases. The data showed that seeing a concrete example of how to break down a problem was far more valuable than any amount of abstract instruction or internal deliberation.

In contrast, the more elaborate strategies that involve generating multiple paths of reasoning did not provide the expected benefits. One approach, which asks the model to generate three different solutions and pick the most common answer, failed to beat the single-example method. While this technique is known to help larger, more powerful models, the study found that for these smaller systems, the extra effort of generating multiple answers did not translate into a statistically reliable improvement in correctness. The researchers noted that the models were not making enough diverse errors for a vote to correct them, meaning the extra computing power spent on this method was largely wasted.

Even more striking was the performance of the tree-of-thought strategy, a complex method where the model is encouraged to explore different branches of thought, evaluate them, and backtrack if a path leads to a dead end. This technique had previously shown great promise for difficult puzzles that require looking ahead or trying multiple possibilities. However, on these straightforward arithmetic problems, it performed no better than the basic method of simply asking the model to think step-by-step without any examples. In two of the three models tested, this complex search method actually performed worse than the simple example-based approach. The study suggests that for short, linear math problems, the overhead of exploring multiple branches is a burden that the models cannot justify, as the problems do not require the kind of deep exploration that this strategy provides.

The findings offer a practical guide for those using these technologies. The research indicates that for small, open-source models tackling standard arithmetic, the most effective path is not to add more complex reasoning machinery or to force the model to think in multiple directions. Instead, the most reliable way to boost performance is to invest time in crafting a few high-quality examples that demonstrate the correct way to solve a problem. While the more sophisticated methods might seem like a way to get smarter results, the data suggests they add significant cost and time without delivering a corresponding gain in accuracy for this specific type of task. For practitioners looking to deploy these models efficiently, the study concludes that a well-constructed example is worth far more than a complex search algorithm.

Technical Summary: Does More Deliberation Help?

Problem Statement

While Chain-of-Thought (CoT) prompting and its refinements (zero-shot CoT, few-shot CoT, self-consistency, and tree-of-thought) have demonstrated significant reasoning improvements on frontier, closed-weight large language models (LLMs), there is a lack of controlled empirical evidence regarding their efficacy on smaller (3.8–8B parameter), open-weight instruction-tuned models. Practitioners increasingly deploy these smaller models for cost, latency, and privacy reasons, yet it remains unclear whether the complex, search-based deliberation strategies (like self-consistency and tree-of-thought) retain their advantages over simpler prompting methods in this specific model class. This paper addresses the gap by determining if the "more deliberation" paradigm translates to statistically significant accuracy gains for small open-weight models on grade-school arithmetic reasoning.

Methodology

The study conducted a controlled, head-to-head comparison of four prompting strategies across three widely used open-weight instruction-tuned models: Mistral-7B-Instruct-v0.2, Phi-3-mini-4k-instruct, and Meta-Llama-3-8B-Instruct.

Dataset: A fixed, randomly drawn subset of 100 questions from the GSM8K benchmark (grade-school arithmetic word problems). The same 100 items were used for all model-strategy combinations to ensure paired outcomes.
Prompting Strategies:
1. Zero-shot CoT: A single greedy completion triggered by "Let's think step by step" without examples.
2. Few-shot CoT: A single greedy completion preceded by a fixed set of manually written exemplars with step-by-step solutions.
3. Self-Consistency: Multiple ( $k=3$ ) independent reasoning chains sampled at non-zero temperature using the few-shot prompt, with the final answer determined by majority vote.
4. Tree-of-Thought (ToT): The problem is decomposed into intermediate steps, generating $k=3$ candidate reasoning branches that are aggregated via majority vote, mirroring a search structure.
Evaluation Protocol:
- Scoring: Exact numeric match against the gold GSM8K answer.
- Statistical Analysis: Accuracy was reported with 95% confidence intervals (Wald approximation). Pairwise differences were tested using McNemar's exact test on the 2x2 tables of discordant outcomes for each model, appropriate for comparing strategies on the identical set of questions.
- Significance Threshold: $\alpha = 0.05$ (uncorrected for multiple comparisons, though conservative Bonferroni corrections were noted in limitations).

Key Contributions

Systematic Comparison: Provides the first controlled, statistically tested comparison of zero-shot CoT, few-shot CoT, self-consistency, and ToT specifically on small (3.8–8B) open-weight instruction-tuned models.
Statistical Rigor: Moves beyond point-estimate accuracy by computing 95% confidence intervals and applying McNemar's exact test to validate pairwise differences, addressing the limitations of previous studies that often relied on unpaired tests or single-model evaluations.
Cost-Accuracy Trade-off Analysis: Explicitly evaluates whether the increased inference cost of stochastic strategies (self-consistency and ToT) yields statistically reliable accuracy benefits over simpler baselines in the context of short, single-path arithmetic problems.

Results

The study evaluated 1,200 total model outputs (3 models $\times$ 4 strategies $\times$ 100 questions).

Few-Shot vs. Zero-Shot: Few-shot CoT significantly outperformed zero-shot CoT across all three models, with accuracy gains ranging from 12 to 39 percentage points (all $p < 0.05$ ). For example, Phi-3-mini improved from 46.0% (zero-shot) to 85.0% (few-shot).
Self-Consistency vs. Few-Shot: Self-consistency with $k=3$ sampled paths did not provide a statistically significant improvement over single-sample few-shot CoT in any model (all $p \geq 0.83$ ). The accuracy gains were negligible (e.g., 85.0% vs. 86.0% for Phi-3-mini).
Tree-of-Thought Performance: ToT performed significantly worse than few-shot CoT in two of the three models (Phi-3-mini and Llama-3-8B, $p < 0.0001$ ) and was statistically indistinguishable from the much cheaper zero-shot baseline in all three models. ToT accuracies (47.0%–54.0%) were consistently closer to zero-shot CoT than to few-shot CoT.
Model Variability: Phi-3-mini-4k-instruct achieved the highest absolute accuracy (85.0–86.0% with few-shot/self-consistency), outperforming larger models like Llama-3-8B and Mistral-7B, suggesting that instruction tuning quality and data curation may outweigh parameter count in this size range.

Significance and Claims

The paper concludes that for short, single-path arithmetic reasoning tasks on small open-weight models:

Exemplars are Dominant: The presence of worked exemplars (few-shot CoT) is the primary driver of accuracy gains. The benefit of "thinking step by step" alone (zero-shot) is substantially lower than providing examples.
Deliberation Costs Outweigh Benefits: Search-based deliberation strategies (ToT) and low-sample self-consistency ( $k=3$ ) add significant inference cost (multiple generations per answer) without yielding a corresponding, statistically reliable accuracy benefit.
Task-Strategy Mismatch: The failure of ToT suggests that its structured search and backtracking mechanisms are ill-suited for linear arithmetic problems that do not inherently require exploring alternative solution branches.

The author asserts that for practitioners deploying 4–8B parameter models on straightforward reasoning tasks, investing effort in constructing high-quality few-shot exemplars offers a superior accuracy-per-unit-cost trade-off compared to implementing complex decoding-time machinery like self-consistency or tree-of-thought.

Limitations

The author acknowledges several constraints:

Sample Size: The use of 100 items results in wide confidence intervals and potentially low statistical power for detecting smaller effect sizes, particularly in the Mistral-7B model.
Sample Count: The self-consistency and ToT experiments used $k=3$ , which is lower than the 10–40 paths typically used in literature to achieve reported gains; thus, the null result for self-consistency is specific to this modest sampling budget.
Domain Specificity: The findings are limited to GSM8K arithmetic. ToT may still be advantageous in domains requiring genuine search or backtracking (e.g., planning, combinatorial puzzles).
Cost Accounting: The study reports accuracy but does not quantify token-level costs or latency, though it notes that stochastic strategies are inherently more expensive.