enhancing reasoning accuracy in large language models during inference time

1. Problem Statement

Large Language Models (LLMs) demonstrate strong linguistic fluency but often fail at tasks requiring reliable multi-step reasoning, logical consistency, or structured problem-solving. While prior research has addressed this through architectural changes, specialized training, or reinforcement learning, these methods require access to model weights, extensive training data, or significant computational resources.

The paper addresses the need for inference-time techniques that can improve reasoning accuracy without modifying model parameters or requiring additional training. The goal is to identify strategies that can be deployed immediately to boost reliability across different risk domains.

2. Methodology

The authors conducted a systematic empirical study evaluating three distinct inference-time strategies on the Logical Reasoning Improvement Dataset (curated by garage-bAInd/Open-Platypus). This dataset contains deterministic problems from diverse domains (Science, Math, Coding) with unambiguous ground-truth answers.

All strategies utilized Chain-of-Thought (CoT) prompting to elicit explicit intermediate reasoning steps. The three evaluated approaches were:

A. Self-Consistency with Controlled Stochastic Decoding

• Mechanism: Instead of a single deterministic pass, the model generates $n=6$ independent reasoning paths.
• Decoding Strategy: Uses controlled temperature ($T=0.8$) and nucleus sampling ($top\text{-}p=0.9$) to encourage diverse yet semantically plausible trajectories.
• Aggregation: A low-temperature model ($T=0.1$) extracts the final answer from each trace. A majority voting mechanism (implemented as an LLM-based judge) selects the most semantically consistent answer, handling lexical variations (e.g., "2" vs. "two").
• Hypothesis: Correct reasoning paths are more likely to recur across stochastic samples than incorrect ones.

B. Dual-Model Reasoning with Cross-Model Verification

• Mechanism: Two independent LLMs (different architectures/training data) solve the same problem.
• Verification: An independent verifier model assesses whether both outputs reach the same correct conclusion. It ignores stylistic differences and focuses solely on the logical conclusion.
• Decision Logic: Agreement triggers an ACCEPT verdict; disagreement triggers a REJECT.
• Goal: To use cross-model consensus as a proxy for correctness, prioritizing precision over recall.

C. Self-Reflection via Iterative Critique

• Mechanism: A single model generates an initial solution, then critiques its own reasoning (identifying flaws, missing steps, or incorrect assumptions), and finally revises the solution.
• Process: Initial Generation $\rightarrow$ Reflection Prompt $\rightarrow$ Revision $\rightarrow$ Final Answer.
• Goal: To simulate an iterative refinement process to correct errors without external models.

3. Key Results

The experiments yielded distinct performance profiles for each strategy:

• Self-Consistency (Stochastic):

  • Performance: Achieved the highest accuracy, with 64.9% of outputs verified as correct.
  • Comparison: Significantly outperformed self-consistency with greedy decoding (56.2% accuracy).
  • Insight: Controlled stochasticity is critical; greedy decoding limits the diversity of reasoning paths needed for self-consistency to work effectively.

• Dual-Model Reasoning:

  • Performance: The raw single-model accuracy was 48.7%. After cross-model verification, the acceptance rate dropped slightly to 47.4%.
  • Insight: While this did not increase the number of correct answers, it acted as a highly effective filter. It successfully rejected incorrect outputs that lacked consensus, ensuring that only high-confidence answers were accepted. This makes it a robust mechanism for estimating reliability in the absence of ground truth.

• Self-Reflection:

  • Performance: Showed only marginal improvement, increasing accuracy from 47.2% (baseline) to 50.6%.
  • Insight: For smaller, non-specialized reasoning models, iterative self-critique offers limited benefit, suggesting that effective self-correction may require larger or more specialized architectures.

4. Key Contributions

1. Systematic Comparative Evaluation: Provides a controlled comparison of three major inference-time strategies under identical prompting and verification settings.
2. Decoding Strategy Optimization: Demonstrates that Self-Consistency is heavily dependent on the decoding strategy; combining it with controlled temperature and nucleus sampling is a necessary component for substantial gains, whereas greedy decoding yields diminishing returns.
3. Risk-Based Strategy Framework: Proposes a practical framework for selecting inference strategies based on domain risk:
   • Low-Risk/Efficiency: Self-Consistency (Stochastic) for education, customer support, and general retrieval.
   • High-Risk/Safety: Dual-Model Reasoning for finance, healthcare, and legal sectors where precision is paramount.
   • Limited Utility: Self-Reflection is currently less effective for smaller models.

5. Significance

This work is significant because it offers a parameter-free, resource-efficient pathway to enhance LLM reasoning. It proves that organizations can significantly improve model reliability without the prohibitive costs of retraining or fine-tuning.

• Practical Deployment: It provides actionable guidelines for deploying LLMs in production. For example, in high-stakes environments, the "Dual-Model" approach acts as a safety gate, reducing the risk of hallucinations or logical errors.
• Cost-Benefit Analysis: It clarifies the trade-off between compute cost and reliability. Self-consistency offers the best balance for general use, while dual-model verification justifies higher latency for critical applications.
• Future Direction: The marginal gains from self-reflection on smaller models highlight a gap in current research, suggesting that future work should focus on scaling self-reflection mechanisms or integrating them with larger, more capable reasoning models.