Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning

1. Problem Statement

Large Language Models (LLMs) often struggle with reliable reasoning. Self-Consistency (SC) is a popular technique to improve accuracy by generating multiple reasoning paths (samples) for a single problem and selecting the most frequent answer (majority voting). However, standard SC suffers from a critical bottleneck:

• High Computational Cost: Achieving optimal accuracy typically requires generating a large number of samples (e.g., 40), leading to significant inference latency and cost.
• Inefficient Diversity: Existing methods rely on high temperature sampling to induce diversity. In practice, this often yields samples with superficial syntactic variations but identical semantic reasoning paths, failing to provide robust consensus signals.
• Limitations of Early Stopping: Previous adaptive consistency methods (which stop sampling early if a majority emerges) still require many samples (minimum 4–5) and often yield lower accuracy than full sampling.

The core challenge is to improve both the accuracy and efficiency of self-consistency, specifically aiming to solve complex reasoning tasks with a drastically reduced sample budget.

2. Methodology

The authors propose a Cross-Modal Ensembling approach that combines two distinct reasoning modalities: Chain-of-Thought (CoT) and Program-of-Thought (PoT).

Core Concept: Complementary Strengths

• CoT: Generates step-by-step natural language inferences. It is flexible but prone to arithmetic and calculation errors.
• PoT: Generates executable code (Python) to solve the problem. It is robust in computation but prone to symbolic formulation errors (e.g., incorrect logic in code structure).
• Hypothesis: If a CoT solution and a PoT solution agree on the final answer, the likelihood of correctness is extremely high because the two modalities have low-correlated error modes. An agreement implies alignment between logical framing (CoT) and computational execution (PoT).

Framework Components

A. Full Sampling Strategies
Instead of sampling 40 CoT paths, the authors sample an equal mix of CoT and PoT (e.g., 20 CoT + 20 PoT). They propose three aggregation strategies:

1. CPMaj (Majority Voting): Simple majority vote across all CoT and PoT answers combined.
2. CPMax (Max Confidence): Selects the most frequent answer from the modality that has the highest internal frequency.
3. CPAgr (Agreement Prioritization): Prioritizes answers that appear in both modalities, using total frequency as a tie-breaker.

B. Early-Stopping Strategies (Bayesian Model)
The paper introduces a general Bayesian hypothesis testing model to determine when to stop sampling.

• Mechanism: The model alternates sampling one CoT and one PoT answer. It tracks the posterior probability that an answer is "safe" (either correct or the full sampling would also be wrong) given an agreement between modalities.
• Key Observation: Empirically, the probability of safety given cross-modal agreement ($a_2$) is extremely high ($\approx 0.99$).
• Strategies:
  • Data-Driven: Learns parameters ($c, a_1, a_2$) from held-out data to define stopping thresholds (e.g., CPDAA: stop when any CoT/PoT pair agrees; CPDFA: stop when the first pair agrees).
  • Data-Independent (Heuristic): Assumes $a_2 = 1$ (agreement implies safety).
    • CPAA: Stop as soon as any CoT and PoT answer agree.
    • CPFA: Stop as soon as the first CoT and PoT answers agree.
    • CPFF: Stop only if the initial (temperature-0) CoT and PoT answers agree.
• Adaptive Consistency Integration: The method also incorporates standard adaptive consistency (checking for a dominant majority within a single modality) as a parallel signal to handle cases where one modality dominates.

3. Key Contributions

1. Novel Ensembling Paradigm: First to combine CoT and PoT within the self-consistency framework, leveraging their complementary error modes to boost accuracy and efficiency.
2. Two-Sample Breakthrough: Demonstrates that 78.6% of tasks can be solved with only two samples (one CoT + one PoT) based on cross-modal agreement. This is a significant leap, as prior methods required a minimum of 4–5 samples.
3. Efficiency Gains: Reduces the number of samples required for Self-Consistency by a factor of 9.3× compared to full sampling, and outperforms the best prior early-stopping methods (which achieved ~5.6× reduction).
4. General Bayesian Framework: Formalizes cross-modal agreement as a Bayesian stopping rule, providing both data-driven and data-independent strategies adaptable to different model capabilities.
5. Bootstrapping for Small Models: Shows how to improve PoT capabilities in smaller models (e.g., Llama 3B) by bootstrapping PoT rationales from CoT outputs, enabling cross-modal ensembling even for models with weak native coding skills.

4. Experimental Results

The authors evaluated their method across 5 benchmarks (GSM8K, MATH, FinQA, SVAMP, TabMWP) and 5 LLMs (GPT-3.5, GPT-4o, Mistral-Large, Qwen3-Coder, DeepSeek-R1).

• Accuracy:

  • CoT-PoT full sampling outperformed standard CoT-only and PoT-only self-consistency by an average of 1.1%.
  • The best early-stopping method (CPAA) achieved higher accuracy than full sampling baselines while using far fewer samples.
  • DeepSeek-R1 (a reasoning-intensive model) showed the most significant benefits from cross-modal consistency.

• Efficiency:

  • Sample Reduction: The CPAA method reduced the average sample count from 40 to 4.3 (a 9.3× reduction).
  • Two-Sample Success: On average, 78.6% of problems were resolved with just 2 samples. For some benchmarks/models, this exceeded 90%.
  • Comparison: Prior state-of-the-art early-stopping methods (ASC, ESC) required a minimum of 4–5 samples and achieved lower accuracy.

• Ablation Studies:

  • Cross-modal agreement was proven to be a stronger stopping signal than within-modality agreement (e.g., two CoT samples agreeing).
  • Data-independent methods (CPAA) were highly efficient, while data-driven methods (CPDFA) offered better robustness for weaker models or specific domains (e.g., FinQA).

5. Significance and Limitations

Significance:

• Cost-Effective Inference: This method makes high-accuracy reasoning feasible for production environments by drastically reducing the compute cost (tokens/inference calls) required for LLM reasoning.
• Scalability: It enables the use of Self-Consistency with larger models where generating 40 samples is prohibitively expensive.
• Theoretical Insight: It challenges the notion that "more samples" are always better, showing that "diverse modalities" are more valuable than "diverse samples" of the same modality.

Limitations:

• Task Applicability: The method is most effective for tasks amenable to programmatic reasoning (math, logic, structured data). For subjective or long-form QA, the PoT signal may be weak.
• Bootstrapping Dependency: For very small models lacking PoT capabilities, the method requires a bootstrapping phase (generating PoT from CoT), which may not work if the base model has zero prior exposure to code.
• Shared Conceptual Errors: In rare failure cases, both modalities may agree on an incorrect answer if they share a fundamental conceptual misunderstanding of the problem (e.g., misinterpreting a temporal reference), though this is rare compared to stochastic errors.

In conclusion, this paper presents a paradigm shift in LLM inference, demonstrating that cross-modal consistency between natural language and code is a powerful, efficient signal for reasoning, enabling high-accuracy solutions with minimal sampling.