Improving Chain-of-Thought Reasoning via Quasi-Symbolic Abstractions

1. Problem Statement

Large Language Models (LLMs) have demonstrated significant capabilities in reasoning through Chain-of-Thought (CoT) prompting, which decomposes complex tasks into intermediate steps. However, standard CoT approaches suffer from two critical limitations:

• Content Bias and Lack of Robustness: Explanations generated via natural language CoT are susceptible to content biases, leading to unfaithful reasoning and poor performance when faced with adversarial variations (e.g., shuffled answer options or superficial lexical changes).
• The Formalization Bottleneck: Recent attempts to improve robustness involve coupling LLMs with external symbolic solvers or fully symbolic formalisms. While effective, these methods require a complete translation from natural language to formal logic. This process is often inefficient, inflexible, and creates a bottleneck that hinders the model's ability to handle diverse, real-world tasks.

The core challenge is to achieve a trade-off: how to disentangle content from logical reasoning to improve robustness without the inefficiency of full symbolic formalization.

2. Methodology: QuaSAR

The authors propose QuaSAR (Quasi-Symbolic Abstract Reasoning), a framework that guides LLMs to operate at a higher level of abstraction by mixing natural language with symbolic elements. Unlike fully symbolic approaches, QuaSAR does not require a complete translation to a formal language; instead, it uses a quasi-symbolic approach where the LLM formalizes only the relevant variables and predicates.

The framework structures the reasoning process into a four-step quadruple $(Q, S, R, A)$:

1. Abstraction ($s_1$): The model analyzes the problem and explicitly identifies relevant predicates, variables, and constants (both numerical and verbal). This step isolates the core logical components from the narrative text.
2. Formalisation ($s_2$): The model reformulates the problem using a semi-structured symbolic representation. It translates natural language facts into a mix of symbols and natural language (e.g., defining $d_{total}$, $v_{initial}$, and constraints). This reduces ambiguity and content effects while retaining necessary context.
3. Explanation ($s_3$): The model generates a step-by-step reasoning chain based on the symbolic structure defined in step 2. This ensures that logical connections are explicit and less prone to hallucination or reliance on implicit world knowledge.
4. Answering ($s_4$): The model generates the final solution in a strict, concise format (e.g., "The answer is: [number]"), facilitating automated evaluation.

Application Modes:

• In-Context Learning (ICL): QuaSAR is used as a prompting strategy for larger, capable models (e.g., GPT-4o, Llama-3-70B) to guide their reasoning directly.
• Synthetic Demonstrations (SFT): High-performing models generate QuaSAR-style reasoning traces, which are then filtered for correctness and used as training data (demonstrations) to fine-tune smaller models (e.g., Llama-3-1B, Qwen2-1.5B).

3. Key Contributions

• QuaSAR Framework: The first method to apply quasi-symbolic abstractions across a broad spectrum of reasoning tasks, enabling the coexistence of symbolic abstractions and natural language explanations.
• Disentanglement of Content and Logic: The method successfully separates logical reasoning from specific world knowledge, mitigating content biases that typically plague CoT.
• Dual-Mode Applicability: Demonstrates effectiveness both as a direct prompting strategy for large models and as a data generation strategy to improve smaller models via supervised fine-tuning (SFT).
• Robustness Verification: Provides empirical evidence that quasi-symbolic reasoning significantly improves model resilience against adversarial perturbations (e.g., shuffled choices, parameter variations).

4. Experimental Results

The authors evaluated QuaSAR on diverse benchmarks including mathematical reasoning (GSM8K, SVAMP, AQuA, OlyBench), natural language understanding (MMLU-Redux, GPQA, DROP), and adversarial variations (GSM-Symbolic).

Key Findings:

• Performance Gains:
  • On GPT-4o, QuaSAR achieved an average accuracy increase of 8% over standard CoT and outperformed Faithful CoT and CoMAT.
  • On Llama-3-70B, it showed an 11.8% improvement over the baseline.
  • For smaller models (e.g., Llama-3-1B, Qwen2-1.5B) fine-tuned with QuaSAR demonstrations, performance gains were substantial (up to 13.4% over baseline), often surpassing models trained with standard CoT demonstrations.
• Robustness:
  • In MMLU-Redux (with shuffled answer choices), QuaSAR maintained stable performance (0.0% drop), whereas CoT dropped by 1.2–1.4%.
  • In GSM-Symbolic (adversarial math variations), QuaSAR showed significantly lower sensitivity to superficial changes compared to CoT.
• Efficiency: Unlike methods requiring external solvers, QuaSAR operates via a single prompting step, reducing computational costs and latency.
• Ablation Studies: Removing the Formalisation ($s_2$) or Explanation ($s_3$) steps caused the most significant performance drops (over 3.5 points), confirming that the semi-structured symbolic representation is the core driver of success.
• Self-Correction: Models trained with QuaSAR demonstrated better self-correction capabilities. Because the reasoning is structured, errors are easier to detect and fix compared to unstructured CoT outputs.

5. Significance

This paper addresses a critical gap in LLM reasoning: the trade-off between the flexibility of natural language and the rigor of symbolic logic.

• Practical Impact: QuaSAR offers a scalable, cost-effective solution that does not rely on external tools or complex pipelines, making it accessible for various model sizes.
• Theoretical Insight: It validates the hypothesis that "quasi-symbolic" abstractions—where models explicitly define variables and predicates without full formalization—are sufficient to disentangle reasoning from content bias.
• Future Directions: The framework lays the groundwork for improving multilingual reasoning and integrating with Retrieval-Augmented Generation (RAG) systems to resolve knowledge conflicts, suggesting a path toward more reliable and verifiable AI reasoning.

In summary, QuaSAR represents a significant step forward in making LLM reasoning more faithful, robust, and verifiable by leveraging the structural benefits of symbolic logic while maintaining the flexibility of natural language.