💻 computer science

Improving Symbolic Translation of Language Models for Logical Reasoning

This paper proposes a framework for enhancing the logical reasoning capabilities of smaller language models by categorizing translation errors, fine-tuning models with synthesized data, and introducing an incremental inference approach with predicate verification to improve the accuracy and reliability of natural language to first-order logic translation.

Original authors: Ramya Keerthy Thatikonda, Jiuzhou Han, Wray Buntine, Ehsan Shareghi

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ramya Keerthy Thatikonda, Jiuzhou Han, Wray Buntine, Ehsan Shareghi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly impatient, robot assistant (a small Language Model). You want to give it a complex logic puzzle written in plain English, like a riddle, and you want it to solve the puzzle using strict mathematical rules (First-Order Logic, or FOL).

The problem is that while the robot understands the story, it often trips over the rules of the math language. It might forget a parenthesis, use a word with the wrong number of "slots" (like saying "is happy" when it needs "is happy with someone"), or get stuck in a loop repeating the same phrase. When this happens, the external math engine (the "solver") that checks the answer crashes, and the whole system fails.

This paper is about teaching these smaller, more affordable robots to be much better at translating English into this strict math language without breaking the rules.

Here is how they did it, explained through three simple analogies:

1. The "Practice with a Master Chef" (Data Synthesis)

The Problem: Small robots don't have enough practice examples of perfect translations. They are like junior chefs who have never seen a perfectly plated dish.
The Solution: The researchers used a super-smart, expensive "Master Chef" (a large AI model) to cook up thousands of perfect examples. They took raw ingredients (English sentences), had the Master Chef turn them into perfect math recipes (FOL), and then carefully checked every single recipe to make sure there were no typos or missing ingredients.
The Result: They fed these high-quality, pre-checked recipes to the small robots to learn from. This is like giving the junior chefs a stack of perfect cookbooks to study before they try to cook on their own.

2. The "Two-Step Dance" (Incremental Inference)

The Problem: When asked to translate a whole story into math all at once, the small robot gets overwhelmed. It tries to guess the next word while also remembering the whole story, and it often gets stuck in a "loop," repeating the same word over and over (like a broken record).
The Solution: Instead of asking the robot to do everything in one giant leap, the researchers broke the task into two distinct steps:

Step 1: "First, just list the main characters and concepts (the predicates)."
Step 2: "Now, using that list, write the full math sentences."
The Analogy: Think of it like building a house. Instead of asking a builder to "build the whole house," you first ask them to "draw the blueprint and list the materials." Once that list is locked in, you ask them to "build the house using exactly those materials." This stops the builder from forgetting what they were doing halfway through or inventing new materials on the fly.

3. The "Spellchecker for Logic" (Verification Module)

The Problem: Even with the two-step process, the robot might still make a specific type of mistake: using a word with the wrong number of "slots." For example, using the word "Parent" to mean "Parent of X" (one slot) in one sentence, and "Parent of X and Y" (two slots) in another. The math engine hates this inconsistency.
The Solution: They added a tiny, lightweight "Spellchecker" (a verifier) that sits between Step 1 and Step 2. Before the robot writes the final math sentences, this Spellchecker looks at the list of concepts and says, "Hey, you used 'Parent' with two people here, but only one person there. Fix that."
The Result: This catches specific errors that the main robot keeps missing, making the final translation much more reliable.

The Bottom Line

The researchers tested these tricks on four different small robots and four different logic puzzles. They found that:

Fine-tuning (studying the Master Chef's recipes) made the robots much better at following instructions.
Incremental Inference (the two-step dance) stopped the robots from getting stuck in loops and made their output much cleaner.
The Verifier (the Spellchecker) fixed the remaining "slot" errors.

By combining these three methods, they turned small, accessible, and cheap AI models into reliable translators that can handle logical reasoning tasks almost as well as the much larger, more expensive models. They didn't just make the robots faster; they made them more accurate and less likely to crash the system.

Technical Summary: Improving Symbolic Translation of Language Models for Logical Reasoning

Problem Statement

The integration of Large Language Models (LMs) with external symbolic solvers (e.g., Z3, Prover9) offers a verifiable path for deductive logical reasoning. This paradigm requires translating Natural Language (NL) into First-Order Logic (FOL). While effective for large models, smaller, more accessible LMs struggle significantly with this translation task. They frequently produce incorrect symbolic outputs due to formatting errors, syntax violations, and semantic inconsistencies. Existing approaches often rely on self-correction loops driven by solver feedback, but these methods depend heavily on the underlying model's capability to interpret error messages and regenerate accurate code, a strength typically reserved for larger, more expensive models. Furthermore, there is a scarcity of high-quality symbolic data to fine-tune smaller models effectively.

Methodology

The authors propose a three-pronged approach to enhance the performance of smaller LMs (3B–4B parameters) in NL-to-FOL translation:

1. Data Synthesis and Supervised Fine-Tuning (SFT)

To address the lack of symbolic training data, the authors synthesize a dataset using a large language model (GPT-4o) on the ProofWriter dataset. This process involves:

Generation: Creating NL-to-FOL pairs.
Filtering: A multi-stage pipeline including format correction, syntax validation using Prover9, and semantic filtering to ensure high-quality translations.
Training: The synthesized data is combined with existing symbolic datasets (FOLIO) to perform Supervised Fine-Tuning (SFT) on four small models: Gemma-3-4B, Phi-4-mini, Qwen2.5-3B, and Qwen3-4B. The training input concatenates the NL statement, the generated predicate set, and the FOL representation to enforce structured generation patterns.

2. Incremental Inference

The paper identifies that standard "next-token" inference often leads to error propagation, where a single token-level mistake (e.g., a formatting error) causes the model to enter repetitive loops or generate malformed outputs. To mitigate this, the authors introduce Incremental Inference, a two-stage process:

Stage 1 (Predicate Generation): The model first outputs a consistent set of predicates ( $\hat{P}$ ) required for the entire problem context.
Stage 2 (FOL Translation): Using the augmented input (NL statements + $\hat{P}$ ), the model generates the full FOL statements.
This decomposition enforces predicate consistency across statements and allows for early stopping, significantly reducing formatting errors.

3. Verification Module

Leveraging the separation of predicate generation from full statement generation, the authors integrate a lightweight Verifier. This module, implemented as a small LM with few-shot examples, checks the generated predicate set for arity errors (e.g., using a predicate with inconsistent argument counts) before the FOL translation stage. If errors are detected, the verifier outputs a corrected predicate set; otherwise, it passes the original set.

Key Contributions

Error Taxonomy: The authors categorize common NL-to-FOL translation errors into four types: Parsing (missing quantifiers, parenthesis errors), NoneType (quantifier location, missing variables), Token (special tokens, unknown operators), and Predicate (arity mismatch, subject-predicate conflicts).
Data Synthesis Pipeline: A robust framework for generating high-quality, verified symbolic data to train smaller models, bridging the gap caused by the scarcity of such data.
Incremental Inference Framework: A novel inference strategy that decouples predicate generation from statement translation, improving output stability and format adherence without requiring additional instructions at inference time.
Predicate-Level Metrics: Introduction of Coverage (fraction of used predicates present in the initial set) and Usage (fraction of generated predicates actually used) to evaluate semantic consistency, alongside standard execution rates.

Results

The study evaluates the proposed methods across four logical reasoning datasets: ProofWriter, FOLIO, ProntoQA, and ProverQA.

Performance Gains: Supervised Fine-Tuning (SFT) significantly improves both execution rates (the percentage of valid FOL statements) and reasoning accuracy compared to In-Context Learning (ICL) baselines across all model families.
Error Reduction: The combination of SFT and Incremental Inference leads to substantial reductions in formatting and syntax errors. For instance, on the FOLIO dataset, the Qwen3-4B model showed a 57.4% drop in total errors.
Predicate Fidelity: Incremental inference substantially improves predicate coverage, particularly on Out-of-Distribution (OOD) datasets like ProntoQA and ProverQA, by ensuring the model explicitly defines the predicate set before generating logic.
Verification Impact: Adding the arity verification module further improves reasoning accuracy, particularly for models struggling with abstract concepts (e.g., ProntoQA), though the gains are modest compared to the improvements from SFT and incremental inference.
Model Sensitivity: The results indicate that performance is highly sensitive to the specific model architecture and the nature of the dataset. For example, Qwen3-4B showed significant gains on OOD datasets, while Phi-4-mini struggled with context retention in long premise sets under standard inference.

Significance and Claims

The paper claims to move the field closer to developing reliable and accessible symbolic-reasoning systems. By demonstrating that smaller, cost-effective models can achieve high-fidelity logical reasoning through structured data synthesis and inference decomposition, the authors argue that the barrier to deploying neuro-symbolic systems in resource-constrained settings is lowered.

The authors maintain a modest tone regarding limitations:

The synthesized data, while curated, may contain minor inconsistencies.
The framework addresses syntactic and formatting robustness but does not fully guarantee semantic correctness (i.e., the FOL may be syntactically valid but still misrepresent the intended NL meaning).
The verification module introduces additional inference latency, though the incremental inference process itself does not increase latency beyond the token count.
Direct behavioral comparisons between models are difficult due to the opacity of pre-training data.

Ultimately, the work suggests that systematic error categorization, targeted fine-tuning, and structured inference are critical for enabling smaller LMs to function effectively as components in verifiable logical reasoning pipelines.