Technical Summary: From Reasoning to Code: GRPO Optimization for Underrepresented Languages

Problem Statement

Generating accurate and executable code in underrepresented programming languages (e.g., Prolog, Lisp) remains a significant challenge for Large Language Models (LLMs). While models excel in high-resource languages like Python, they consistently fail in logic programming due to the scarcity of public training data. This data scarcity leads to specific failure modes: hallucination of non-existent predicates, the imposition of imperative control flow on declarative paradigms, and mishandling of unification and backtracking. Standard Supervised Fine-Tuning (SFT) is insufficient because it relies on scarce human-written examples and does not directly optimize for execution correctness.

Methodology

The paper proposes a Reinforcement Learning with Verifiable Rewards (RLVR) framework that utilizes execution-driven feedback to train small-scale LLMs (Qwen2.5-Coder family, 0.5B to 7B parameters) without relying on extensive human-curated datasets.

Core Framework

• Algorithm: The authors employ Group Relative Policy Optimization (GRPO), a policy-gradient algorithm that eliminates the need for a separate value-function network (unlike PPO). GRPO estimates the advantage of an output relative to a group of peer outputs sampled from the same prompt, reducing computational overhead.
• Training Loop: The system integrates a Prolog interpreter (via PySwip) directly into the training loop. The LLM generates code in response to natural language prompts (specifically from the GSM8K dataset), and the interpreter executes the code to provide a ground-truth correctness signal.
• Reward System: A multi-objective reward function aggregates four signals:
  1. Format Rewards: Penalties and bonuses for adhering to a strict XML template (<reasoning>, <code>, <query>), ensuring the code is extractable. A specific penalty is applied if the query is incorrectly nested inside the code block.
  2. Correctness Rewards: Based on execution results:
     • +1.0: Exact match with the ground truth.
     • -1.0: Logical error (executes but wrong result).
     • -0.5: Syntax error (fails to compile/execute).
     • -0.1: Timeout or no output.

Key Technical Interventions

The authors identified a failure mode termed "reasoning contraction," where standard GRPO training with a Kullback–Leibler (KL) divergence penalty caused the 7B model to collapse into short, memorized output patterns, degrading accuracy over time. To address this, they introduced two critical modifications:

1. Removal of KL Divergence: Eliminating the KL penalty allowed the policy to shift further from the supervised baseline, enabling the model to internalize the distinct logic programming paradigm.
2. Length-Aware Reward: A specific reward mechanism was added to encourage reasoning chains within a target token-length range, preventing the model from truncating its reasoning process.

Key Contributions

1. New State-of-the-Art for Prolog: Demonstrated that RLVR with execution-based feedback can compensate for extreme data scarcity, establishing a new benchmark for Prolog code generation on GSM8K using a 7B-parameter model.
2. Systematic Ablation of GRPO Dynamics: Identified "reasoning contraction" as a specific failure mode of KL-regularized RL in logic programming and characterized the conditions under which relaxing this constraint is beneficial.
3. Generalization Beyond Arithmetic: Showed that improvements transfer to 20 diverse Prolog tasks (Rosetta Code) and the GSM-Symbolic benchmark, where zero-shot accuracy more than doubled.
4. Cross-Language Validity: Validated the approach on Lisp, demonstrating that interpreter-driven RL is effective across distinct families of underrepresented languages.

Results

Experiments were conducted on the GSM8K dataset, with evaluations on Rosetta Code and GSM-Symbolic (P1 and P2 variants).

• GSM8K Performance:
  • The optimized Qwen2.5-Coder-7B (Zero-shot, no KL, length constraint) achieved a pass@4 of 0.882.
  • In One-shot settings, the same configuration reached 0.893, outperforming PROPER (Mistral-7B, 0.72), Llama 3.3 70B (0.839), and Qwen2.5-Coder-32B (0.873).
  • Without the KL removal, the 7B model suffered from performance drift, dropping from a baseline of 0.814 to 0.672.
• GSM-Symbolic:
  • On the challenging P2 dataset (requiring deeper symbolic manipulation), the zero-shot pass@4 for the 7B model increased from a baseline of 0.298 to 0.673.
• Lisp Performance:
  • The approach improved zero-shot pass@4 on Lisp from 0.572 to 0.876. However, the removal of the KL constraint yielded more modest gains in Lisp compared to Prolog, suggesting that "un-regularized" training is most critical for highly specialized, low-resource languages.
• Rosetta Code:
  • The 7B model nearly doubled its zero-shot baseline, reaching a pass@4 of 0.46.

Significance and Claims

The paper argues that for formal languages with deterministic semantics, the interpreter itself can serve as the primary training signal, replacing the scarce human data that current approaches depend upon. By treating the symbol system not merely as a tool but as a "teacher" that supervises the neural network via execution traces, the authors provide a rigorous, automated supervisor.

The work claims to offer a robust, adaptable path for enhancing code generation in underrepresented languages. It suggests that bridging the gap between neural intuition and symbolic rigor does not require massive scale or massive datasets, but rather a shift in the learning objective toward maximizing execution correctness via reinforcement learning. The authors conclude that this methodology enables models to respect the syntax and logic of target languages, moving closer to the "Holy Grail" of programming where a user states a problem and the computer solves it via verifiable, logical code.