Elementary Math Word Problem Generation using Large Language Models

1. Problem Statement

Mathematics is often perceived as a complex subject, leading to high failure rates. To improve skills, students require diverse practice problems. However, manually creating Math Word Problems (MWPs) is time-consuming for educators due to the need for natural language generation, grammar adherence, and curriculum alignment.

Existing automated solutions face significant limitations:

• Input Dependency: Early methods require users to provide a seed text, an equation, or specific context, which is not user-friendly.
• Lack of Control: Existing models struggle to generate problems that strictly adhere to specific grade levels (e.g., Grade 1 vs. Grade 6) and question types (e.g., single-digit addition vs. fractions).
• Quality Issues: Generated MWPs often contain grammatical errors, unsolvable logic, unrealistic scenarios, or incorrect units.
• Limited Diversity: Models tend to produce repetitive outputs.

2. Methodology: The MathWiz System

The authors propose MathWiz, an end-to-end MWP generation system based on Large Language Models (LLMs) that requires only minimal user input: the number of questions, the grade level, and the question type. The system development followed a multi-stage experimental pipeline:

A. Dataset Creation

• MathWizard Dataset: A new, manually curated dataset of 4,000 MWPs aligned with the US Common Core Mathematics Standards (Grades 1–6). Unlike existing datasets (e.g., MAWPS, SVAMP), this includes explicit metadata for grade and section (question type).
• Preference Dataset: A subset of 1,005 MWPs annotated with human preferences ("accepted" vs. "rejected") for specific grade/section combinations to support preference optimization.
• Error Annotation: A comprehensive corpus of 4,000 MWPs annotated with 12 distinct error categories (e.g., co-reference issues, unsolvability, unrealistic answers, grade mismatch).

B. System Architecture & Experiments

The research followed a sequential optimization process (illustrated in Figure 2 of the paper):

1. LLM Selection: Zero-shot prompting experiments compared open-source 7B parameter models (Llama-2, Mistral, Zephyr). Llama-2-7b-chat was selected as the base model due to its balance of quality and inference latency (Mistral produced poor quality; Zephyr was too slow).
2. Prompt Engineering: Three patterns were tested: Persona, Template, and Dialogue. The Persona pattern (instructing the LLM to act as a math teacher) yielded the best results.
3. Diversity Optimization: To prevent repetitive outputs, the authors tuned decoding hyper-parameters using Contrastive Search rather than standard Greedy Decoding. The optimal combination found was top_k=40, penalty_alpha=0.6, and no_repeat_ngram_size=5.
4. Fine-Tuning: The selected LLM was fine-tuned using QLoRA (Quantized Low-Rank Adaptation) on the MathWizard dataset.
5. Few-Shot Prompting: The fine-tuned model was tested with 1, 3, and 5 in-context examples. 5-shot prompting provided the best balance of guidance and reasoning capability.
6. Preference Optimization: Techniques including DPO, KTO, and CPO (Contrastive Preference Optimization) were applied using the human-preference dataset. CPO showed the most promise for specific error categories.
7. Solvability Verification: A secondary commercial LLM (Gemini) was integrated as a "solvability detector." It performs a binary classification on generated problems. If a problem is marked unsolvable, the system discards it and regenerates a new one (up to 2 iterations).

3. Key Contributions

• MathWiz System: A fully automated MWP generator that eliminates the need for seed equations or context, accepting only grade and question type as input.
• MathWizard Dataset: The first publicly released, comprehensively error-annotated English MWP corpus (4K samples) covering Grades 1–6 with curriculum-aligned metadata.
• Comprehensive Error Taxonomy: Defined and evaluated 12 error categories, expanding beyond the 3 categories used in previous works. This includes surface errors (grammar, spelling) and deep mathematical errors (unsolvability, unrealistic answers, grade mismatch).
• Hybrid Evaluation Framework: Combined human evaluation (with Inter-Rater Reliability testing) and LLM-based evaluation (using G-Eval on Copilot) to create a robust assessment pipeline.
• Performance Analysis: Demonstrated that while LLMs struggle with strict curriculum adherence, a combination of fine-tuning, few-shot prompting, and a solvability filter significantly improves output quality.

4. Results

The system was evaluated against a baseline of Llama-2 zero-shot prompting. Key findings include:

• Quality Improvement: The final pipeline (Fine-tuned Llama-2 + 5-shot + Solvability Filter) improved the Weighted Average Accuracy by 3.33% over the zero-shot baseline.
• Curriculum Adherence:
  • Grade Relevance: Improved by 26.93% (from 52.69% to ~79.62% before the solvability filter drop).
  • Section Relevance: Improved by 38.46% (from 42.69% to ~81.15%).
• Error Reduction: The solvability module significantly reduced "Unsolvable," "Unrealistic," and "Trivial" problem rates.
• Trade-offs:
  • Preference Optimization (CPO): Improved specific categories (e.g., co-reference, misspellings) but slightly degraded overall solvability.
  • Solvability Filter: While it removed unsolvable problems, it occasionally discarded valid problems (False Positives), leading to a slight dip in overall weighted average accuracy but a net gain in reliability.
• Evaluation Correlation: Human evaluators showed high agreement (AC1 > 0.90). LLM judges (Copilot) correlated well with humans on surface errors (grammar, safety) but struggled with complex mathematical reasoning (unsolvability, grade relevance).

5. Significance and Future Work

• Educational Impact: MathWiz provides a scalable solution for generating high-quality, curriculum-aligned practice materials, reducing the burden on teachers.
• Research Benchmark: The release of the MathWizard dataset and the 12-category error taxonomy sets a new standard for evaluating MWP generation.
• Limitations & Future Directions:
  • Curriculum Adherence: LLMs still struggle to perfectly match specific grade/section constraints. The authors propose using Retrieval Augmented Generation (RAG) in the future to ground the model in specific curriculum documents.
  • Learning Outcomes: Future work aims to generate problems based on specific "Learning Outcomes" rather than just broad sections.
  • Multilingual Expansion: Plans to extend the system to support multiple languages.

In conclusion, MathWiz demonstrates that while current LLMs cannot perfectly generate curriculum-specific math problems without assistance, a pipeline combining fine-tuning, few-shot prompting, preference optimization, and a secondary solvability check can produce high-quality, diverse, and safe educational content.