Technical Summary: Parse – An Open-Domain Reasoning Question Answering Benchmark for Persian

Problem Statement

While Large Language Models (LLMs) have rapidly advanced Reasoning-focused Question Answering (QA), high-quality benchmarks for low-resource languages remain scarce. Persian, spoken by approximately 130 million people, lacks a comprehensive open-domain resource for evaluating reasoning-capable QA systems. Existing Persian datasets (e.g., PersianQA, PQuAD, PersianMHQA) are limited by narrow scope, focusing on specific domains (medical, Islamic) or single task types (factoid extraction, multi-hop only). There is no existing benchmark that jointly supports a broad spectrum of QA phenomena—including Boolean, multiple-choice, and factoid formats—combined with diverse reasoning requirements (multi-hop, complex inference) and answer structures (single-answer, multi-answer, unanswerable) in an open-domain setting.

Methodology

Dataset Construction (Parse)

The authors introduce Parse, the first open-domain Persian reasoning QA benchmark, constructed via a controlled, LLM-based generation pipeline validated through human evaluation.

• Taxonomy and Design: The dataset comprises 10,800 questions organized into a cross-product of three primary QA types (Boolean, Multiple-choice, Factoid), two reasoning dimensions (Multihop, Reasoning), and various content subtypes (e.g., Negation, Comparative, List-based, Non-answerable). This results in 18 distinct configuration families.
• Generation Pipeline:
  • Model: GPT-4o was used for generation.
  • Prompting: Configuration-specific prompts were designed to encode the QA type, reasoning dimension, and difficulty level (Easy/Medium/Hard). Prompts included role descriptions, task blocks, constraints (e.g., option counts, Persian language usage), and CSV-style output instructions.
  • Batching: Questions were generated in batches of 30 (10 per difficulty level) to maintain output quality.
• Quality Control and Validation:
  • Automated Checks: Structural validity (field completeness, option cardinality), de-duplication, and semantic checks (e.g., ensuring "None" is the only correct option for non-answerable items) were enforced.
  • Human Evaluation: Two studies were conducted:
    1. Quality Evaluation: 1,080 items (10% of the dataset) were rated by 12 native Persian speakers on Ambiguity, Readability, and Correctness (1–5 scale). Average scores exceeded 4.0 across all metrics.
    2. Difficulty Evaluation: Annotators answered a subset of questions without knowing the assigned difficulty labels. Performance correlated with difficulty levels (highest on Easy, lowest on Hard), validating the label assignments.
• Final Composition: The dataset is evenly split by difficulty (200 Easy, 200 Medium, 200 Hard per configuration), totaling 10,800 QA pairs.

Experimental Setup

The authors evaluated six general-purpose multilingual LLMs (Qwen-2.5, LLaMA-3, Mistral, Gemma-2) and one Persian-specialized model (Dorna).

• Prompting Strategies: Zero-shot, Few-shot, and Chain-of-Thought (CoT) were tested in both English and Persian.
• Metrics: Accuracy was used for Boolean and single-answer tasks; Jaccard similarity for multi-answer and list-based tasks; and specific metrics for non-answerable cases.
• Fine-tuning: A subset of 8,640 questions was used to fine-tune LLaMA-3 8B and Dorna, with a held-out test set of 2,160 questions.

Key Results

Model Performance and Prompting

• Language Impact: Persian prompts consistently outperformed English prompts across all models and task types, likely because the questions are native Persian, and Persian prompts guide models to interpret them more faithfully.
• Prompting Strategy:
  • Boolean & Multiple-choice: Chain-of-Thought (CoT) prompting yielded the strongest performance, suggesting these tasks benefit from stepwise reasoning.
  • Factoid: Few-shot prompting performed best, indicating that example-driven prompting is more effective than stepwise reasoning for factoid retrieval.
• Model Capabilities: Even large models (e.g., LLaMA-3 70B, Qwen-2.5 72B) showed significant room for improvement on Persian reasoning tasks, particularly in complex reasoning and multi-hop scenarios.

Fine-tuning Effects

• Specialization: The Persian-specialized model, Dorna, outperformed the base LLaMA-3 8B, highlighting the value of pretraining on Persian corpora.
• Adaptation: Fine-tuning on Parse yielded substantial performance gains for both models. Fine-tuned Dorna achieved the highest overall performance, demonstrating that Parse serves as an effective resource for adapting LLMs to Persian reasoning tasks.

Significance and Contributions

The paper claims three primary contributions:

1. Resource Creation: The introduction of Parse, the first open-domain reasoning QA benchmark for Persian, covering 10,800 diverse questions across multiple formats, reasoning categories, and difficulty levels.
2. Validation: The execution of rigorous human evaluation studies to validate the linguistic quality, factual correctness, and difficulty calibration of the benchmark.
3. Empirical Analysis: Comprehensive experiments demonstrating the utility of Parse for benchmarking multilingual and Persian LLMs, analyzing the effects of prompting strategies, and showing the efficacy of fine-tuning for model adaptation.

The authors position Parse as a critical resource that fills a gap in Persian NLP and Information Retrieval research. By providing a structured, high-quality taxonomy of reasoning tasks, Parse enables fair comparisons and supports the development of reasoning-capable LLMs in low-resource settings, bridging the gap between Persian QA research and established English benchmarks.