Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach

1. Problem Statement

Automated Essay Scoring (AES) has advanced significantly for high-resource languages like English, particularly through the use of Large Language Models (LLMs) for trait-specific evaluation (e.g., organization, vocabulary, development). However, Arabic AES remains underdeveloped due to:

• Data Scarcity: A lack of large-scale, annotated datasets with granular trait-level labels.
• Linguistic Complexity: The morphological and syntactic complexity of Arabic.
• Methodological Gaps: Existing Arabic AES tools predominantly focus on holistic scoring or content correctness (e.g., fact-checking) rather than assessing specific linguistic proficiency traits.
• Resource Constraints: The high cost and computational requirements of fine-tuning custom models for low-resource educational contexts.

The paper addresses the need for a scalable, fine-tuning-free framework that can evaluate Arabic essays based on distinct linguistic traits using prompt engineering.

2. Methodology

The authors propose a three-tier prompting framework designed to guide LLMs in evaluating seven specific linguistic traits: Organization, Vocabulary, Style, Development, Mechanics, Structure, and Relevance. The study utilizes the QAES dataset, the first publicly available Arabic AES corpus with trait-level annotations (195 essays).

The Three-Tier Prompting Strategy:

1. Level 1: Standard Prompting (Zero-Shot)
   • A single prompt instructs the LLM to assign scores for all seven traits simultaneously based on a structured template.
   • No examples or rubrics are provided; the model relies on its pre-trained knowledge.
2. Level 2: Hybrid Trait Prompting (Zero-Shot Multi-Agent Simulation)
   • Simulates a multi-agent evaluation system where five specialized "raters" (A–E) are assigned to specific linguistic domains (e.g., Rater A for Organization, Rater B for Vocabulary).
   • Each rater evaluates only their specific domain.
   • Final scores are calculated by averaging the scores assigned by the relevant raters for each trait (Equation 1). This forces the model to focus on specific linguistic features rather than holistic judgment.
3. Level 3: Rubric-Guided Few-Shot Prompting
   • The model is provided with explicit scoring rubrics and three scored exemplars (low, mid, high scores) for each trait.
   • The model compares the target essay against these examples and the rubric to generate a score with justification in a structured JSON format.

Experimental Setup:

• Models: Eight LLMs of varying sizes and architectures were tested, including proprietary (ChatGPT-4) and open-source models (Fanar-1-9B, Jais-13B, ALLaM-7B, Qwen series, LLaMA-2).
• Metrics: Quadratic Weighted Kappa (QWK) was used to measure agreement between model predictions and human gold scores. 95% Confidence Intervals (CI) were calculated via bootstrapping to assess the stability of the agreement.

3. Key Contributions

• First Trait-Specific Arabic AES Framework: This is the first study to apply structured prompt engineering (zero-shot and few-shot) specifically for trait-level scoring in Arabic, moving beyond holistic or content-based scoring.
• Novel Hybrid Prompting: Introduces a "multi-agent" simulation approach where specialized raters evaluate specific traits, reducing cognitive load on the model and mimicking expert human assessment workflows.
• QAES Dataset Utilization: Leverages the QAES dataset to benchmark LLM performance on a resource-scarce task, establishing a baseline for future Arabic AES research.
• Empirical Validation of Prompting over Scale: Demonstrates that prompt structure is a more critical factor for performance than model size in this domain.
• Statistical Rigor: Introduces the use of Confidence Intervals alongside QWK scores to provide a more robust interpretation of model agreement in low-data regimes.

4. Results and Analysis

The experiments yielded the following key findings:

• Performance Gains: Structured prompting significantly improved performance. Level 3 (Rubric-Guided Few-Shot) consistently yielded the highest QWK scores and narrowest confidence intervals across most models.
• Top Performing Model: Fanar-1-9B-Instruct achieved the highest trait-level agreement in both zero-shot and few-shot settings (QWK = 0.28, CI Upper Bound = 0.41).
• Trait Variability:
  • Easier Traits: Vocabulary and Relevance showed higher agreement across models.
  • Harder Traits: Development, Organization, and Style showed lower QWK scores and wider confidence intervals, indicating these discourse-level traits are more challenging for LLMs to assess without fine-tuning.
  • Mechanics showed the most stable agreement (narrowest CI), likely due to its surface-level nature.
• Model Size vs. Prompting: Smaller models (e.g., Qwen1.5, LLaMA-2) struggled with complex prompts (Level 3), often failing to generate meaningful outputs (QWK = 0). However, Fanar-1-9B and ALLaM-7B (models with strong Arabic instruction tuning) performed competitively, proving that specialized training + good prompting can outperform larger, less specialized models.
• Comparison to Baselines: The best result (QWK 0.28) surpassed the previous baseline of 0.26 (achieved by Logistic Regression/AraBERT) but remains below the human inter-rater agreement threshold (0.72).

5. Significance and Conclusion

• Scalability: The study proves that prompt engineering is a viable, cost-effective alternative to fine-tuning for Arabic AES, making it accessible for low-resource educational contexts.
• Pedagogical Value: By focusing on specific traits (e.g., "Development" or "Style"), the framework provides diagnostic feedback that is more useful for learners than a single holistic score.
• Future Directions: The authors highlight the need for larger, more diverse Arabic datasets to improve generalizability and suggest exploring open-source alternatives to reduce reliance on expensive proprietary models.

Conclusion: The paper establishes that structured, trait-centric prompting is the key to unlocking effective Arabic essay scoring with LLMs. While current models do not yet match human raters, the hybrid and rubric-guided approaches significantly narrow the gap, offering a foundation for scalable, proficiency-oriented assessment in Arabic.