Zero-shot Large Language Models for Automatic Readability Assessment

1. Problem Statement

Automatic Readability Assessment (ARA) is critical for ensuring educational and medical materials are accessible to target audiences. While traditional readability formulas (e.g., Flesch-Kincaid) are easy to implement, they rely on shallow features (sentence length, syllable count) and often fail to capture contextual nuances. Conversely, supervised machine learning methods offer higher accuracy but require large, manually annotated datasets, which are costly and time-consuming to produce.

Recent research has explored unsupervised methods using Large Language Models (LLMs) via zero-shot prompting. However, existing studies (e.g., Trott & Rivière, 2024) have limitations:

• They rely on narrow evaluations (often single English datasets).
• They use simplistic prompting techniques (e.g., asking for a number without scale definitions).
• They primarily test closed-source models (GPT-4) rather than diverse open-source alternatives.
• There is a lack of understanding regarding how these methods perform across different languages, text lengths, and technical domains.

The paper addresses the gap in comprehensive evaluation and methodological rigor for using zero-shot LLMs as a robust, unsupervised ARA tool.

2. Methodology

The authors propose two main methodological advancements: a refined zero-shot prompting strategy for LLMs and a novel ensemble method called LAURAE.

A. Refined Zero-Shot Prompting for LLMs

The authors propose two specific improvements over standard prompting techniques:

1. Scale Definitions: Instead of asking for an arbitrary score, the LLM is prompted using the same scale and definitions used by human annotators (e.g., the Common European Framework of Reference for Languages - CEFR). For datasets without specific scales, a 1–9 scale is used with detailed definitions of readability factors (grammar, clarity).
2. Expected Value Scoring: Rather than accepting only the single "vanilla" token generated by the model (the highest probability token), the authors calculate the readability score as the expected value over the model's output token probabilities.
   • Formally, if the model generates a token at position $n$, the score $s_{LLM}$ is computed by summing the product of the probability and the token value for all numeric tokens in the top-ranked probability distribution until a non-numeric token is encountered. This reduces ties and captures model uncertainty.

B. LAURAE (LLM-based Automatic Unsupervised Readability Assessment Ensemble)

To improve robustness, the authors introduce LAURAE, an ensemble method that combines:

• Deep Contextual Features: Scores from the zero-shot prompted LLM.
• Shallow Features: Scores from traditional readability formulas (e.g., FKGL, ARI, Lix).

Weighting Mechanism:
Since the setting is unsupervised (no ground truth for tuning), LAURAE uses the LLM's self-reported verbal confidence as the weighting factor.

1. The LLM is prompted to output a confidence score (1–9) regarding how well its rating aligns with human raters.
2. This confidence score is converted into a weight $c$ (normalized to 0.1–0.9).
3. The final score is a weighted combination of the standardized LLM score and the standardized formula score:
   $$\text{LAURAE} = c \cdot \left(\frac{s_{LLM} - \mu_{LLM}}{\sigma_{LLM}}\right) + (1-c) \cdot \left(\frac{s_{rf} - \mu_{rf}}{\sigma_{rf}}\right)$$
   Where $c$ is the confidence weight, and $\mu, \sigma$ represent dataset-level means and standard deviations.

3. Experimental Setup

• Datasets: 14 diverse datasets were used, covering:
  • Languages: English, French, Hindi, Arabic, Russian, and Greek.
  • Lengths: Sentence-level (e.g., Asset, OneStop) to paragraph/article-level (e.g., MedReadMe, Greek textbooks).
  • Domains: General, medical (MedReadMe), educational (textbooks), and technical.
  • Ground Truth: 11 datasets with explicit ratings; 3 with pairwise comparisons.
• Models: 10 diverse open-source, instruction-tuned LLMs were evaluated, including Llama 3.1/3.2, Aya Expanse, Gemma 2, Mixtral, and Phi-4.
• Baselines: Traditional readability formulas (FKGL, ARI, etc.), RSRS (a PLM-based unsupervised method), and a baseline zero-shot LLM method (LLM-v-ns) mimicking prior work.

4. Key Results

A. Zero-Shot LLM Performance

• Prompting Improvements: The proposed prompting method (Scale Definitions + Expected Value) outperformed the baseline zero-shot method (LLM-v-ns) on 13 of 14 datasets.
• Model Selection:
  • Llama 3.1 70B performed best on English datasets.
  • Aya Expanse 32B (a multilingual model) outperformed Llama 70B on non-English datasets, highlighting the importance of multilingual training data.
• Generalization: The proposed method showed significant gains on non-English datasets, suggesting that providing explicit scale definitions helps LLMs overcome a lack of internalized knowledge in lower-resource languages.

B. LAURAE Performance

• Superiority: LAURAE outperformed all baselines (standalone LLMs, formulas, and RSRS) on 13 of 14 datasets.
• Robustness:
  • It successfully bridged the gap where standalone LLMs struggled (e.g., very short texts or specific technical domains) by leveraging shallow formula features.
  • It maintained high performance where formulas failed (e.g., complex contextual texts) by leveraging LLM understanding.
• Ablation Studies:
  • Using verbal confidence scores as weights proved superior to using entropy-based weights or equal weighting (naive ensemble).
  • Replacing readability formulas with RSRS in the ensemble resulted in slightly lower performance, confirming that simple formulas are a more effective "shallow" component for this specific task.

5. Key Contributions

1. Methodological Advancement: Introduced a robust zero-shot prompting framework involving scale definitions and expected value scoring, significantly improving LLM performance over prior art.
2. Novel Ensemble Method (LAURAE): Proposed a confidence-weighted ensemble that combines the contextual strength of LLMs with the stability of readability formulas, achieving state-of-the-art unsupervised performance.
3. Comprehensive Evaluation: Provided the first large-scale evaluation of 10 open-source LLMs across 14 diverse datasets (languages, lengths, domains), challenging the assumption that GPT-4 is the only viable option and revealing the necessity of multilingual models for non-English tasks.
4. Open Resources: Released code, datasets (including curated Greek textbook datasets), and prompts to facilitate future research.

6. Significance

• Practical Adoption: LAURAE offers a viable, high-accuracy alternative to manual annotation for readability assessment, particularly useful for healthcare and education sectors where manual labeling is too costly.
• Shift in Paradigm: The paper demonstrates that while pure LLMs are powerful, hybrid approaches (combining deep learning with shallow heuristics) yield the most robust results in unsupervised settings.
• Multilingual Insight: The findings emphasize that for global applications, multilingual-specific models (like Aya) are essential, and explicit prompting strategies are required to bridge performance gaps in non-English languages.
• Future Directions: The work suggests that combining zero-shot LLM ratings with shallow NLP tools could enhance other unsupervised tasks like sentiment analysis or toxicity detection.

Limitations Noted: The method requires Python literacy and significant computational resources (GPUs) compared to simple formula calculators. Additionally, the "black box" nature of LLM explanations was not fully evaluated for plausibility in this study.