Technical Summary: Investigating Learner-Aware Design of LLM-Generated Educational Feedback

Problem Statement

While Large Language Models (LLMs) offer promise for generating diverse educational feedback at scale, it remains unclear how specific feedback designs—such as tone, information coverage, and instructional specificity—should be structured to effectively support answer revision and learner acceptance across diverse learner profiles. Previous research has largely relied on coarse-grained comparisons between human and LLM feedback or automated evaluations, failing to systematically investigate how specific pedagogical elements influence learner behavior and perception. Furthermore, while individual differences (e.g., personality traits) are known to affect feedback effectiveness in human-designed contexts, their role in LLM-generated feedback remains underexplored. This study addresses the gap in understanding which feedback elements improve revision performance, which yield favorable subjective evaluations, and how these preferences vary according to learner personality profiles.

Methodology

The study employed a three-step workflow involving feedback definition, empirical experimentation, and analysis:

1. Feedback Design and Generation:
   The authors defined six distinct feedback elements for multiple-choice biology questions. A baseline "Normal" condition was manually designed, and five additional elements were derived from prior work (Borges et al., 2024):

   • Keywords: Highlighting important terms within explanations.
   • Actionability: Providing concrete revision steps and checklists.
   • Novelty: Introducing advanced perspectives beyond the learned content.
   • Coverage: Describing all logical elements required for the correct answer.
   • Positivity: Incorporating praise and encouraging comments.
     All feedback was generated in advance using GPT-5 (Singh et al., 2026) to ensure consistency, employing an indirect feedback approach that encourages learners to derive answers without explicitly revealing them.

2. Empirical Experiment:
   An experiment was conducted with 321 first-year high school students in Japan. Participants completed a pre-questionnaire assessing their Big Five personality traits (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) and background knowledge. They then engaged in a web-based task involving seven biology questions. For each question, a specific feedback element was randomly assigned and remained fixed throughout the revision process. Participants iteratively revised their answers and evaluated the feedback using a three-point scale across six criteria: Trustworthiness, Review Guideline, Expression Quality, Ease of Understanding, Key Points Clarity, and New Knowledge.

3. Analysis:
   The study analyzed immediate revision performance (accuracy of the first revised attempt) and subjective evaluations. To account for repeated observations from the same learner, Generalized Estimating Equations (GEE) were used. Additionally, learners were clustered into three personality profiles (Social, Reserved, Adaptable) using k-means clustering to examine how preferences varied across groups.

Key Results

Revision Performance (RQ1)

Feedback elements significantly influenced immediate revision accuracy. The Normal design yielded the highest accuracy (64.2%), followed by Keywords (58.2%) and Coverage (52.9%). Conversely, Actionability and Positivity were associated with significantly lower revision accuracy compared to the baseline. This suggests that feedback which clearly directs attention to task-relevant points and provides comprehensive guidance is most effective for immediate correction, whereas overly specific action steps or purely affective support may distract or fail to guide the revision process effectively.

Subjective Evaluations (RQ2)

Learners generally favored feedback that was practical and easy to apply. Normal, Keywords, and Actionability received the highest subjective ratings across most criteria. Positivity consistently received the lowest ratings across all six evaluation criteria, indicating that praise-oriented feedback alone is perceived as less trustworthy and less helpful for revision. Novelty also received lower ratings for clarity and expression quality, suggesting that introducing information beyond the scope of the question can hinder the perceived usefulness of the feedback.

Personality-Based Preferences (RQ3)

Preferences for feedback elements varied significantly across learner personality clusters:

• Social Learners (High Extraversion/Agreeableness): Prioritized validity and trustworthiness, showing balanced preferences for Normal and Keywords.
• Reserved Learners (Low Extraversion/High Neuroticism): Showed strong preferences for Keywords and Normal, while rating Positivity and Novelty significantly lower. This aligns with findings that learners high in neuroticism prefer clear cues and low cognitive load.
• Adaptable Learners (High Openness/Conscientiousness/Agreeableness): Accepted a broader range of elements, showing particular preference for Actionability and Coverage, suggesting a higher tolerance for complex or detailed guidance.

Statistical tests (Kruskal–Wallis) confirmed significant differences in evaluation patterns across clusters for all criteria, indicating that learner profiles partially influence how generated feedback is perceived.

Contributions and Significance

The paper makes three primary contributions:

1. Definition and Empirical Examination: It defines six controllable educational elements for LLM-generated feedback and empirically examines their effects on both revision performance and subjective evaluations through a large-scale study with 321 students.
2. Evidence of Profile-Dependent Effects: It demonstrates that feedback preferences and perceived effectiveness are not uniform but vary significantly across learner profiles defined by Big Five personality traits.
3. Implications for Personalization: The findings suggest that effective LLM-generated feedback design must consider learner profiles. While clear, comprehensive guidance is broadly effective, elements like informational novelty and affective framing should be adapted to individual personality traits to maximize acceptance and utility.

The study concludes that while certain feedback characteristics (clarity, comprehensiveness) are universally beneficial, the optimal design for specific elements (such as tone or novelty) depends on the learner's personality. This underscores the potential and necessity of personalizing LLM-generated feedback based on learner profiles to enhance educational outcomes. The authors note limitations regarding the specific subject domain (biology), the use of a single LLM model, and the focus on immediate rather than long-term learning gains, suggesting future work should explore broader domains and longitudinal effects.