Personalizing explanations of AI-driven hints to users' characteristics: an empirical evaluation

This paper presents an empirical study demonstrating that personalizing AI-driven hint explanations in an Intelligent Tutoring System for students with low Need for Cognition and Conscientiousness significantly increases their engagement, understanding, and learning outcomes.

The Gist

Imagine you are trying to learn how to fix a complex engine, and you have a super-smart robot mechanic helping you. Every time you get stuck, the robot gives you a hint on what to do next. But sometimes, the robot also has a "Why?" button. If you click it, the robot opens a long, detailed manual explaining exactly why it gave that hint, how it calculated the answer, and what it was thinking.

The problem? Many people just want to fix the engine. They see the long manual, think, "That's too much reading," and ignore it. They miss out on the deep understanding that could help them become better mechanics in the long run.

This paper is about a team of researchers who asked: "What if we could tailor that manual to the specific personality of the student so they actually want to read it?"

Here is the breakdown of their experiment in simple terms:

1. The Target Audience: The "Hurry-Up" and "Overwhelmed" Students

The researchers focused on two specific types of students:

• Low "Need for Cognition": These are people who prefer to avoid heavy mental lifting. They'd rather guess than think deeply.
• Low "Conscientiousness": These are people who might get distracted easily or struggle to stick with a difficult task until it's finished.

Previous studies showed that these students usually ignored the robot's detailed explanations, even though the explanations would actually help them learn better. They were like students skipping the teacher's extra notes because they looked too boring or scary.

2. The Solution: The "Proactive Teacher"

The researchers decided to stop waiting for these students to click the "Why?" button. Instead, they changed the robot's behavior to be more like a proactive teacher who knows you might run away if you aren't guided.

They made two main changes for these specific students:

• The "Front-Page" Trick: Instead of hiding the explanation behind a button, the robot now automatically pops up the first page of the explanation right next to the hint. It's like the teacher handing you the first page of the notes before you even ask for them.
• The "Wait a Minute" Check: If a student tries to close the explanation window too quickly (before they've really read it), a little pop-up box appears. It says, "Hey, this is actually really helpful for you! Are you sure you want to leave?" It's like a gentle nudge from a friend saying, "Don't quit yet, you're almost there!"

3. The Experiment: A Race to Learn

The researchers set up a test with two groups of these specific students:

• Group A (The Control Group): Got the standard robot. They had to click the button to see the explanation, and they could close it whenever they wanted.
• Group B (The Experimental Group): Got the "Proactive Teacher" robot with the automatic pop-ups and the gentle nudges.

4. The Results: The Magic of Personalization

The results were a huge success for the "Proactive Teacher" group:

• They Read More: Group B looked at the explanations much longer and read more pages than Group A. The automatic pop-up worked like a charm to get their attention.
• They Learned More: Because they actually read the explanations, Group B scored significantly higher on their tests. They understood the "engine" better.
• They Trusted the Hints More: They felt the robot's advice made more sense because they understood the reasoning behind it.

5. The Catch: A Little Bit of Noise

There was one small downside. Because the explanations popped up automatically, some students felt a little bit distracted or confused at first. It's like having a teacher stand right next to you while you work; it's helpful, but it can feel a bit intrusive if you aren't used to it. However, the researchers found that the learning benefits far outweighed this minor annoyance.

The Big Picture Takeaway

This study proves that one size does not fit all when it comes to AI.

Think of AI explanations like a menu at a restaurant.

• For a foodie who loves to read about ingredients (High Need for Cognition), you give them the full, 10-page menu description.
• For someone who is hungry and just wants to eat (Low Need for Cognition), you don't hand them the menu and wait for them to order. You bring the dish out, maybe with a small card explaining the main ingredient, and you make sure they see it.

By adjusting how the AI delivers its explanations based on the user's personality, the researchers turned a tool that was being ignored into a tool that was actually helping people learn. It's a step toward AI that doesn't just "know" the answer, but knows how to talk to you so you'll listen.

Technical Summary

Here is a detailed technical summary of the paper "Personalizing explanations of AI-driven hints to users' characteristics: an empirical evaluation."

1. Problem Statement

Intelligent Tutoring Systems (ITS) increasingly utilize Explainable AI (XAI) to justify pedagogical actions (e.g., hints) to students. However, research indicates that the effectiveness of these explanations is not uniform; it depends heavily on user characteristics such as personality traits and cognitive capabilities.

Specifically, prior work identified that students with low Need for Cognition (N4C) (low inclination for effortful cognitive tasks) and low Conscientiousness (low dedication to goals/task management)—referred to as LNLC students—tend to ignore complex, on-demand hint explanations. Despite ignoring them, these students are the ones who benefit most from engaging with the explanations. The core problem is how to design an XAI interface that overcomes the natural tendency of LNLC students to disengage, thereby improving their learning outcomes.

2. Methodology

System and Context

The study utilized the ACSP (Adaptive CSP) applet, an ITS designed to teach Constraint Satisfaction Problems (CSPs) using the AC-3 algorithm. The system generates personalized hints based on a user model (FUMA framework) and provides multi-page, incremental explanations (combining text and graphics) detailing why a hint was given and how it was computed.

Personalization Strategy

The researchers implemented a Personalized Explainable AI (PXAI) interface targeting LNLC students. The intervention involved two main modifications to the original "on-demand" interface:

1. Upfront Delivery: Instead of requiring a click to view explanations, the first page of the explanation interface is automatically displayed alongside the hint. To mitigate confusion, the hint's visual prominence was increased (font size/color), and a text cue directed attention to the explanation window.
2. Proactive Retention (Confirmation Box): If a user attempts to close the explanation interface before spending a minimum threshold of time (set at 27 seconds, based on the median time high-N4C users spent in a prior study), a confirmation box appears. This box highlights the value of the explanation and asks for confirmation to close.

Experimental Design

• Participants: 39 university students recruited and pre-screened to ensure they scored below the median on both N4C and Conscientiousness tests (LNLC profile).
• Groups:
  • Experimental Group (n=23): Used the personalized interface (upfront delivery + confirmation box).
  • Control Group (n=16): Used the original non-personalized interface (on-demand access).
• Procedure: Participants studied the AC-3 algorithm, took a pre-test, solved three CSPs using the ACSP applet, took a post-test, and completed subjective questionnaires.
• Data Collection:
  • Behavioral Logs: Tracking access to explanation pages.
  • Eye-Tracking: Using a Tobii Pro X3-120 to measure fixation duration (Attention to Explanation - AE).
  • Learning Metrics: Percentage Learning Gains (PLG) calculated from pre/post-test scores.
  • Subjective Measures: Likert-scale questionnaires on usefulness, intrusiveness, and understanding.

3. Key Contributions

• Empirical Validation of PXAI in Complex Learning: This is one of the few studies demonstrating the efficacy of personalizing complex, multi-page AI explanations during a cognitively demanding task (learning CSPs), moving beyond simple text-based or music-recommender contexts.
• Targeted Intervention for LNLC Users: The study provides evidence that specific interface modifications (proactive delivery) can successfully counteract the disengagement tendencies of students with low N4C and low Conscientiousness.
• Bridging the Gap between Engagement and Learning: The work establishes a causal link between increased interaction with XAI and improved learning outcomes for a specific demographic that typically ignores such features.

4. Results

Interaction and Attention

• Access Rates: In the control group, only 43.8% (7/16) accessed the first explanation page, and only 3 explored beyond it. In the experimental group, 70% (14/20) accessed pages beyond the first one.
• Attention (Eye-Tracking): The experimental group showed significantly higher Attention to Explanation (AE) than the control group (Mean 34.1s vs. 6.6s; $p=0.001$, large effect size $r=0.70$).
• Confirmation Box Efficacy: The confirmation box had limited impact; only 3 out of 10 triggered users stayed longer. The primary driver of increased engagement was the upfront delivery mechanism.

Learning Outcomes

• Learning Gains: The experimental group achieved significantly higher Percentage Learning Gains (PLG) compared to the control group (Mean 0.7 vs. 0.2; $p<0.001$, very large effect size $d=2.88$).
• Correlation: A moderate, significant positive correlation ($r_s=0.5, p=0.007$) was found between the amount of attention paid to explanations (AE) and learning gains (PLG) across all participants.

Subjective Perceptions

• Understanding: The experimental group reported significantly higher General Understanding of the hints ($p=0.04$).
• Intrusiveness: While the experimental group showed trends toward rating the interface as more distracting or confusing (likely due to the upfront window covering the problem space), these differences were not statistically significant, and overall ratings remained positive (Usefulness > 3.5/5).

5. Significance and Future Work

This study provides robust evidence that Personalized XAI is not just a theoretical concept but a practical tool for improving education. By adapting the delivery of explanations to user traits (specifically LNLC), the system successfully transformed passive users into active learners, resulting in measurable learning gains.

Limitations and Future Directions:

• Dynamic Personalization: Currently, personalization relies on pre-study screening. Future work aims to use real-time detection of user traits (via eye-tracking and interface actions) to dynamically adjust explanations during the session.
• Refining Interventions: The study suggests refining the "confirmation box" logic (e.g., adjusting time thresholds based on previous viewing history) and potentially making upfront explanations "on-demand" if attention drops, to reduce perceived intrusiveness.

In conclusion, the paper demonstrates that for students who naturally avoid cognitive effort, proactive, personalized explanation delivery is a critical intervention that enhances engagement, comprehension, and ultimately, learning success in AI-driven educational environments.