💻 computer science

Engagement Intensity as a Learner-Modeling Signal for Adaptive AI Ethics Instruction

This study demonstrates that among graduate trainees, self-reported LLM usage frequency is a more consistent and effective predictor of pre-instruction AI ethics perceptions than prior coursework or self-rated familiarity, suggesting that simple behavioral signals can effectively inform adaptive AI ethics education.

Original authors: Yongkyung Oh, Lynn Talton, Alex Bui

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yongkyung Oh, Lynn Talton, Alex Bui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef preparing a special cooking class for a group of aspiring chefs. You know that some of them have never touched a knife, while others have been cooking for years. To teach them effectively, you need to know where they are starting from.

This paper is about figuring out the best way to ask students, "How much experience do you have with AI?" before teaching them about AI ethics. The researchers wanted to know: What question gives us the most accurate picture of a student's starting point?

They tested three different ways to ask about experience:

The "Resume" Question: "Have you taken a class or workshop on AI before?"
The "Confidence" Question: "On a scale of 1 to 5, how familiar do you feel with AI?"
The "Action" Question: "How often do you actually use AI tools?"

Here is what they found, using some simple analogies:

1. The "Resume" Was a Bad Map

The researchers asked students if they had attended any AI courses or workshops. They expected this to be a strong signal.

The Result: It was like asking a traveler, "Have you ever been to Paris?" and assuming that if they said "yes," they know how to navigate the streets. It turned out that just having a "ticket" (taking a class) didn't tell the teachers much about how the students actually felt about AI or how they planned to use it. The data showed no clear link between having taken a class and the students' current views on AI.

2. The "Action" Question Was the Best Compass

The researchers looked at how often students actually used AI (from "Never" to "Daily").

The Result: This was the most powerful signal. It was like asking a driver, "How many miles have you driven this year?" The answer perfectly predicted their driving habits and confidence.
- Students who used AI daily had very different views than those who never used it.
- Specifically, heavy users were more trusting of AI for general facts, felt more capable of spotting fake info, and were less worried that AI would ruin their critical thinking skills.
- Even the students who used AI "occasionally" showed a clear jump in confidence compared to those who never used it.

3. The "Confidence" Check Was a Good Second Opinion

The "How familiar do you feel?" question was helpful, but not as perfect as the "How often do you use it?" question.

The Result: It worked well for some things (like spotting fake news or trusting AI with complex tasks) but was a bit fuzzy for others. It's like a passenger saying, "I feel like I know the way," which is often true, but the driver's actual mileage (usage frequency) tells a more consistent story.

The "Threshold" Surprise

One interesting discovery was that the change didn't happen gradually like a ramp. Instead, it looked more like a light switch.

If you had never used AI, your views were one way.
As soon as you started using it even a little bit, your views shifted significantly.
However, for some specific fears (like "Will AI make me lazy?"), the change only happened among the daily heavy users.

Why Does This Matter?

The researchers suggest that if you are designing a class to teach AI ethics, don't just look at a student's transcript to see if they took a class. Instead, ask them a simple question: "How often do you use AI?"

If they say "Never": They probably need a basic introduction to what AI is and why it matters.
If they say "Daily": They already know the basics. They might need a different kind of lesson focused on the dangers of over-trusting AI or spotting when it fails, because they seem less worried about those risks than they should be.

The Bottom Line

In short, what you do (using the tool) is a much better predictor of what you think and feel than what you've studied (taking a class) or how confident you feel. For teachers trying to customize their lessons, asking about actual usage habits is the simplest and most effective way to group students before the class starts.

Note: The paper is a snapshot in time (a survey) and doesn't prove that changing the teaching style based on these answers will definitely make students learn better—that would require a future experiment. It simply says these are the best questions to ask to understand the students right now.

Technical Summary: Engagement Intensity as a Learner-Modeling Signal for Adaptive AI Ethics Instruction

Problem Statement
Graduate trainees entering AI ethics instruction possess heterogeneous prior experiences with Large Language Models (LLMs) and varying perceptions of their capabilities. While established AI literacy frameworks emphasize learner-centered instruction, there is a lack of empirical evidence regarding which pre-instruction (intake) features best differentiate learners to inform instructional personalization. Intuitive proxies such as prior coursework or workshop attendance have not been rigorously tested against behavioral signals to determine if they capture the variation necessary for effective adaptive design in graduate research settings.

Methodology
The study analyzed pre-instruction survey data from 93 bioscience graduate and postdoctoral trainees enrolled in a required Responsible Conduct of Research (RCR) course at UCLA. The research evaluated three candidate intake features:

Self-reported usage frequency (ordinal, 5 levels: Never to Daily).
Self-rated LLM familiarity (ordinal, 5 levels: Not at all to Very).
Prior AI education (categorical: Formal, Informal, or None).

These features were tested against five focal perception outcomes, each measured on a five-point Likert scale:

Accuracy trust (belief in LLM accuracy for general scientific info).
Distinguishing capability (ability to separate factual from incorrect LLM output).
Complex-task trust (trust in LLMs for nuanced ethical/scientific concepts).
Critical-thinking risk (belief that over-reliance impairs critical thinking).
Training interest (interest in formal LLM training).

Statistical analysis employed Spearman correlations for ordinal predictors (usage frequency and familiarity) and Kruskal–Wallis $H$ tests for the categorical predictor (prior education). To address multiple comparisons, Holm correction was applied across the five outcomes within each predictor family. Effect sizes for binary contrasts were summarized using Cliff's $\delta$ .

Key Results
The analysis revealed distinct hierarchies in the predictive power of the intake features:

Usage Frequency: This feature demonstrated the strongest and most consistent associations, surviving Holm correction across all five outcomes. It showed the largest effect sizes for complex-task trust ( $\rho = .409$ ), accuracy trust ( $\rho = .370$ ), and critical-thinking risk ( $\rho = -.346$ ).
Self-Rated Familiarity: This served as a meaningful but weaker secondary signal, surviving Holm correction for three outcomes: distinguishing capability, complex-task trust, and critical-thinking risk. The association pattern was "bottom-anchored," showing clear separation at the lower end of the scale (non-familiar vs. slightly familiar) but often identical medians for higher familiarity levels.
Prior AI Education: Neither the three-group coding (Formal/Informal/None) nor the collapsed two-group coding (Any prior vs. None) yielded statistically significant associations with any of the five outcomes after Holm correction. The largest effect sizes were negligible (Cliff's $|\delta| < .19$ ).

A notable finding was the negative association between usage/familiarity and critical-thinking risk; heavy users and those feeling more familiar reported less concern about over-reliance impairing critical thinking.

Key Contributions

Comparative Validation of Intake Features: The study empirically demonstrates that simple behavioral signals (reported usage frequency) and self-perception signals (familiarity) are more informative for pre-instruction segmentation than education-history labels (prior coursework/workshops) in a graduate bioscience cohort.
Identification of Threshold Effects: The data suggests a threshold-like pattern rather than a uniform gradient, particularly for training interest and accuracy trust, where the distinction between non-users and any sustained users is more pronounced than differences among varying levels of frequent use.
Calibration Concerns: The results highlight a potential calibration issue where highly engaged learners (high usage/familiarity) exhibit lower sensitivity to the risks of over-reliance, suggesting a need for targeted instructional interventions for this subgroup.

Significance and Claims
The paper claims that lightweight intake profiling for adaptive AI ethics education can be effectively achieved using short pre-instruction surveys that prioritize reported LLM use and self-rated familiarity over traditional educational history. The authors argue that these behavioral and self-perception signals provide a more reliable basis for distinguishing learners who may require basic orientation from those who might benefit from calibration-focused work on trust, over-reliance, and failure modes.

The study explicitly positions these findings as associational and cross-sectional, noting that the single-item self-report nature of the measures precludes claims of causality or multi-item reliability. The authors conclude that while these signals are useful for intake-level grouping, future intervention studies are required to determine if such differentiated instruction actually improves learning outcomes.