🤖 AI

AI-Automation Tooling in Computer Engineering Education: Mixed-Methods TAM/UTAUT Evidence for a General Acceptance Attitude

This mixed-methods study of 103 Thai undergraduate computer engineering students demonstrates a strong, generalized acceptance of AI automation tools (specifically n8n) driven primarily by performance expectancy, while revealing a divergence between quantitative enthusiasm and qualitative skepticism regarding output quality, thereby supporting the curricular integration of such tools with targeted instructional scaffolds.

Original authors: Aung Pyae

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aung Pyae

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to introduce a new, super-smart robot assistant to a class of future engineers. Before you let them loose to build things with it, you want to know: Do they actually like it? Do they think it's useful? And do they trust it?

This paper is the report card from a study where 103 computer engineering students in Thailand tried out a specific type of "robot brain" tool (called n8n) during a series of workshops. The researchers wanted to see if these students were ready to embrace AI automation in their future careers.

Here is the breakdown of what they found, using simple analogies:

1. The Setup: A "Test Drive" for the Future

Think of the workshops as a test drive for a new car. The students didn't just read about the car; they got behind the wheel for two hours. They used a free, open-source tool called n8n to connect different digital tasks together (like making an AI agent that reads a message and saves it to a database).

After the "test drive," the students filled out a short survey. The survey asked them to rate the tool on six different "gauge meters" borrowed from famous technology theories:

Performance Expectancy: "Will this help me get my job done?"
Effort Expectancy: "Is this hard to learn?"
Behavioral Intention: "Will I use this again?"
Self-Efficacy: "Do I feel capable of using this?"
Hedonic Motivation: "Is this fun?"
Output Quality: "Does the tool give good results?"

2. The Results: The "Halo Effect"

The students gave the tool very high marks across the board. Almost everyone agreed that the tool was useful, easy to use, and something they would use again.

The Big Surprise (The "Halo"):
Usually, when you ask people about a new tool, you expect them to have different opinions on different things (e.g., "It's useful, but boring," or "It's fun, but hard to use").

However, the researchers found something interesting: The students couldn't tell the difference between these categories yet.

The Analogy: Imagine looking at a bright, shiny new smartphone through a slightly foggy window. You can see it's a phone, and it looks great, but you can't quite distinguish the camera from the screen or the battery from the processor yet.
The Finding: Because the students only used the tool for a short time (one workshop), their brains lumped all six questions together into one single feeling: "I like this tool." The researchers call this a "general acceptance attitude." The specific details (like whether it's fun vs. useful) hadn't separated in their minds yet.

3. The "Trust Gap": The Quiet Skeptics

While the survey numbers were glowing (everyone said the tool produced high-quality results), the open-ended comments told a slightly different story.

The Analogy: Imagine a group of people tasting a new cake. 95% say, "This is delicious!" But a small group of 5% whispers, "Wait, the frosting is a bit inconsistent; sometimes it's perfect, sometimes it's lumpy."
The Finding: The survey said the "Output Quality" was great. But the few students who wrote comments mentioned that the AI sometimes gave inaccurate or inconsistent results.
Why it matters: The researchers realized that just because students say they trust the tool (based on the survey), it doesn't mean they have calibrated trust. They might be too trusting, or they might not know when to double-check the work. The paper suggests that students need to be taught not just how to use the tool, but how to spot when the tool makes a mistake.

4. What This Means for Education

The paper concludes with three "levers" (tools) that teachers can use to help students get the most out of AI automation:

The Scaffolding Ladder: Since students found the tool useful but sometimes tricky, teachers need to build a ladder of support. Start with easy steps and slowly remove the hand-holding so students feel confident building things on their own.
The Confidence Boost: Some students felt they could use the tool, but only if they had help. Teachers need to give them specific support to bridge the gap between "I know what this does" and "I can do this myself."
The Trust Calibration: This is the most important one. Teachers shouldn't just show students how the tool works perfectly. They need to show students when the tool fails. This helps students learn to verify the AI's work rather than blindly trusting it.

Summary

In short, this paper says: Future engineers are excited about AI automation tools and think they are useful. However, after just one short workshop, they see the tool as a single "good thing" rather than a complex machine with different strengths and weaknesses.

The key takeaway for educators is: Don't just celebrate the enthusiasm. You also need to teach students how to handle the tool's occasional mistakes so they become smart, critical users rather than just happy ones.

Technical Summary: AI-Automation Tooling in Computer Engineering Education

Problem Statement
As generative AI and low-code workflow platforms become integral to software practice, a critical empirical question remains for computer engineering education: do undergraduate students accept these tools as useful, usable, and worthy of sustained engagement? While the next generation of engineers will be expected to design and reason about AI-augmented automations, there is a lack of evidence regarding student attitudes toward such tooling, particularly within Southeast Asian educational contexts. Furthermore, existing literature often lacks rigorous methodological handling of short-form, single-source instruments used to measure acceptance immediately after instructional interventions.

Methodology
The study employs a cross-sectional, explanatory mixed-methods design involving 103 undergraduate computer engineering students at a Thai university. The participants, possessing basic-to-intermediate Python skills and daily experience with general-purpose AI tools, attended three identically scripted workshops utilizing the open-source low-code platform n8n.

Instrumentation: A 12-item, five-point Likert questionnaire was administered post-workshop, mapping to six constructs derived from the Technology Acceptance Model (TAM), Unified Theory of Acceptance and Use of Technology (UTAUT), and UTAUT2: Performance Expectancy (PE), Effort Expectancy (EE), Behavioral Intention (BI), Self-Efficacy (SE), Hedonic Motivation (HM), and Output Quality (OQ). Each construct was measured by two items.
Quantitative Analysis: To address the limitations of a parsimonious design (two items per construct, $N=103$ $N = 103$ ), the study avoided Structural Equation Modeling (SEM) in favor of robust non-parametric and ordinal-specific techniques. Analyses included:
- Reliability estimation using Spearman-Brown coefficients and McDonald's omega ( $\omega$ ) based on polychoric correlations.
- Bootstrap confidence intervals (95%, $B=5,000$ ) for construct means.
- One-sample Wilcoxon signed-rank tests against the neutral midpoint (3.0).
- Spearman correlations with Holm correction for multiple comparisons.
- Dimensionality diagnostics: Horn's parallel analysis, Harman's single-factor test, and Kaiser-Meyer-Olkin (KMO) measures to assess common-method bias and factor separability.
Qualitative Analysis: An inductive thematic analysis was conducted on 13 substantive open-ended comments to identify convergences and divergences with the quantitative data.

Key Results

High Acceptance Levels: All six constructs demonstrated means significantly above the neutral midpoint ( $p < 0.0001$ ) with large effect sizes ( $r$ ranging from 0.68 to 0.81). The ranking of mean scores was PE (4.28) > OQ (4.16) > EE (4.11) > BI (4.05) > SE (4.04) > HM (4.00).
Dimensional Collapse: Dimensionality diagnostics revealed that the six theoretically distinct constructs collapsed into a single general acceptance factor. Horn's parallel analysis retained only one factor (eigenvalue 8.77 vs. random benchmark ~1.76), and Harman's test showed a first-factor variance of 73.1%. This suggests that in a short-form, post-workshop context, students do not empirically differentiate between the sub-facets of acceptance; they respond with a holistic "halo" attitude.
Quantitative-Qualitative Divergence: While quantitative scores for Output Quality (OQ) were high, qualitative feedback revealed a "reliability-sceptical minority." A small subset of students expressed concerns about inconsistent prompts and inaccurate results, indicating that aggregate endorsement does not equate to calibrated trust in automation.
Session Consistency: Kruskal-Wallis tests confirmed no statistically significant differences in acceptance profiles across the three workshop sessions, validating protocol fidelity.

Key Contributions

Empirical: The study provides the first construct-level evidence of AI-automation tooling acceptance among Thai computer engineering undergraduates, a population under-represented in TAM/UTAUT literature.
Methodological: It offers a transparent analytical framework for short-form, ordinal, single-source instruments. By utilizing polychoric correlations, non-parametric tests, and explicit dimensionality diagnostics, the paper demonstrates how to rigorously analyze data where SEM is under-identified, avoiding default assumptions of factor separability.
Theoretical: The finding that canonical TAM/UTAUT sub-facets collapse into a single general acceptance attitude in short-term instructional settings challenges the assumption that these constructs are empirically separable immediately after a single exposure. It suggests the full nomological network of acceptance may require sustained, longitudinal exposure to manifest.
Pedagogical: The integration of mixed methods identifies three theory-grounded instructional levers:
1. Instruction-sequencing scaffolds to address Effort Expectancy friction.
2. Self-efficacy supports to bridge the gap between recognizing utility and feeling capable.
3. Trust-calibration interventions (e.g., structured opportunities to encounter and verify failures) to address the divergence in Output Quality perceptions.

Significance and Claims
The paper claims that while undergraduate acceptance of AI-automation tooling is broadly favorable and anchored in instrumental usefulness, the immediate post-workshop attitude is a generalized, undifferentiated construct rather than a set of distinct psychological factors. Consequently, the authors argue that short-form acceptance instruments should be interpreted as measures of a single attitudinal dimension, supplemented by qualitative probes to detect specific concerns like reliability.

The study concludes that curricular adoption of such tools is supported by the data, but effective pedagogy must move beyond simple exposure. It must actively address the "appropriate-reliance problem" by designing instruction that calibrates student trust through verification behaviors, rather than assuming initial enthusiasm translates to mature, calibrated reliance on automated systems. The authors explicitly note that their findings are limited to a single institution and cross-sectional design, precluding causal inference, and call for longitudinal studies to track whether initial favorable attitudes mature into appropriate long-term usage behaviors.