🤖 AI

A Scaffolded GenAI Lab in Early Undergraduate CS: A Mixed-Methods, Multi-Course Evaluation

This mixed-methods study demonstrates that a brief, scaffolded "AI-Lab" intervention across multiple undergraduate courses successfully increased students' comfort and openness toward using Generative AI for conceptual and debugging tasks while fostering more critical, iterative engagement strategies without increasing overall self-reported usage on graded assignments.

Original authors: Ethan Dickey, Andres Bejarano, Rhianna Kuperus, Bárbara Fagundes

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ethan Dickey, Andres Bejarano, Rhianna Kuperus, Bárbara Fagundes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of computer science as a massive, bustling library where students go to learn how to build things with code. For years, the only tools on the shelves were textbooks, lecture notes, and a little help from a teacher or a friend. But recently, a new, super-fast librarian has arrived: Generative AI (GenAI). This librarian can write code, explain tricky concepts, and solve problems in a blink. While this sounds like a dream, it's also a bit of a nightmare for teachers. They worry that if students just ask the librarian for answers, they won't actually learn how to build anything themselves. They also worry about cheating. So, the big question for educators is: How do we teach students to use this powerful new librarian wisely, without them just letting it do all the work? This is the puzzle researchers are trying to solve in the field of computing education.

Enter a team of researchers from Purdue University who decided to test a "training camp" for students called the AI-Lab. They didn't try to ban the librarian, nor did they let students run wild. Instead, they set up a short, structured workshop to teach students how to be smart partners with AI. They wanted to see if this quick lesson would change how students felt about using AI and how they actually used it. Did the students become more confident? Did they start using AI more for homework? Or did they learn to use it like a tool rather than a crutch?

The researchers ran this experiment across three computer science classes and one engineering class over two semesters, involving hundreds of students. They gave the students surveys before and after the AI-Lab to measure their feelings and habits, and they also held group discussions to hear the students' stories in their own words.

Here is what they found. After the workshop, students felt much more comfortable and open to using AI for tricky conceptual questions and for getting help with homework. They felt less scared of asking the AI for help. However, and this is a big "however," the number of times students said they used AI for their graded homework and big projects didn't go up. It stayed exactly the same. The only place where they used AI more was for debugging—fixing broken code.

The students' stories from the focus groups explained why. Before the lab, many students treated AI like a magic button: they typed a vague question and hoped for the best. If the answer was wrong, they gave up. After the lab, they became like skilled detectives. They learned to give the AI more context, to ask better questions, and to check the AI's work carefully. They realized the AI isn't perfect; it can make mistakes, especially in math or complex logic. They started treating it like a study partner who is very fast but sometimes gets things wrong, rather than a solution generator that does the work for them. They set clear boundaries, deciding that while AI is great for understanding a concept or fixing a bug, they still needed to do the heavy lifting on their own for their final grades.

The researchers are careful to say that they didn't prove that students actually cheated less or learned more, because they only asked the students what they said they did. But the results suggest that a short, smart lesson can change how students think about AI. It seems to make them more confident and strategic without turning them into people who just let the AI do their homework. It's like teaching a driver how to use a GPS: they might feel more confident driving in a new city, but they still have to steer the car themselves.

Technical Summary: A Scaffolded GenAI Lab in Early Undergraduate CS

Problem Statement
The widespread availability of Generative AI (GenAI) tools has disrupted computing coursework, enabling on-demand code generation and explanations. While educators recognize the need to clarify learning goals and establish integrity boundaries, many lack the bandwidth to redesign entire courses around GenAI or build extensive new assessment infrastructures. Existing literature highlights the necessity of educating students on ethical implications and limitations, yet there is limited evidence regarding how brief, course-embedded instructional interventions can foster responsible, learning-oriented use without requiring full curriculum overhauls. Specifically, it remains unclear whether lightweight scaffolds can shift students' self-reported usage patterns, openness, and comfort levels across common academic scenarios (conceptual questions, debugging, homework) without inadvertently increasing reliance on GenAI for graded work.

Methodology
The study employed a mixed-methods, multi-course evaluation of "AI-Lab," a scaffolded GenAI literacy intervention. The study was conducted over two semesters (Spring and Fall 2024) at a large U.S. research university across three early undergraduate Computer Science courses (DSA-CS, DSA-DSAI, CP) and one first-year Engineering course (ENGR).

Intervention Structure: The AI-Lab consists of three components: (1) a pre-lab orientation covering expectations, integrity boundaries, and basic prompting; (2) an in-class activity where students critique plausible but incomplete or incorrect GenAI outputs on course-aligned topics; and (3) a post-lab assignment requiring students to document and reflect on their interaction with GenAI for a specific homework task (e.g., prompt refinement, verification steps, model failures).
Quantitative Data: Paired pre- and post-intervention surveys were administered immediately around the lab. The survey battery included a "Usage" instrument (frequency of use) and a "Perception" instrument (openness and comfort). The primary analysis focused on 778 paired responses for perception and 773 for usage across the CS courses. Statistical analysis utilized paired non-parametric Wilcoxon signed-rank tests to assess shifts in ordinal Likert-scale responses, supplemented by rank-biserial correlation ( $r_{rb}$ ) for effect size estimation.
Qualitative Data: Six semi-structured focus groups were conducted 5–7 weeks post-intervention with voluntary participants. Transcripts were analyzed via thematic analysis to contextualize survey shifts, focusing on prompting strategies, verification behaviors, and boundary setting.

Key Contributions

Evaluation Framework: A mixed-methods, multi-course evaluation of a brief, scaffolded GenAI lab intervention using paired pre/post surveys and focus groups.
Empirical Evidence of Shifts and Stability: Documentation of where student perceptions and practices shifted (increased comfort and openness in specific scenarios) versus where they remained stable (frequency of use on graded work), including pedagogically consequential null findings.
Implementable Intervention Structure: A transferable lab framework detailing timing, components, and activities, explicitly noting boundary conditions and threats to validity (e.g., reliance on self-report).

Results

Quantitative Findings:
- Comfort and Openness: Students reported statistically significant increases in comfort using GenAI for conceptual questions, debugging, and homework. Openness increased significantly for conceptual questions and homework help.
- Usage Frequency: Self-reported frequency of GenAI use for homework and programming projects remained stable (no statistically significant change). However, self-reported frequency of use for debugging increased significantly.
- Effect Sizes: For items showing significant shifts, effect sizes were generally large ( $|r_{rb}| > 0.50$ ), indicating a consistent directional shift among respondents who changed their ratings.
- Subgroups: No statistically significant differences in shift direction or magnitude were detected across demographics (gender, race/ethnicity, residency, class standing), though these were treated as exploratory.
Qualitative Findings:
- From Casual to Strategic Use: Participants shifted from trial-and-error prompting to more iterative, context-rich strategies (e.g., providing constraints, refining prompts).
- Improved Awareness of Limitations: Students became more skeptical of GenAI correctness, particularly in math and complex logic, and more likely to verify outputs against course materials.
- Boundaries and Integrity: Students articulated clearer boundaries, often framing GenAI as a "TA-like" support for understanding concepts or debugging rather than a solution generator for graded work.
- Internal Tension: Participants expressed tension between the efficiency of GenAI and the desire to maintain ownership of their learning and avoid becoming "puppets" of the tool.

Significance and Claims
The paper claims that a short, structured intervention (AI-Lab) can effectively shift students' reported comfort with and willingness to use GenAI, as well as influence the strategies they describe for engaging with it. Crucially, the study found that these shifts occurred without increasing the overall self-reported frequency of GenAI use on graded work (homework and projects).

The authors position these results as evidence that lightweight scaffolds can shape student readiness and decision-making regarding GenAI. They explicitly state that the study does not claim to measure actual behavioral traces or learning outcomes, nor does it prove that the intervention prevents academic misconduct. Instead, the findings serve as an informative signal for instructors, suggesting that explicit instruction on limitations and boundaries does not necessarily lead to increased inappropriate reliance on GenAI for graded tasks. The paper concludes by motivating future work to triangulate self-reports with behavioral traces and learning measures to validate these patterns.

More like this